You can get all the details from CLTBlog.

If you want to come down to meet the people also we’ll be getting an invite out. Hopefully see you there.

Since coming to Charlotte several years ago I have tried several paths to connecting with the tech scene in the city,  restarting the Charlotte Java User Group and then helping organize two Barcamps. I’m met up with some great people in the city – but I still have felt like I haven’t really met the start-up elements,  those individuals who are driven to creating new technology companies and working in environments which foster innovation.

After the last Barcamp I was chatting with some of the guys from CLTBlog and Area15 and we started talking about doing a set of podcasts to explore that side of Charlotte and hopefully bring more of these activities to the surface.  After that there was the usual lull of getting back to daily work and filtering out enough time – then I decided to get together with Justin from CLTBlog and the Netphase guys and very quickly (within a few days) we had our first podcast at Area15.   One of the big hits was meeting up with Josh from Charlotte 2.0 – he is a man on a mission to help bring more start-ups to Charlotte and I’m sure I’m going to be listening and helping him with it all I can.

Charlotte 2.0.1 from Justin Ruckman on Vimeo.

We are working to produce another podcast in mid January.  Hopefully it will establish the conversation – something that needs to start if we are going to build some new tech start-ups here in Charlotte.  It all makes me feel much more excited about 2010 – and I’m looking forward to sharing more about what I’m going to be doing too!

I can’t really go to much into the functionality of the app,  however I thought I would share some of the experiences I’ve had building the application in the Google environment with Grails.  First off,  if you are going to be doing any serious work with Grails then you’ll need to get a copy of Intellij,  the experience developing with it is much better than others I tried (and improving even more in the 9.0M1 release).  The initial cut of the application was written using GORM and against a local database,  however after trying GORM-JPA (it is very early still though moving quickly), I decided that trying to take a RDBMS mapping over to AppEngine datastore was not going to be a productive route.  The first real lesson of AppEngine is understanding the data store,  it is strange but several of the applications I have worked on over the past 4-5 years have started to move away from traditional RDBMS.  I’m not about to get into the NoSQL argument, other than to say ultimately you need to sit down and understand the tools that are available.  AppEngine DataStore is a powerful storage engine (I’m purposefully not calling it a database),  though it does take a little getting used to.  If you are planning to take a look I would start with the recent Google IO presentation by Max Ross.

Since I have been working with HBase lately some of the concepts weren’t so alien, however it I did find that using JPA annotations actually lead me into incorrect thinking at times,  while adding the annotations I was immediately thinking in the RDBMS mapping terms.  Since I wanted to use the JPA annotations, but didn’t feel that GORM-JPA was quite ready I switched over and used the Grails JPA plugin,  in some ways this meant creating a DAO to allow me to interact with the store.  In the end I did this through the creation of a service which I used in the controllers,  it was pretty generic thanks to the nature of groovy – in the end I can see myself probably moving back to GORM later on,  though in the meantime having a closer relationship with the JPA interface allowed me to better understand how relationships were being managed by AppEngine.  GORM has a tendency to abstract you a little to much at times and coming to terms with a new type of storage engine means that abstraction leads to another level of indirection.  I can see that as time passes and the ins and outs of AppEngine Datastore become more like second nature then that indirection is something that is easy to handle,  though at the beginning I think it is easier to get used to the datastore without it there.

Plugins is one of the powerful aspects of Grails, however one of the problems you encounter in the AppEngine world is the file limit (currently 3,000). While adding a lot of UI plugins speeds up development you can quickly find that you have blown the file limit. Also make sure you watch the $USER_HOME/grails/1.1.1/projects/{yourproject}/stage directory, if you start removing things from your application (ie. unneeded JS) then you might well find that they are remaining in the stage directory.

If you are using YUI you can switch to their servers, so that you don’t have to hold all the YUI files,  first add the following to your BootStrap:

def init = {servletContext ->
JavascriptTagLib.LIBRARY_MAPPINGS.yui = []
}

Then in your main.gsp make sure you reference the CSS and JavaScript you need.

If you are using the grails-ui plugin 1.0.4 it is currently using the javax.rmi.UID class to generate UUIDs, this class is blacklisted by GAE/J and therefore you would need to change the GrailsUITagLibService.groovy and remove the import.  I found that I was able to use the UUID class to get a similar effect (see JIRA)

def getUniqueId = { 'gui_' + DigestUtils.md5Hex(UUID.randomUUID().toString()) }

Once you have the basics in place, then you will need security. Obviously Google provides API’s to allow this to exist, however often you need more complex security than is currently provided by the Google API. In Grails I naturally jumped for the Aecgi plugin, however after a few battles with its close connection with Hibernate (I get the feeling that plugin does a little too much and would be well served by being broken up into a several plugins rather than a lot of configuration in one plugin). Next up was Stark Security plugin – I have to admit it was a little new to me, however I was able to quickly implement this in grails and also get it functioning in GAE/J. Basically I needed to create the basic DAO stuff using the plugin then change it to use JPA methods (as seen below):

User.groovy

import org.datanucleus.jpa.annotations.Extension
import org.springframework.security.GrantedAuthority
import org.springframework.security.userdetails.UserDetails
import javax.persistence.*

/**
* This class is the default UserDetails implementation for the Stark Security plugin.
* Since instances of this class will be used by the underlying Spring Security framework across
* Hibernate sessions, we can't have any lazy loading in here (see special handling of roles below).
*/
@Entity
class User implements UserDetails {

@Id
@GeneratedValue (strategy = GenerationType.IDENTITY)
@Extension (vendorName = "datanucleus", key = "gae.encoded-pk", value = "true")
String id

@Column
String username

@Column
String password

List<String> roles = new ArrayList<String>();

GrantedAuthority[] getAuthorities() {
return roles.collect { new Role(authority:it) } as GrantedAuthority[]
}

boolean isAccountNonLocked() {
return true
}

def setAccountNonLocked(boolean nonLocked) {}

boolean isCredentialsNonExpired() {
return true
}

def setCredentialsNonExpired(boolean nonExpired) {}

boolean isAccountNonExpired() {
return true
}

def setAccountNonExpired(boolean acctNonExpired) {}

boolean isEnabled() {
return true
}

def setEnabled(boolean enabled) {}
}

Role.groovy

class Role implements GrantedAuthority {

String authority

static final ANONYMOUS = 'IS_AUTHENTICATED_ANONYMOUSLY'
// Add your roles here so you can reference them, for instance:
static final ADMIN = 'ROLE_ADMIN_USER'

// This list holds all roles, convenient when you're declaring controller methods
// that should be available to everybody (see AccessController for instance).  When
// you add roles to your system, make sure you add them to this list as well.
static final ALL_ROLES = [ANONYMOUS,ADMIN]

int compareTo(Object o) {
if (o instanceof Role) {
return this.authority.compareTo(o.authority)
}
return 0
}

String toString() {
return authority
}
}

Also once in AppEngine it is a little more sensitve to reflection, with the standard plugin I got:

java.lang.IllegalAccessException: Reflection is not allowed on private int java.util.ArrayList.size

This was down to line 217 of the StarkSecurityGrailsPlugin.groovy having:

if (authFilters.size < 1) {

When it should have been

if (authFilters.size() < 1) {

I opened a JIRA for this one.  Also you might find that plugins embed javascript (such as the Grails UI plugin),  in which case you will want to change the StarkSecurityConfig.groovy to allow access to these files:

authorizations = [
'/': Role.ALL_ROLES,
'/js/**': Role.ALL_ROLES,
'/css/**': Role.ALL_ROLES,
'/images/**': Role.ALL_ROLES,
'/plugins/**': Role.ALL_ROLES,
'/j_acegi_logout': Role.ALL_ROLES
]

Another problem was changing the log4j not to write to the file system (you’ll get a restricted class exception). This is done with:

'null' name:'stacktrace'

While building the system up,  I also discovered the problems of not setting the seralizationUIDs on classes that might end up in session (ie. Role),  make sure you set it on the class or you will need to flush you local cookies after deployment.

Also I found a nasty little problem in the formRemote tag,  basically it would operate in the development server but not on AppEngine itself,  in the end I dug around I found that the JavascriptTagLib.groovy made some assumptions about a variable being a map which it wasn’t (in fact it was a String in this case).  Due to Groovy’s ability to reflect private values (which just feels dangerous),  you would end up with:

<span>[deve11/5.334872088869876828]</span>.<stderr>: java.lang.SecurityException: java.lang.IllegalAccessException: Reflection is not allowed on private final char[] java.lang.String.value

I ended up having to pull the JavascriptTagLib.groovy into the project and make the fix there, and opened another JIRA in grails for it.

So there it is – a brief little walkthrough getting Grails operating in Google AppEngine. GAE/J feels a little clunky at times and it probably does suffer from being a little rushed out of the door.  However,  after you start to get your head around the restrictions it does start to become a good platform.  Over time I’m sure we are going to see further improvements in the Google Platform,  and while I am still working with the Amazon style Infrastructure as s Service world starting to look more closely at Platform as a Service has opened my eyes to some of its benefits.  The days of managing your database and server could soon be gone – though I think it will be a specific type of application that benefits most.

Grab the HBase POM here (not official) and then set HBASE_HOME and run the deploy_deps.sh script.

Hope it helps :)

Over the past 10 years I have certainly been in and out of the field working with a number of large banks and also with vendors such as Goldensource.  Back in 2003 I started a small SOA-based data management platform with some business partners and while I went off to live in the world of open source for a couple of years then continued forward.  Now with my time split between data management and cloud technologies I have started to look to how they can be brought together to build something different and new.

In the past the concept was that an organization would choose a single source of information and map it to a fixed model that would represent the information they are holding.  However,  with the range of data that is available now many organizations are consuming multiple sources of information and trying to build what represents a golden copy.  This tended to be a painful process,  usually managed mainly by technology and has traditionally resulted in large scale implementation of complex products that attempt to provide mapping and rule technology to allow the creation of a single representation.  Often times this has taken too long to meet the requirements and ended up too inflexible to meet the needs of internal data consumers, who’s view of the information and needs can vary wildly.

Changing views tend now to look about building more than one golden representation depending on the needs of the internal consumer,  and also now look at how ‘source tagging’ information can allow for a more defined path between the internal consumer and the external information.  The principal change here has been that information entering the enterprise is usually nothing more than a collection of facts that need to be brought together to give meeting,  what this is starting to mean is that rather than placing the effort at the beginning of the process with mapping to a golden representation (or model) we see simply storing the facts and then materializing a view of the information is more critical.  Also we see that consumers need more flexibility in the being able to decide what source they want – with more complex patterns of determination by comparison and fuzzy matching.  Through all of these we need to never lose sight of the tight connection between the source of the information as it entered the enterprise and ultimately the link to the end consumer within the organization.

So what does this mean for the future of Enterprise Data Management,  well I believe why the concept of a golden structure and understanding of it is still critical the actual underpinning technology will probably move away from a fixed relational model.  The need to store and track the bitesize facts of knowlegde from multiple sources will lead to looking at different approaches.  Also I believe that we will be looking at a move away from the weight of BPEL style rules processing to match and identify relationships to a convergence between the representation of the data and how elements of it – or relationships – are defined.  This approach would allow a consumer to not only map source information that they need into the structure (which would be extensible by the consumer) but also define how they might see relationships or source choices based on different criteria.  This is an important point since many EDM projects first need to become the single source of information and at the same time reconstruct existing representations and relationships to service existing consumers.

The technologies for scaling and building such solutions has come a long way in the last 6 years and now we see more credence given to key/value databases and more dynamic storage engines,  there is no doubt in my mind that as we move into the next stage of data management (driven in no small part by the needs to control the vast amount of information within the financial services sector) we will see more products looking to leverage these types of storage.  Finally,  in the past 18 months I have been working with a mix of cloud technologies on various projects and the ability to start understanding how we can look to not only host but also process these complex processes in a highly scalable and elastic environment will be the cornerstone of making much of this new generation of solution feasible and available.

While the other project – The Cloud Market – has marked a return to Ruby on Rails for me.  Took a little bit of time to balance switching daily between the two – i have to admit I enjoyed the challenge.  Its also good to be working with some great people on the project and certainly a different world to the Enterprise Data Management space.

Right now we are still in a closed beta (and looking for interested parties) – however the aim’s of the site are quite simple.

• To provide a simple way to find Amazon images that suit your needs
• A way for public image owners to produce a shop front for their offerings
• A scanner technology that allows people to request a ‘scan’ of a machine image,  which details information such as the installs RPM’s,  available space and more

If you are interested in getting involved drop me a line (philip.dodds at me dot com), sign up at the site or follow @thecloudmarket on twitter.  We are hoping to be live in the coming weeks and continuing to add some new and interesting features as we go.

Anyone in Charlotte should get down and meet up the Charlotte JUG for the December Drinks.   Also I have been messed around far too much with themes – though hopefully that will settle down soon.  And also we will have a big update on the BarCamp Charlotte.

First of all I wanted to put my Rails application into a Maven like directory structure so I created /src/main/rails and put the app in there,  then I knocked up a pretty standard POM for a WAR file.  Next up I wanted to Warble the project so I first up I installed the gem.

pdodds:(svn)trunk[trunk:4184]/src/main/rails$ sudo gem install warbler
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed warbler-0.9.11
1 gem installed
Installing ri documentation for warbler-0.9.11...
Installing RDoc documentation for warbler-0.9.11...
pdodds:(svn)trunk[trunk:4184]/src/main/rails$

Then I pluginize the Warble so that it will be available in my project using warble pluginize.

Next up I wanted to get my Maven pom sorted out – the quickest way I could find to get Warble integrated was through the jruby-rake-plugin. Adding the following plugin to my pom.xml ensures that the gems that I need are installed by the jruby plugin and then warble can be called in the compile phase.

<plugin>
	<groupId>org.jruby.plugins</groupId>
	<artifactId>jruby-rake-plugin</artifactId>
	<version>1.1.4</version>
	<executions>
		<execution>
			<id>install-gems</id>
			<phase>process-resources</phase>
				<goals>
					<goal>install-gems</goal>
				</goals>
				<configuration>
					<gems>warbler,rails</gems>
				</configuration>
		</execution>
		<execution>
				<id>build-war</id>
				<phase>compile</phase>
				<goals>
					<goal>rake</goal>
				</goals>
				<configuration>
					<launchDirectory>${basedir}/src/main/rails
					</launchDirectory>
			    	        <args>war:clean war</args>
				</configuration>
	           </execution>
	</executions>
</plugin>

In the first instance of the execution of the jruby plugin we pull down and install the gems that we are going to use, in the second we actually call the Warble goals in the project to create the WAR file.

In order to quickly integrate this process with the standard WAR building in Maven we need to configure the Maven WAR plugin, you could also change the Warble output but in the end I found this way worked when I wanted to use the Jetty plugin.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-war-plugin</artifactId>
	<configuration>
		<webappDirectory>${basedir}/src/main/rails/tmp</webappDirectory>
        </configuration>
</plugin>

While this is pretty brief hopefully it helps others get going and I’m very open to alternate or better ways to achieve the same goal.