I can’t really go to much into the functionality of the app,  however I thought I would share some of the experiences I’ve had building the application in the Google environment with Grails.  First off,  if you are going to be doing any serious work with Grails then you’ll need to get a copy of Intellij,  the experience developing with it is much better than others I tried (and improving even more in the 9.0M1 release).  The initial cut of the application was written using GORM and against a local database,  however after trying GORM-JPA (it is very early still though moving quickly), I decided that trying to take a RDBMS mapping over to AppEngine datastore was not going to be a productive route.  The first real lesson of AppEngine is understanding the data store,  it is strange but several of the applications I have worked on over the past 4-5 years have started to move away from traditional RDBMS.  I’m not about to get into the NoSQL argument, other than to say ultimately you need to sit down and understand the tools that are available.  AppEngine DataStore is a powerful storage engine (I’m purposefully not calling it a database),  though it does take a little getting used to.  If you are planning to take a look I would start with the recent Google IO presentation by Max Ross.

Since I have been working with HBase lately some of the concepts weren’t so alien, however it I did find that using JPA annotations actually lead me into incorrect thinking at times,  while adding the annotations I was immediately thinking in the RDBMS mapping terms.  Since I wanted to use the JPA annotations, but didn’t feel that GORM-JPA was quite ready I switched over and used the Grails JPA plugin,  in some ways this meant creating a DAO to allow me to interact with the store.  In the end I did this through the creation of a service which I used in the controllers,  it was pretty generic thanks to the nature of groovy – in the end I can see myself probably moving back to GORM later on,  though in the meantime having a closer relationship with the JPA interface allowed me to better understand how relationships were being managed by AppEngine.  GORM has a tendency to abstract you a little to much at times and coming to terms with a new type of storage engine means that abstraction leads to another level of indirection.  I can see that as time passes and the ins and outs of AppEngine Datastore become more like second nature then that indirection is something that is easy to handle,  though at the beginning I think it is easier to get used to the datastore without it there.

Plugins is one of the powerful aspects of Grails, however one of the problems you encounter in the AppEngine world is the file limit (currently 3,000). While adding a lot of UI plugins speeds up development you can quickly find that you have blown the file limit. Also make sure you watch the $USER_HOME/grails/1.1.1/projects/{yourproject}/stage directory, if you start removing things from your application (ie. unneeded JS) then you might well find that they are remaining in the stage directory.

If you are using YUI you can switch to their servers, so that you don’t have to hold all the YUI files,  first add the following to your BootStrap:

def init = {servletContext ->
JavascriptTagLib.LIBRARY_MAPPINGS.yui = []
}

Then in your main.gsp make sure you reference the CSS and JavaScript you need.

If you are using the grails-ui plugin 1.0.4 it is currently using the javax.rmi.UID class to generate UUIDs, this class is blacklisted by GAE/J and therefore you would need to change the GrailsUITagLibService.groovy and remove the import.  I found that I was able to use the UUID class to get a similar effect (see JIRA)

def getUniqueId = { 'gui_' + DigestUtils.md5Hex(UUID.randomUUID().toString()) }

Once you have the basics in place, then you will need security. Obviously Google provides API’s to allow this to exist, however often you need more complex security than is currently provided by the Google API. In Grails I naturally jumped for the Aecgi plugin, however after a few battles with its close connection with Hibernate (I get the feeling that plugin does a little too much and would be well served by being broken up into a several plugins rather than a lot of configuration in one plugin). Next up was Stark Security plugin – I have to admit it was a little new to me, however I was able to quickly implement this in grails and also get it functioning in GAE/J. Basically I needed to create the basic DAO stuff using the plugin then change it to use JPA methods (as seen below):

User.groovy

import org.datanucleus.jpa.annotations.Extension
import org.springframework.security.GrantedAuthority
import org.springframework.security.userdetails.UserDetails
import javax.persistence.*

/**
* This class is the default UserDetails implementation for the Stark Security plugin.
* Since instances of this class will be used by the underlying Spring Security framework across
* Hibernate sessions, we can't have any lazy loading in here (see special handling of roles below).
*/
@Entity
class User implements UserDetails {

@Id
@GeneratedValue (strategy = GenerationType.IDENTITY)
@Extension (vendorName = "datanucleus", key = "gae.encoded-pk", value = "true")
String id

@Column
String username

@Column
String password

List<String> roles = new ArrayList<String>();

GrantedAuthority[] getAuthorities() {
return roles.collect { new Role(authority:it) } as GrantedAuthority[]
}

boolean isAccountNonLocked() {
return true
}

def setAccountNonLocked(boolean nonLocked) {}

boolean isCredentialsNonExpired() {
return true
}

def setCredentialsNonExpired(boolean nonExpired) {}

boolean isAccountNonExpired() {
return true
}

def setAccountNonExpired(boolean acctNonExpired) {}

boolean isEnabled() {
return true
}

def setEnabled(boolean enabled) {}
}

Role.groovy

class Role implements GrantedAuthority {

String authority

static final ANONYMOUS = 'IS_AUTHENTICATED_ANONYMOUSLY'
// Add your roles here so you can reference them, for instance:
static final ADMIN = 'ROLE_ADMIN_USER'

// This list holds all roles, convenient when you're declaring controller methods
// that should be available to everybody (see AccessController for instance).  When
// you add roles to your system, make sure you add them to this list as well.
static final ALL_ROLES = [ANONYMOUS,ADMIN]

int compareTo(Object o) {
if (o instanceof Role) {
return this.authority.compareTo(o.authority)
}
return 0
}

String toString() {
return authority
}
}

Also once in AppEngine it is a little more sensitve to reflection, with the standard plugin I got:

java.lang.IllegalAccessException: Reflection is not allowed on private int java.util.ArrayList.size

This was down to line 217 of the StarkSecurityGrailsPlugin.groovy having:

if (authFilters.size < 1) {

When it should have been

if (authFilters.size() < 1) {

I opened a JIRA for this one.  Also you might find that plugins embed javascript (such as the Grails UI plugin),  in which case you will want to change the StarkSecurityConfig.groovy to allow access to these files:

authorizations = [
'/': Role.ALL_ROLES,
'/js/**': Role.ALL_ROLES,
'/css/**': Role.ALL_ROLES,
'/images/**': Role.ALL_ROLES,
'/plugins/**': Role.ALL_ROLES,
'/j_acegi_logout': Role.ALL_ROLES
]

Another problem was changing the log4j not to write to the file system (you’ll get a restricted class exception). This is done with:

'null' name:'stacktrace'

While building the system up,  I also discovered the problems of not setting the seralizationUIDs on classes that might end up in session (ie. Role),  make sure you set it on the class or you will need to flush you local cookies after deployment.

Also I found a nasty little problem in the formRemote tag,  basically it would operate in the development server but not on AppEngine itself,  in the end I dug around I found that the JavascriptTagLib.groovy made some assumptions about a variable being a map which it wasn’t (in fact it was a String in this case).  Due to Groovy’s ability to reflect private values (which just feels dangerous),  you would end up with:

<span>[deve11/5.334872088869876828]</span>.<stderr>: java.lang.SecurityException: java.lang.IllegalAccessException: Reflection is not allowed on private final char[] java.lang.String.value

I ended up having to pull the JavascriptTagLib.groovy into the project and make the fix there, and opened another JIRA in grails for it.

So there it is – a brief little walkthrough getting Grails operating in Google AppEngine. GAE/J feels a little clunky at times and it probably does suffer from being a little rushed out of the door.  However,  after you start to get your head around the restrictions it does start to become a good platform.  Over time I’m sure we are going to see further improvements in the Google Platform,  and while I am still working with the Amazon style Infrastructure as s Service world starting to look more closely at Platform as a Service has opened my eyes to some of its benefits.  The days of managing your database and server could soon be gone – though I think it will be a specific type of application that benefits most.

Over the past 10 years I have certainly been in and out of the field working with a number of large banks and also with vendors such as Goldensource.  Back in 2003 I started a small SOA-based data management platform with some business partners and while I went off to live in the world of open source for a couple of years then continued forward.  Now with my time split between data management and cloud technologies I have started to look to how they can be brought together to build something different and new.

In the past the concept was that an organization would choose a single source of information and map it to a fixed model that would represent the information they are holding.  However,  with the range of data that is available now many organizations are consuming multiple sources of information and trying to build what represents a golden copy.  This tended to be a painful process,  usually managed mainly by technology and has traditionally resulted in large scale implementation of complex products that attempt to provide mapping and rule technology to allow the creation of a single representation.  Often times this has taken too long to meet the requirements and ended up too inflexible to meet the needs of internal data consumers, who’s view of the information and needs can vary wildly.

Changing views tend now to look about building more than one golden representation depending on the needs of the internal consumer,  and also now look at how ‘source tagging’ information can allow for a more defined path between the internal consumer and the external information.  The principal change here has been that information entering the enterprise is usually nothing more than a collection of facts that need to be brought together to give meeting,  what this is starting to mean is that rather than placing the effort at the beginning of the process with mapping to a golden representation (or model) we see simply storing the facts and then materializing a view of the information is more critical.  Also we see that consumers need more flexibility in the being able to decide what source they want – with more complex patterns of determination by comparison and fuzzy matching.  Through all of these we need to never lose sight of the tight connection between the source of the information as it entered the enterprise and ultimately the link to the end consumer within the organization.

So what does this mean for the future of Enterprise Data Management,  well I believe why the concept of a golden structure and understanding of it is still critical the actual underpinning technology will probably move away from a fixed relational model.  The need to store and track the bitesize facts of knowlegde from multiple sources will lead to looking at different approaches.  Also I believe that we will be looking at a move away from the weight of BPEL style rules processing to match and identify relationships to a convergence between the representation of the data and how elements of it – or relationships – are defined.  This approach would allow a consumer to not only map source information that they need into the structure (which would be extensible by the consumer) but also define how they might see relationships or source choices based on different criteria.  This is an important point since many EDM projects first need to become the single source of information and at the same time reconstruct existing representations and relationships to service existing consumers.

The technologies for scaling and building such solutions has come a long way in the last 6 years and now we see more credence given to key/value databases and more dynamic storage engines,  there is no doubt in my mind that as we move into the next stage of data management (driven in no small part by the needs to control the vast amount of information within the financial services sector) we will see more products looking to leverage these types of storage.  Finally,  in the past 18 months I have been working with a mix of cloud technologies on various projects and the ability to start understanding how we can look to not only host but also process these complex processes in a highly scalable and elastic environment will be the cornerstone of making much of this new generation of solution feasible and available.

While the other project – The Cloud Market – has marked a return to Ruby on Rails for me.  Took a little bit of time to balance switching daily between the two – i have to admit I enjoyed the challenge.  Its also good to be working with some great people on the project and certainly a different world to the Enterprise Data Management space.

Right now we are still in a closed beta (and looking for interested parties) – however the aim’s of the site are quite simple.

• To provide a simple way to find Amazon images that suit your needs
• A way for public image owners to produce a shop front for their offerings
• A scanner technology that allows people to request a ‘scan’ of a machine image,  which details information such as the installs RPM’s,  available space and more

If you are interested in getting involved drop me a line (philip.dodds at me dot com), sign up at the site or follow @thecloudmarket on twitter.  We are hoping to be live in the coming weeks and continuing to add some new and interesting features as we go.

So when I saw on Justin Kestelyn twitter than Oracle now has a Cloud Computing Center I had to give it a go.  After only playing for a very short while I have to say I really liked the experience from the AMI’s,   I thought the easiest way to tell you about it was just to give some basic instructions on getting going.  First I’m going to assume you have an AWS account and that you have also set-up the EC2 command line tools to get you going (if you are having trouble with the command line tools I find the tools at RightScale just rock for managing the servers).

So to get started I kicked up a terminal on my trusty mac and listed down the available images.

pdodds:/usr/local/ec2/bin$ ./ec2-describe-images -a | grep oracle-corporation
IMAGE	ami-02cb2f6b	/oracle-corporation/database-ami/64-bit/oracle_11106_SE_SE1_64Bit-image.manifest.xml	725966715235	available	public		x86_64	machine	aki-0d9f7b64	ari-369f7b5f
IMAGE	ami-7acb2f13	oracle-corporation/database-ami/32-bit/oracle_10g_XE_32Bit-image.manifest.xml	725966715235	available	public		i386	machine	aki-c5e703ac	ari-c2e703ab
IMAGE	ami-cecb2fa7	oracle-corporation/database-ami/32-bit/oracle_11106_EE_32Bit-image.manifest.xml	725966715235	available	public		i386	machine	aki-c5e703ac	ari-c2e703ab
IMAGE	ami-7ecb2f17	oracle-corporation/database-ami/64-bit/oracle_11106_EE_64Bit-image.manifest.xml	725966715235	available	public		x86_64	machine	aki-0d9f7b64	ari-369f7b5f
IMAGE	aki-c5e703ac	oracle-corporation/oracle_linux_kernels/32_bit/2.6.18-53.1.13.9.2.el5xen/vmlinuz-2.6.18-53.1.13.9.2.el5xen.manifest.xml	725966715235	available	public		i386	kernel
IMAGE	ari-c2e703ab	oracle-corporation/oracle_linux_ramdisks/32_bit/2.6.18-53.1.13.9.2.el5xen/initrd-2.6.18-53.1.13.9.2.el5xen.img.manifest.xml	725966715235	available	public	i386	ramdisk

Since I just wanted a little instance to play with I picked up the /oracle_11106_EE_32Bit, and started that, note that I had already created a default security group (with port 22 access) and a default keypair which I had on my machine.

pdodds:/usr/local/ec2/bin$ ./ec2-run-instances -k primary-keypair -g default ami-cecb2fa7
RESERVATION	r-2ed30047	207015290718	default
INSTANCE	i-0b842462	ami-cecb2fa7			pending	primary-keypair	0	m1.small	2008-09-22T21:14:17+0000	us-east-1a	aki-c5e703ac	ari-c2e703ab
pdodds:/usr/local/ec2/bin$

Next up I wanted to stick the database on an EBS volume to I decided to create one of those while I’m here (and waiting for that server to start). Note that since I was a bit lazy on setting the availability zone I wanted for the server I’ll be a little more disciplined on the volume (note that volumes need to be in the same availability zone as the server).

pdodds:/usr/local/ec2/bin$ ./ec2-create-volume -s 5 -z us-east-1a
VOLUME	vol-1104e178	5		us-east-1a	creating	2008-09-22T21:17:14+0000
pdodds:/usr/local/ec2/bin$

Now if all is well I should be able to check back on my EC2 instance to see if it is available.

pdodds:/usr/local/ec2/bin$ ./ec2-describe-instances i-0b842462
RESERVATION	r-2ed30047	207015290718	default
INSTANCE	i-0b842462	ami-cecb2fa7	ec2-75-101-230-110.compute-1.amazonaws.com	domU-12-31-39-00-E0-07.compute-1.internal	running	primary-keypair	0		m1.small	2008-09-22T21:14:17+0000	us-east-1a	aki-c5e703ac	ari-c2e703ab
pdodds:/usr/local/ec2/bin$

Once it is available, as above then we can attach the EBS volume that we created.

pdodds:/usr/local/ec2/bin$ ./ec2-attach-volume vol-1104e178 -i i-0b842462 -d /dev/sdh
ATTACHMENT	vol-1104e178	i-0b842462	/dev/sdh	attaching	2008-09-22T21:24:02+0000
pdodds:/usr/local/ec2/bin$

While that is getting hooked up, we can now log into the new server – with simple ssh -i primary-keypair root@ec2-75-101-230-110.compute-1.amazonaws.com.

At this point we accept the license terms and then set the new password, though I would recommend switching off password based connections to the server (for anyone). When we get to the “Would you like to create a database now [y|n]:” lets say no first time. Since we still need to get that EBS drive ready for use. This will drop you back to the root login, though it is easy to get back to the script.

$ mkfs /dev/sdh
mke2fs 1.39 (29-May-2006)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y

I’m probably far to lazy, but this will just get me a drive ready for use, and then its back to the Oracle script, this time we select our new device for our database to be stored, and also the flash recovery area.

/home/oracle/scripts/run_dbca.sh
The Oracle Database Software (ORACLE_HOME) has been installed at
/u01/app/oracle/product/11.1.0/db_1.
Before you can use the Oracle Software, you will need to create a database.

Would you like to create a database now [y|n]:  y

You chose to create a database now.  Before we begin, please provide the following details: 

Select a disk device to store your datafiles and redo logs on:

ID  DEVICE_NAME
--- -----------
[1] /dev/sda2
[2] /dev/sdh 

Type in the ID from the list above [DEFAULT: 1]:
2
Enter the mount point you would like this device mounted as [DEFAULT:  /u02]:

You have selected to mount /dev/sdh on to /u02, is this correct [y|n] [DEFAULT:  n]:
y

Select a disk device to store your flash recovery area files on:

ID  DEVICE_NAME
--- -----------
[1] /dev/sda2
[2] /dev/sdh 

Type in the ID from the list above [DEFAULT: 1]:
2
Using mount point /u02

You have selected to mount /dev/sdh on to /u02, is this correct [y|n] [DEFAULT:  n]:
y

Checking disk devices and mount points for Datafiles and Logfiles
Checking if directory /u02 exists...

Creating directory /u02
Trying to mount /dev/sdh on /u02
DBFILES:  /u02/oradata
FRA:  /u02/flash_recovery_area

Please enter the name for your Oracle Database.
This name will be used as your
ORACLE SID (System Identifier):

From here on in its a simple case of providing a name for the database and all the passwords for the Oracle users, and away it goes. After just a few minutes you will have your database ready to roll, the files stored on EBS and running against the Oracle AMI. It will have started the Enterprise Manager web server etc (though to be honest I kinda wish that was optional since I don’t tend to use it a whole lot).

All in all I have to say the experience was smooth and quick, making Oracle suddenly a lot more welcoming in a cloud environment and I’m sure with Oracle World in full swing we’ll be hearing a lot more about it.

And as always once you are done its nice and easy to clean up…

pdodds:/usr/local/ec2/bin$ ./ec2-terminate-instances i-0b842462
INSTANCE	i-0b842462	running	shutting-down
pdodds:/usr/local/ec2/bin$ ./ec2-detach-volume vol-1104e178
ATTACHMENT	vol-1104e178	i-0b842462	/dev/sdh	detaching	2008-09-22T21:24:02+0000
pdodds:/usr/local/ec2/bin$ ./ec2-delete-volume vol-1104e178
VOLUME	vol-1104e178
pdodds:/usr/local/ec2/bin$