I have been working in the past 10 months in the Enterprise Data Management space to target both existing technologies and also looking at where the space is headed. While EDM covers a huge range of business domains I’ve been specifically focused on the management of reference data in financial services companies. The recent events in the world have left a lot of questions open to how to manage much of the information that underpins the way in which the market manages risk, matching and more.
Over the past 10 years I have certainly been in and out of the field working with a number of large banks and also with vendors such as Goldensource. Back in 2003 I started a small SOA-based data management platform with some business partners and while I went off to live in the world of open source for a couple of years then continued forward. Now with my time split between data management and cloud technologies I have started to look to how they can be brought together to build something different and new.
In the past the concept was that an organization would choose a single source of information and map it to a fixed model that would represent the information they are holding. However, with the range of data that is available now many organizations are consuming multiple sources of information and trying to build what represents a golden copy. This tended to be a painful process, usually managed mainly by technology and has traditionally resulted in large scale implementation of complex products that attempt to provide mapping and rule technology to allow the creation of a single representation. Often times this has taken too long to meet the requirements and ended up too inflexible to meet the needs of internal data consumers, who’s view of the information and needs can vary wildly.
Changing views tend now to look about building more than one golden representation depending on the needs of the internal consumer, and also now look at how ‘source tagging’ information can allow for a more defined path between the internal consumer and the external information. The principal change here has been that information entering the enterprise is usually nothing more than a collection of facts that need to be brought together to give meeting, what this is starting to mean is that rather than placing the effort at the beginning of the process with mapping to a golden representation (or model) we see simply storing the facts and then materializing a view of the information is more critical. Also we see that consumers need more flexibility in the being able to decide what source they want – with more complex patterns of determination by comparison and fuzzy matching. Through all of these we need to never lose sight of the tight connection between the source of the information as it entered the enterprise and ultimately the link to the end consumer within the organization.
So what does this mean for the future of Enterprise Data Management, well I believe why the concept of a golden structure and understanding of it is still critical the actual underpinning technology will probably move away from a fixed relational model. The need to store and track the bitesize facts of knowlegde from multiple sources will lead to looking at different approaches. Also I believe that we will be looking at a move away from the weight of BPEL style rules processing to match and identify relationships to a convergence between the representation of the data and how elements of it – or relationships – are defined. This approach would allow a consumer to not only map source information that they need into the structure (which would be extensible by the consumer) but also define how they might see relationships or source choices based on different criteria. This is an important point since many EDM projects first need to become the single source of information and at the same time reconstruct existing representations and relationships to service existing consumers.
The technologies for scaling and building such solutions has come a long way in the last 6 years and now we see more credence given to key/value databases and more dynamic storage engines, there is no doubt in my mind that as we move into the next stage of data management (driven in no small part by the needs to control the vast amount of information within the financial services sector) we will see more products looking to leverage these types of storage. Finally, in the past 18 months I have been working with a mix of cloud technologies on various projects and the ability to start understanding how we can look to not only host but also process these complex processes in a highly scalable and elastic environment will be the cornerstone of making much of this new generation of solution feasible and available.
