Enterprise Data Management, some thoughts

The role of the Datamodel in the datawarehouse

For all you Dutch 'listeners' to this post, I wanna direct your attention to a great post from Lidwine van As. A collegea of mine I worked with on various clients the last 10 years and with whom I write various articles at the moment.

It addresses the DB/M Article posted by Karien verhagen about the Data Vault and how a methodology like Data Vault can be totally misunderstood....

http://blog.grey-matter.nl/

Tooling - just a heads up

In the last 2 weeks I have been to BI seminars. Off course I heard a lot of buzzword bull&^*^# or - to put it more polite - I heard a lot of fairytales.

There are however things I wanna be held up to date of. Among others, I find it very important to hear from the vendors how there tooling is evolving. And again, among these sessions you still have vendors that are painting the 'garden of eden'....but let's not repeat myself.

Let me summarize some things:
- Microstrategy  - Still lonely at the top in query and reporting (dashboarding as well). Still lonely at the top in terms of usability and deployment. Their metadata layer is simple something the others can only dream of. Further more, there in-database processing - their SQL generation - is extremely powerfull and tailored (certified) to various (R)DBMS. Their support for AIX 64 bits (I think since version 8) was a huge boost in the large enterprise setting. They are without a doubt alone at the top (Q&R).

- Informatica - again....lonely at the top in Data integration in general and ETL specific. I just love the way they approach the martket. They came from a pure-ETL player and reformed their product and marketing to a full data integration product (synchronisation, ETL, Data quality, metadata management etc.). I just love there concept where the logical layer is seperated from the fysical layer. I also had a peek on their InformaticaOnDemand.com service. Not just a peek - they demonstrated it (Bert Oosterhof did - extremely knowledgable fellow!). As a customer you can login to informaticaOndemand.com and setup a data logistic routine to extract data from comapnies like SalesForce.com and feed the data into your data environment or whereever you want it. This is done without any Informatica software installed within you organisation. It's DataIntegration SAAS and I was impressed. The possibilities can be huge if you let your fanatasy on this one go.....I have to emphasize that I do see a lot of hurdles here that need to be taken; privacy, regulatory rules, firewall issues etc...But....it still has great potential.

- SAS - And again - lonely at the top in....Analytics. Enterprise Guide is simply unique software - there is no equivalent in the market. It is in fact a monopoly. Other BI tooling claim the Analytics position, but they either do not know what ' analytics'  really is or they have not seen the power of SAS in this area with the SAS offering consisting of Enterprise guide, Forecast server, Enterprise miner and text mining abilities.

- SAS - yes again....because they seem to be the only real cross-platform fully equiped - covering nearly all aspects - BI platform. Data integration, Data Quality, an integrated metadata platform, cube technology, Query and reporting, and - as I said - analytics. With cross platform I mean they support Mainframe, Intel platform Unix and Linux flavors. z/linux is still missing, but I respect the fact that every1 needs to make choices. There support for mainframe z-OS gives them again - a monopoly in this area.....
To complete the SAS story - I am very much impressed by their interoperability with 3rd party software that is present in any Enterprise. HP Open View, Tivoli Workload scheduler, their integration with OS-based security (like RACF on mainframe) etc....They really deliver on this on.

There are pros and cons in any platform. SAS needs to work hard on their more ' classic' BI functionality like query and reporting, Dashboards, narrowcasting etc..They seem to lack major functionality here. I also believe firmly that the information maps in SAS are still not very mature. If you compare these with the universes in Business Objects or the packacges in Cognos or the architect layer in MicroStrategy.....they are like 10 years behind.

MicroStrategy, Informatica and SAS......Companies with a vibe....

And....still on their own. They seem to be - in my perspective - able to innovate with a good and steady pace, always watching the market, the requirements and adapting to it.

Maybe the ' eaten' once.....those I did not mention....should be carefull....

Fairy tales

Disclaimer: This is a rand....

The BI fairytale book has several chapters - let me give you a sample:
- Neuro BI
- Ambient BI
- Sentient BI
- Proces driven BI
- BI 2.0
- Pervasive BI
- BI as a Service
- BI and SOA (oh please)
- Mobile BI
- Google BI
- personal BI
- DAAS (data as as service) - it's not a car, it's not a plain, it's a super dooper data solution you can buy on the net- gogogo
- BI tools where you don't need DWH and ETL anymore - BUY IT NOW!!!
- Mashups - the new BI desktop
- BPM integrated with BI (so?)
- BI in the box (oh my god)

And please feel free to add some chapters in this amazing book - to be bought on fairytailesInBI.com

I am getting increasingly frustrated by the people that NEVER - I repeat NEVER - had the pleasure of actually building and implementing a succesfull BI solution. It is hard work - it's guerilla. And the Intelligence that is actually executed and needed in companies is so incredibly not in sync with the fairy tales listed above.Everybody knows - who have been in these kinds of projects - what I am talking about ! The Powerpoint porn (copyright RvdG) shown by the so-called analyst (and some vendors as well) are promising a lot. And of course - when it actually needs to be build...they are far gone. Spreading the word somewhere else......

They are never held accounted for....nooo....when the project is finished the client is totally frustrated with his project team because it took more euros than he thought, the data quality is bad, the time it took to implement was longer than he expected and his dashboard does not flash like the dashboard he read about in a magazine......

This is killing our trade. There is a huge need for more professionalism in our trade. Executing solid data management and BI projects - business case and requirements driven. Promising realistic results and making companies aware (don't go rushing in selling your software or hardware - make them aware!!!!!) of their hidden data and information potential in terms of competiveness and/or effectiveness.

Stop the fairytales......I urge you all...join me in the 'alliance-against-BI-fairytales'

A 'KINSTEDT' Architecture?

Sometimes you just need to look differently to something you always took for granted.....

Look at the picture I posted stated all layers of data transport/storage in a new generation EDW.

Remember; in the Central warehouse we only have factual data, no business rules involved here. Business Rules go downstream (after the CDW). The CDW is ofc. Data Vault modelled.

Often the sum of CDW (including a pre-staging), staging out and datamarts is called the Enterprise Data Warehouse.

Now - the idea of a central data warehouse is not new. I mean Bill Inmon was the first to actually write about it in a logical manner in his book the Corporate Information Factory.

Datamarts (often dimensionally modelled) are also not new....Kimball set the standard in his books and practical efforts.  Ok - let's remodel the above picture...

Ok. Remember - with Dan Linstedt Data Vault we store all the data 100% of the time into the CDW....NO BUSINESS RULES. Within Data Vault there is only a very light semantic integration into the Data Vault model (only on the HUBS for the experts).

Let's look at the picture here and let's describe the Central Data Warehouse as a kind of a 'Standardized Source Extender'. You are extending your source into a standardized model that can be loaded in a parallel and uniform way.

So - looking at the CDW as a kind of a Source Extender (remember - this can only be achieved if you use the architectural patterns of Data Vault)...in that sense...that the CDW is part of your operational environment.  Now...what teached Kimball us in his archectural style? Well - you got to stage the data before you move into demand-driven subject oriented datamarts. So before I load my datamarts in a next generation EDW I need to stage SOME data (not all!!) in order to load it into the datamarts.

The goal of the staging-out is however somewhat different compared to a classing staging area. The staging out is used for two reasons; to be able to re-use business rules and store the results of business rules in a historic manner. The second reason is to be efficient in loading datamarts that use similar dimensions or factual data (lets build that once and not every time we load another datamart).

I often hear the complaint from 'experts' that a Data Vault like architecture puts all processing (the hard stuff or the big 'T' as I call it) after the CDW. True - but why is this typical for Data Vault architectures??? Kimball got the same problem....Inmon got the same problem.

In fact; there is a good case to be made stating that a Data Vault Architecture with a Staging Out has got in fact the smallest big 'T' compared to other architectural styles. In the coming months I will focus myself on describing the patterns that can be applied to a Staging-Out area used in a next generation EDW. My next article in DB/M (I will write that in close cooperation with Lidwine van As) will go into this one. This blog will help me go through the thought process of getting it on paper in a logical manner.

My last picture shows my so called KINSTEDT Architecture (Kimball, Inmon, Linstedt). My point is that Kimball, Inmon and Linstedt all deserve their place into the EDW evolution and that when looked at at a different angle......are complementary at some abstract level of thought.

The next evolution in EDW makes sense when you look at this picture....Operational systems and CDW will either merge or the operational system will become leaner and meaner while the CDW part will take care of a lot of the other functions. Maybe - just maybe - the operational system will only hold the data it needs for a certain transaction (hmm....doesn't that sound SOA'ish?). All history etc... will be moved to the CDW part.