Enterprise Data Management, some thoughts

We need more rigor - thank you Stephen Few

Please read the following posts: 

1) - Stephen Few takes on Forrester Research

2) - Neil Raden's response to the rumble

3) - Wouter's heartfelt response 

And finally my response:

I think my tweet initially was that ‘the establishment was not happy’. And they were not – Stephen got burned big time and it was shamefull. I hold Stephen in high regard – he really knows his stuff, wrote some classic books etc..

He took on Forrester head on when they wrote about his expertise.The response of Forrester was simply something like ‘I have 30 year experience, how dare you…’. He was in his right to respond. And please, if you tend to disagree with someone it is hard to distinguish between the person and the sh*t he’s writing. So Stephen went for the ball and took the person with him…but in that order.

In the analyst commmunity nearly ALL took side…the side of forrester (source:twitter). For me, that was quite a shock. That ‘thought leaders’ took the side of the big analyst firm and did not went for the real arguments was quite a shock to me…..

Two exceptions; Mark Madsen and Neil Raden, where Neil Raden wrote the above post. I bow to these giants

Sharing data; the story continues

Stumbled on a blog post of Evan Levy from Baseline, posted a comment. 

 So happy to meet a kindred spirit.

Tied selling......let's take a stand

Lately I come across vendors of OLTP systems that engage highly in tied selling. Now - if there is complete transparency to the customer, this is off course no problem. But, the sad thing is, this transparency is usually extremely bad or even non existent. 

This tied selling combined with (complete) lack of transparency has caused severe damage to the economy as we have seen especially in the financial sector. In Europe the European Commission is highly active in forcing vendors to stop tied selling (for example Internet Explorer and the OS of Microsoft). Vendors receives huge fines if they engage in these practices.

Unfortunately these practices are also quite mainstream in the data exploitation industry.

Suppose I am a customer and I am buying software to support my primary processes. This software is implemented and I eventually get it working. Now I want management information and since I am quite a large enterprise with several OLTP (legacy) systems I decide to go for a Enterprise Data Warehouse solution build according to more or less open standards. 

Off course I go to my software vendor and I am asking him to supply the data.....not a weird question because it's MY data.....

Now the vendor is saying; 'my system is extremely complex, business rules are fired off through a logical layer, it's very complex to interpret the data and (I hear this one all the time) we got a system that is metamodelled (they probably mean Row=Value kind of stuff). And NO....you are not allowed to get the underlying datamodel; that is proprietary'. 

But - the software vendor is offering me a brilliant off-the-shelf-super-dooper-it-can-do-it-all-and-I-only-have-to-go-through-wizard-solution; you can buy our management information software.........

So - to summarize: I bought an OLTP system to support my business processes, it's my data that is registered. But it turns out that I can't access it. I can not get my data without buying ADDITIONAL software and off course I need their consultants to implement the stuff and to make it fit for my needs (and that's off course is the next tied selling stuff; to fit the management information to my needs), I need to buy support fees, I need to buy next versions...and even worse....I might be obliged to buy all kinds of other software (broker, workflow, reporting, analysis, dashboarding etc..) to get required functionality. 

Now, if the vendor gave full disclosure of this tied selling stuff - BEFORE the product was actually purchased - there is no problem. But, if they don't....well, then we need to take a stand.

Two lessons:
- The customer needs to stand more firm in selecting software that is able to extract it's data to other targets. Customers need to add these criteria in selecting OLTP systems. You, my dear customer, OWN your data. It just can not be true, that you are not able to access it without buying additional stuff.

- The customer needs to take a stand to vendors that engage in non-transparant tied selling. It's just not acceptable as a means of doing business.

Data is and will always be the ultimate proprietary asset

Yep - bringing up an oldie, but still o so true.

Financial assets: A single Euro can only be spend once.....

Human captial: They can just walk out of the door and leave or be hunted by your competitor

Not so with your data; It is uniquely yours, closely connected to your business language, full of nuance, in the proper context and the ability to procreate new data. Every organization can use it's data differently to create it's own niche. When data is used by anyone in your organization, it will not be consumed and it will not expire (a patent will). There is simply no other asset that anyone can think one that owns these capabilities.

To make the case even stronger; the movement of data in your organization is your lifeblood. Any company will simply seize to exist if this data is of bad quality. Orders will not get through, customers credit information can not be seen and invoices can not be send.

Another fascinating attribute of data is that it defines the other assets; financial data, property data, employee data. Badly managed data will jeopardize the management of the other assets big time. So, it's kind of a 'super', ultimate asset.

Nicholas Carr wrote in 2003 in the Harvard Business Review 'IT doesn't matter'. Systems, infrastructure, services, hardware, software, ERP...whatever...they will and probably are already commodities. They are not a differentiating factor in competitive advantage, sustainability or reliability.  Data is however always unique, competitors can not get hold of it (well....unless you got a security problem) and if they got it, it will probably be hard to use, since data is often very closely linked to the organizational lingua franca.

Data must be nurtured, managed aggressively and - most of all- must be enabled to be exploited to the max in order to get a good return.It is by far the ultimate proprietary asset.

I believe that in business nowadays huge amounts of money are spend on very sophisticated technology, financial controls and process management  (the process taliban). The marginal returns from ever increasing investments into these developments will be small. Investments in data are quite the opposite. The opportunities are limitless. And, more importantly, any investment you make is proprietary to your organization. This is the true definition of competitive advantage. 

This posted is inspired and sometimes quoted by T.Redman and his book 'Data Driven'

Data Vault in the Netherlands

July 2007 I had a first aquaintance with Dan Linstedt - via LinkedIn. I worked for a large government organisation as the architect for Data Warehousing and Business Intelligence. I seriously considered Data Vault as the standard methodology for the Enterprise Data Warehouse. I worked for over 2 years for this govenment organisation and my choice for the Data Vault is still a very good one.

November 2007 I got Dan Linstedt to the Netherlands and got the first 27 peeps trained and certified of that same government organisation. I had dozens of discussions with architects, management, developers, etc...but in the end the consensus on choosing Data Vault was huge.

Since november 2007 Data Vault sky rocketed in the Netherlands. Together with the Genesee Academy and DNV I think we certified over 200 people and I dare to say that most new EDW projects nowadays in the Netherlands have chosen Data Vault as the prime methodology.

Before I get criticised for sanctifying one methodology despite the various (non) functional requirements; I do believe that ANY architecture needs to be in line with the business-, application- and technical architecture. So choosing one methodology on forehand is just plain wrong. There has to be sound justification and alignment for any choice in your architecture. Whether it's a tool, a methodology or whatever.

Since 2 years we (Genesee Academy, me, DNV) certified over 200 people - most of them extremely seasoned and respected experts in our field. On average these consultants are very critical on any new 'trick in the book'. I dare to say that most of the experts that were certified were convinced that Data Vault has got extreme value and potential. The proof of the pudding is in the eating; Data Vault got many implementations in the Netherlands the last two years and it's growing fast.

I am confident that Data Vault in the years to come will hit Europe as hard as it has hit the Netherlands.

But I wonder......I and am still baffled by it. The attention for Data Vault in the United States is extremely limited, why is that? Even a respected institute as TWDI is not paying much attention. Why? Experts in the field do not mention Data Vault what so ever. I just don't get it. Can anybody enlighten me?

For those of you not familiar with Data Vault --> read my articles .

A response to 'The flaws of the classic data warehouse' (2)

note: this blog post was first published on my B-eye-network blog.

It is only by means of good and respectfull discussion that knowledge and insight will evolve. This post should be regarded as such.

This post is a second reaction to the first article in a series of three which were written by a highly respectfull thoughtleader in the field and publisher on the B-Eye-Network; Rick van der Lans. The papers are titled 'The Flaws of the Classic Data Warehouse Architecture'.

This blog post is a reaction to the first part. It deals with the flaws of the classic data warehouse architecture (CDWA).

Rick signals five flaws which will lead in article two and three to a new architecture. This post is addressing the second flaw.

- My reaction to flaw #1 can be read here.

Flaw 2 according to Rick
The CDWA stores a lot of redundant data. The more redundant the data, the less flexible the architecture is. We could simplify our data warehouse architectures considerably by getting rid of most of the redundant data. Hopefuly, the new database technology on the market, such as data warehouse appliances and column-based database technologies, will decrease the need to store so much redundant data. Rick commented on this flaw in his closing keynote statement on a BI event we had last week, stating basically that the DWH professional did an extremely lousy job last decades in building these redundancy monsters. Like in his article he strengthened this argument by research done by Nigel Pendse claiming that the average BI application only needed a fraction of the stored (redundant) data. 

My reaction to flaw 2
First of all, I agree that new technologies can limit the volume of redundant data considerably.

But to say that in the last decades the data warehouse professional did an etremely lousy job because of the huge redundancy they created in their data warehouses...well, that's just plain stupid and for the people that are applauding this statement I would like to say; 'I bet you never actually build a data warehouse'.

BI populism.....thats what it is.

As for the flexibility argument; more redundant data kills flexibility. Hmm...it's a bit of a bs-argument. Because flexibility is not only affected by redundant data. If I had build my data warehouses in the last decades without redundant data I would have ended up with huge complex transformation rules and a big strain on processing capacity. Both issues woud have killed the flexibility big time and I am leaving aside the degradation of performance, degradation in ease of use, degradation in maintainability and the degradation of the testability of the system. But I agree - I would not have redundant data...I would not have any quality of service either....but who cares.

BI populism.....thats what it is.

But is the CDWA architecture flawed by this redundancy problem? I do not think so at all. We would still need a datastore of some kind (Rick seems to acknowledge that by advocating the use of appliances), we would still have several layers after this datastore, preparing the data for several different functionalities (reporting, mining, advanced analytics, datasharing to third parties, etc.). Let's take the datamart layer, will it dissapear? I don't think so. The question is whether it needs to be materialized. And that's where new technology will be extremely valuable. It seems that Rick is translating the word 'Architecture' with 'Technical Architectue' as a 1:1 relationship.

The hub-spoke architecture of the CDWA model is still extremely valid. Off course, technology within this architecture will evolve and will enable us to deliver an even better quality of service.

A response to 'The flaws of the classic data warehouse' (1)

note: this blog post was first published on my B-eye-network blog.

It is only by means of good and respectfull discussion that  knowledge and insight will evolve. This post should be regarded as such. Furthermore, it is from a good friend from whom I understood that Rick meant to be controversial with these papers.....

This post is a first reaction to the first article in a series of three which were written by a highly respectfull thoughtleader in the field and publisher on the B-Eye-Network; Rick van der Lans. The papers are titled 'The Flaws of the Classic Data Warehouse Architecture'.

This blog post is a reaction to the first part. It deals with the flaws of the classic data warehouse architecture (CDWA) according to Rick. If you wanna know what exactly constitutes a CDWA - I would suggest to read this first part.

Rick signals five flaws which will lead in article two and three to a new architecture. This post is addressing the first flaw. In upcoming postings on this blog I will also adress the other four and I will also respond to the solution he is proposing.

Flaw 1 according to Rick
The CDWA does not support the concept of Operational Business Intelligence. This conclusion is drawn from the fact that the CDWA can not include 100% up-to-date information. Rick concludes that we have to remove storage layers and minimize the copy steps.

My reaction to flaw 1
A metaphor; I am driving my car and suddenly I say 'damn; I wanna fly'. Looking at my car, I can not seem to find the 'fly' button and I therefore conclude that my car  is flawed.

Although a bit of a corny metaphor it reflects the core of my criticism. Aparently there is a new requirement called Operational Business Intelligence* that can not be served by the existing architecture. Is the existing architecture then flawed? I do not think so. Does the existing architecture fit the needs of the organisation? I do not think so. So flaw 1 in my opinion is not a flaw, it might simply be not a good fit between requirement and architecture.

Let's take this corny metaphor one step further. Suppose there is a genuine need for me to fly (e.g 100% up-to-date information for decision-like processes*). Is it then considered common sense to build wings on my car and put in a jet engine? I wouldn't ......I would just buy a plane ticket and get to an airfield or maybe I would use a substitute to achieve my objectives....the train.

To conclude; requirements are evolving and architecture needs to follow. The data warehouse architecture depicted as a hub-spoke model is still valid for it's intented use (although the design is evolving). New requirements can lead to new choices in architecture (and subsequently in design). 

Although I do not agree on the flaw issue, I do agree that new requirements can require new architecture which - in the end - is exactly what Rick is proposing (although I do not agree completely on this new architecture - but lets keep that in mind for a next posting). 

   

* as you can see I am eluding the tedious discussion regarding the term Operational Business Intelligence. I am also eluding the so-called 'fact' that organizations all need 100% up-to-date information for decision like processes.

Disruptive innovation - to be or not to be

note: this blog post was first published on my B-eye-network blog.

Vendors, but also the analysts (and I see a trend...), are increasingly using the term 'disruptive' for new products, new technologies or whatever. And lately it kind of got to me, simply because -  most of the time - there's no basis at all to define something 'disruptive'. It kind of inflates the term...big time.

'So what', I hear you say. Well, there is off course not much of a problem when a vendor defines their own technology or produtc as being 'disruptive'. I know where it comes from and I understand the vendors wish to increase its turnover by claiming to sell a disruptive technology/product/etc..

But when analysts do it, I am getting more suspicious and sometimes extremely annoyed. It is the analyst that needs to be neutral, a bit restrained and off course critical. The analyst needs to put this 'disruptive' stuff a bit in perspective for the reader.

Let's try to get some sort of definition to the word 'disruptive'. So I did some research and ended up with Kalle Lyytinen's paper from 2003 in MIS Quaterly called;

"The disruptive nature of Information Technology Innovations: The Case of Internet Computing in Systems Development Organizations"

In my opinion a very good paper. And by the way; in his study he shows that Internet Computing has radically impacted the IT innovations of firms both in terms of development processes and services. Maybe not at all suprising, but if you compare this type of innovation with (let's take an arbitrary example that is often defined as disruptive*) DW appliances......

Lyytinen defines disruptive innovation as:
They radically deviate from an established trajectory of performance improvement, or redefine what performance means in a given industry (Chistensen and Bower 1996). They are radical (Zaltman et al. 1977) in that they significantly depart from existing alternatives and are shaped by novel, cognitive frames that need to be deployed to make sense of the innovation (Bijker 1987). Consequently, disruptive innovations are truly transformative (Abernathy and Clark 1985). To become widely adopted, disruptive architectural innovations demand provisioning of complementary assets in the form of additional innovations that make the original innovation useful over its diffusion trajectory (Abernathy and Clark 1985; Teece 1986). By doing so, disruptive innovations destroy existing competencies (Schumpeter 1934) and break down existing rules of competition.

Are appliances or new technology for data storage and data management really disruptive? Or are they just the natural flow of continuing innovation. I think the latter.

Let's be cautious in using big words like 'disruptive'.......



 * Did a quick search on 'Disruptive' in B-eye-Network and found 128 hits, most of them appliances or other 'revolutionary' database products/technologie

Data warehousing is failure prone......or is it not?

I often critisize vendors and others for not being thorough enough. Now it's time to critisize science....

In several papers I am reading at the moment (ranging from MIS Quarterly, Information & Management, Decision Support Systems and many more journals) I encounter something similar.

Please read the following quotes:

"....the road to DW success has been littered with failures [43,63,80]"
"....nearly half of all DW initiatives end up as failures [38]"
"According to a press release 2005 by Gartner: through 2007 more then 50% of data warehouse projects will have limited acceptance, or will be outright failures."

MIS is a Mirage

note: this blog post was first published on my B-eye-network blog.

In my quest for sustainable knowledge I am constantly searching for those little pieces of 'gold' in the literature. John Dearen's publication in the Harvard Business Review in 1970 is a great example. It's called MIS is a mirage.

Please remember that this article was written in 1970 - so try to go back in time when you read this article or this blog. It's fun!!

Some Quotes:
"Of all the ridiculous things that have been foisted on the long-suffering executive in the name of science and progress, the real-time management information system is the silliest"

In trying to decribe MIS as a term Dearden writes: "It is difficult to even describe MIS in a satisfactory way because this conceptual entity is embedded in a mish-mash of fuzzy thinking and incomprehensible jargon."

I wonder if I replace the word MIS for BI.....whether John Deardens' remark is still valid.