Enterprise Data Management, some thoughts

A response to 'The flaws of the classic data warehouse' (2)

note: this blog post was first published on my B-eye-network blog.

It is only by means of good and respectfull discussion that knowledge and insight will evolve. This post should be regarded as such.

This post is a second reaction to the first article in a series of three which were written by a highly respectfull thoughtleader in the field and publisher on the B-Eye-Network; Rick van der Lans. The papers are titled 'The Flaws of the Classic Data Warehouse Architecture'.

This blog post is a reaction to the first part. It deals with the flaws of the classic data warehouse architecture (CDWA).

Rick signals five flaws which will lead in article two and three to a new architecture. This post is addressing the second flaw.

- My reaction to flaw #1 can be read here.

Flaw 2 according to Rick
The CDWA stores a lot of redundant data. The more redundant the data, the less flexible the architecture is. We could simplify our data warehouse architectures considerably by getting rid of most of the redundant data. Hopefuly, the new database technology on the market, such as data warehouse appliances and column-based database technologies, will decrease the need to store so much redundant data. Rick commented on this flaw in his closing keynote statement on a BI event we had last week, stating basically that the DWH professional did an extremely lousy job last decades in building these redundancy monsters. Like in his article he strengthened this argument by research done by Nigel Pendse claiming that the average BI application only needed a fraction of the stored (redundant) data. 

My reaction to flaw 2
First of all, I agree that new technologies can limit the volume of redundant data considerably.

But to say that in the last decades the data warehouse professional did an etremely lousy job because of the huge redundancy they created in their data warehouses...well, that's just plain stupid and for the people that are applauding this statement I would like to say; 'I bet you never actually build a data warehouse'.

BI populism.....thats what it is.

As for the flexibility argument; more redundant data kills flexibility. Hmm...it's a bit of a bs-argument. Because flexibility is not only affected by redundant data. If I had build my data warehouses in the last decades without redundant data I would have ended up with huge complex transformation rules and a big strain on processing capacity. Both issues woud have killed the flexibility big time and I am leaving aside the degradation of performance, degradation in ease of use, degradation in maintainability and the degradation of the testability of the system. But I agree - I would not have redundant data...I would not have any quality of service either....but who cares.

BI populism.....thats what it is.

But is the CDWA architecture flawed by this redundancy problem? I do not think so at all. We would still need a datastore of some kind (Rick seems to acknowledge that by advocating the use of appliances), we would still have several layers after this datastore, preparing the data for several different functionalities (reporting, mining, advanced analytics, datasharing to third parties, etc.). Let's take the datamart layer, will it dissapear? I don't think so. The question is whether it needs to be materialized. And that's where new technology will be extremely valuable. It seems that Rick is translating the word 'Architecture' with 'Technical Architectue' as a 1:1 relationship.

The hub-spoke architecture of the CDWA model is still extremely valid. Off course, technology within this architecture will evolve and will enable us to deliver an even better quality of service.

A response to 'The flaws of the classic data warehouse' (1)

note: this blog post was first published on my B-eye-network blog.

It is only by means of good and respectfull discussion that  knowledge and insight will evolve. This post should be regarded as such. Furthermore, it is from a good friend from whom I understood that Rick meant to be controversial with these papers.....

This post is a first reaction to the first article in a series of three which were written by a highly respectfull thoughtleader in the field and publisher on the B-Eye-Network; Rick van der Lans. The papers are titled 'The Flaws of the Classic Data Warehouse Architecture'.

This blog post is a reaction to the first part. It deals with the flaws of the classic data warehouse architecture (CDWA) according to Rick. If you wanna know what exactly constitutes a CDWA - I would suggest to read this first part.

Rick signals five flaws which will lead in article two and three to a new architecture. This post is addressing the first flaw. In upcoming postings on this blog I will also adress the other four and I will also respond to the solution he is proposing.

Flaw 1 according to Rick
The CDWA does not support the concept of Operational Business Intelligence. This conclusion is drawn from the fact that the CDWA can not include 100% up-to-date information. Rick concludes that we have to remove storage layers and minimize the copy steps.

My reaction to flaw 1
A metaphor; I am driving my car and suddenly I say 'damn; I wanna fly'. Looking at my car, I can not seem to find the 'fly' button and I therefore conclude that my car  is flawed.

Although a bit of a corny metaphor it reflects the core of my criticism. Aparently there is a new requirement called Operational Business Intelligence* that can not be served by the existing architecture. Is the existing architecture then flawed? I do not think so. Does the existing architecture fit the needs of the organisation? I do not think so. So flaw 1 in my opinion is not a flaw, it might simply be not a good fit between requirement and architecture.

Let's take this corny metaphor one step further. Suppose there is a genuine need for me to fly (e.g 100% up-to-date information for decision-like processes*). Is it then considered common sense to build wings on my car and put in a jet engine? I wouldn't ......I would just buy a plane ticket and get to an airfield or maybe I would use a substitute to achieve my objectives....the train.

To conclude; requirements are evolving and architecture needs to follow. The data warehouse architecture depicted as a hub-spoke model is still valid for it's intented use (although the design is evolving). New requirements can lead to new choices in architecture (and subsequently in design). 

Although I do not agree on the flaw issue, I do agree that new requirements can require new architecture which - in the end - is exactly what Rick is proposing (although I do not agree completely on this new architecture - but lets keep that in mind for a next posting). 

   

* as you can see I am eluding the tedious discussion regarding the term Operational Business Intelligence. I am also eluding the so-called 'fact' that organizations all need 100% up-to-date information for decision like processes.

Disruptive innovation - to be or not to be

note: this blog post was first published on my B-eye-network blog.

Vendors, but also the analysts (and I see a trend...), are increasingly using the term 'disruptive' for new products, new technologies or whatever. And lately it kind of got to me, simply because -  most of the time - there's no basis at all to define something 'disruptive'. It kind of inflates the term...big time.

'So what', I hear you say. Well, there is off course not much of a problem when a vendor defines their own technology or produtc as being 'disruptive'. I know where it comes from and I understand the vendors wish to increase its turnover by claiming to sell a disruptive technology/product/etc..

But when analysts do it, I am getting more suspicious and sometimes extremely annoyed. It is the analyst that needs to be neutral, a bit restrained and off course critical. The analyst needs to put this 'disruptive' stuff a bit in perspective for the reader.

Let's try to get some sort of definition to the word 'disruptive'. So I did some research and ended up with Kalle Lyytinen's paper from 2003 in MIS Quaterly called;

"The disruptive nature of Information Technology Innovations: The Case of Internet Computing in Systems Development Organizations"

In my opinion a very good paper. And by the way; in his study he shows that Internet Computing has radically impacted the IT innovations of firms both in terms of development processes and services. Maybe not at all suprising, but if you compare this type of innovation with (let's take an arbitrary example that is often defined as disruptive*) DW appliances......

Lyytinen defines disruptive innovation as:
They radically deviate from an established trajectory of performance improvement, or redefine what performance means in a given industry (Chistensen and Bower 1996). They are radical (Zaltman et al. 1977) in that they significantly depart from existing alternatives and are shaped by novel, cognitive frames that need to be deployed to make sense of the innovation (Bijker 1987). Consequently, disruptive innovations are truly transformative (Abernathy and Clark 1985). To become widely adopted, disruptive architectural innovations demand provisioning of complementary assets in the form of additional innovations that make the original innovation useful over its diffusion trajectory (Abernathy and Clark 1985; Teece 1986). By doing so, disruptive innovations destroy existing competencies (Schumpeter 1934) and break down existing rules of competition.

Are appliances or new technology for data storage and data management really disruptive? Or are they just the natural flow of continuing innovation. I think the latter.

Let's be cautious in using big words like 'disruptive'.......



 * Did a quick search on 'Disruptive' in B-eye-Network and found 128 hits, most of them appliances or other 'revolutionary' database products/technologie

Data warehousing is failure prone......or is it not?

I often critisize vendors and others for not being thorough enough. Now it's time to critisize science....

In several papers I am reading at the moment (ranging from MIS Quarterly, Information & Management, Decision Support Systems and many more journals) I encounter something similar.

Please read the following quotes:

"....the road to DW success has been littered with failures [43,63,80]"
"....nearly half of all DW initiatives end up as failures [38]"
"According to a press release 2005 by Gartner: through 2007 more then 50% of data warehouse projects will have limited acceptance, or will be outright failures."

MIS is a Mirage

note: this blog post was first published on my B-eye-network blog.

In my quest for sustainable knowledge I am constantly searching for those little pieces of 'gold' in the literature. John Dearen's publication in the Harvard Business Review in 1970 is a great example. It's called MIS is a mirage.

Please remember that this article was written in 1970 - so try to go back in time when you read this article or this blog. It's fun!!

Some Quotes:
"Of all the ridiculous things that have been foisted on the long-suffering executive in the name of science and progress, the real-time management information system is the silliest"

In trying to decribe MIS as a term Dearden writes: "It is difficult to even describe MIS in a satisfactory way because this conceptual entity is embedded in a mish-mash of fuzzy thinking and incomprehensible jargon."

I wonder if I replace the word MIS for BI.....whether John Deardens' remark is still valid.

Blogging on B-eye-network.com now

Just a heads up; I have been given the honor of blogging for B-eye-network.com now; click here

This blog and the B-eye-network blog will be more or less in sync regarding the postings.

The Qlikview chronicles - the sage continues II

A reaction of Gilles Pol on my post regarding Qlikview. @gilles; totally agree with your realistic post, QV can be an extremely good prototyping instrument that can rapidly show the merits of Business Intelligence to stakeholders.

I also agree with the statement of Gilles that QV might be a good fit for companies in the early stages of data management maturity, although I believe that many BI tools can accomodate those early stages.

The Qlikview chronicles - the sage continues

The discussion with QlikView fanatics (look at my previous post) eventually did cost me a lot of time. But I felt it necessary.

------------------------------------------
The discussion on LinkedIn eventually resulted in my simple question:
.....But your remark about DWH and next generation BI tools is interesting. Are you saying that Qlikview replaces all DWH and ETL functionality? ....
------------------------------------------

His answer:
Dear Ronald,
I love to discuss at my blog only as it is readable by everyone who visit my page.
If you look at QlikView architecture its not required to create a data warehouse as any how it is pulling data and storing it in QVW files which have GUI also and it is single layer architecture. So if we pull data either from raw table (Transaction System/Data base) or from Data warehouse it makes no difference as you are pulling required field/dimension and it is being stored in QVW file as its native way. I saw different software companies who propose QlikView as a front end tool that will be pulling data from Data warehouse. It is just to stretch the project so they can make more money on services as QlikView take very less time to implement hence it is not giving much profits to vendors as compared to traditional BI Software to earn from QlikView is only way do rapid sales. Show ROI to customer in short time, Get repeat order implement fast and make it customers success and set example for next prospect.
If you look at ETL functionality QlikView’s edit script is sufficient enough in my view.
Sorry to say but it is not an advertisement I am not employee of QlikTech. I am a sales guy who sales QlikView also apart from Navision and Actionbase kind of tool.
Regards
Sudhir Kumar Singh
------------------------------------------

My answer:
I like qlikview a lot. As I said, it's extemely powerfull in several areas of Business Intelligence. I also acknowledge the fact that it's deployment time is fast and customer statisfaction is high. But claiming that you don't need any form of propagation of data when you have QV is a bit short-sighted.

Somehow you have come to believe that data warehousing is there to serve soly business Intelligence as defined by Qlikview. That's scary. Wake up and smell the roses and welcome to the 21st century- a single system of fact can/should and will offer so much more for any company.

In my opinion data should be stored as (vendor) neutral and open as possible in order to serve as many functionalities as possible. It's called - begin agile for requirements you may not know in advance and creating a sustainable architecture for your data. If might even be a possiblity that an organisation does not want to lock her data into one single tool........aint that realistic?

Has is ever occured to you that there might be several uses/functionalities for the same data? Use/functionality that is not offered by QV (you will probably respond now that QV can do it all.....). Storing the data in a proprietary format aint gonna help this a lot....Or is the storage solution offered by QV open for other vendor's? 

Laugh or cry.....I cried

Read this: http://www.qlikviewvsolap.blogspot.com/

It stands as a symbol for declining rigor in our beautifull industry. Just copy and paste stuff from SAS website, make comparisons that can never be compared and finally write conclusions that have NOOOOO basis whatsoever.

I am not even waisting my time to explain why this is an abomination......

***edit: join the 'Business Intelligence Group' which is a linkedin Group if you wanna add something to this discussion. It's becoming very interesting since I have - once again - stepped on the toes of a vendor's marketing machine. They are to lazy/cheap to buy some decent advertisment space so they spam these groups. And when critical remarks are made, they get pissed. Any help is appreciated here!!

Science meets DWH and BI.....and vice versa.

I have just finished reading several papers that describe the history of DSS (Decision Support Systems)-research [1,2,3,4,5]. Most of these papers are so-called meta-analysis studies.

First of all; I am doing research in the field of Information Management. Decision Support Systems (DSS) are part of that field of research. For this blog post I will use the categorisation of Arnott [4,5] within DSS:
1 - Personal DSS (PDSS)
2- Group Support Systems
3- Enterprise reporting and analysis (includes BI)
4- Data Warehouse
5- Intelligent DSS
6- Knowledge management-based DSS
7- Negotiation Support Systems
8- others

Interesting quote from Arnott [4];
'PDDS remains an important aspect of the IT-based management support in contemporary practice. Modern PDSS can source data from data warehouses and deploy powerfull modelling approaches from management science/operations research. The current industry term for the later class of PDSS is 'Analytics'.'

I like this quote because it de-hypes the phrase 'analytics' a bit.....a bit more grounding which is needed!!

Three things strike me when I read the categorisation of these DSS types:
1- Arnott puts Business Intelligence in category three. Let me quote Arnott [4] again:
'Business Intelligence is a poorly defined terms and it's industry origin means that different software vendors and consulting organizations have defined it to suit their products; some even use 'BI' for the entire range of DSS approaches. We use Business Intelligence as the contemporary term for both model-oriented and data-oriented DSS that focus on management reporting, that is, BI is a contemporary term for EIS.'

I very much agree with Arnott and would like to go that extra mile here; Business Intelligence, as a term, is dead. I would like to summarize the above quote into the fact that Business Intelligence in the idustry is badly grounded in any form of theory. For the sake of this post I will use the term Business Intelligence as defined by Arnott.

2- Arnott distinguishes between data warehousing and business intelligence as two different types of DSS.

Again; something I teach (and preach) for a long time. It will make a more rigor discussion possible if we would all seperate DWH from BI and vice versa.

3- Data warehousing is still being pushed into the decision support category of IS research. I see a DWH more as a prerequisite - in some cases - for decision support. A data warehouse by itself can not support decision making.....and it should be noted that modern day data warehouse not only should be able to support decision making, but also other processes.

I won't go deep into the articles, but let me give you some conclusions:
- There is an extreme low proportion of scientific papers on DSS types 3 and 4
- There is huge indication that DSS Types 3 and 4 are largely responsible for commercial expenditures in DSS.
- In general; the DSS research field is facing a crisis of relevance. More than half of all DSS research was assessed as having low or no practial relevance
- Only type 3 and 4 have reasonable relevance scores
- But...less than 10% of all DSS papers/research are in this area and data warehousing is only at a staggering 1.3%
- In most of the papers there is no reference to any judgement-or decision making methodology.

What do you guys think of this? Aint it weird that companies are investing (generalizing a bit here) hugely in data warehousing and Business Intelligence and their is virtually no scientific attention to these DSS types?  Data warehousing and Business Intelligene are very poorly grounded in science and yet companies seem to throw billions at it.....

Two things;
- The scientific community needs to wake up. They need to forget about the low relevance DSS Types and focus on the ones that seem to matter.....
- Practitioners in data warehousing and Business Intelligence need to strenghten the theoretical foundations a hell of a lot.

Science meets DWH and BI.....and vice versa. I got my work cut out for the years to come.

Edit may 27^th: Partner in crime - Wouter van Aerle - made a good post on his blog regarding this post

[1] Harvard Business Review - J.Dearden - MIS is a mirage - 1970
[2] Decision Support Systems - S.EOM, S.Lee - Leading US universities and most influential contributors in decision support systems research (1971 -1989)
[3] Decision Support Systems - I.Benbasat, B.Nault - An evaluation of Empirical research in Managerial Support Systems - 1990
[4] Journal of Information Technology - D.Arnott, G.Pervan - A critical analysis of decision support systems research - 2005
[5] Decision Support Systems - D.Arnott, G.Pervan - Eight key issues for the decision support systems discipline.