Enterprise Data Management, some thoughts

The Qlikview chronicles - the sage continues

The discussion with QlikView fanatics (look at my previous post) eventually did cost me a lot of time. But I felt it necessary.

------------------------------------------
The discussion on LinkedIn eventually resulted in my simple question:
.....But your remark about DWH and next generation BI tools is interesting. Are you saying that Qlikview replaces all DWH and ETL functionality? ....
------------------------------------------

His answer:
Dear Ronald,
I love to discuss at my blog only as it is readable by everyone who visit my page.
If you look at QlikView architecture its not required to create a data warehouse as any how it is pulling data and storing it in QVW files which have GUI also and it is single layer architecture. So if we pull data either from raw table (Transaction System/Data base) or from Data warehouse it makes no difference as you are pulling required field/dimension and it is being stored in QVW file as its native way. I saw different software companies who propose QlikView as a front end tool that will be pulling data from Data warehouse. It is just to stretch the project so they can make more money on services as QlikView take very less time to implement hence it is not giving much profits to vendors as compared to traditional BI Software to earn from QlikView is only way do rapid sales. Show ROI to customer in short time, Get repeat order implement fast and make it customers success and set example for next prospect.
If you look at ETL functionality QlikView’s edit script is sufficient enough in my view.
Sorry to say but it is not an advertisement I am not employee of QlikTech. I am a sales guy who sales QlikView also apart from Navision and Actionbase kind of tool.
Regards
Sudhir Kumar Singh
------------------------------------------

My answer:
I like qlikview a lot. As I said, it's extemely powerfull in several areas of Business Intelligence. I also acknowledge the fact that it's deployment time is fast and customer statisfaction is high. But claiming that you don't need any form of propagation of data when you have QV is a bit short-sighted.

Somehow you have come to believe that data warehousing is there to serve soly business Intelligence as defined by Qlikview. That's scary. Wake up and smell the roses and welcome to the 21st century- a single system of fact can/should and will offer so much more for any company.

In my opinion data should be stored as (vendor) neutral and open as possible in order to serve as many functionalities as possible. It's called - begin agile for requirements you may not know in advance and creating a sustainable architecture for your data. If might even be a possiblity that an organisation does not want to lock her data into one single tool........aint that realistic?

Has is ever occured to you that there might be several uses/functionalities for the same data? Use/functionality that is not offered by QV (you will probably respond now that QV can do it all.....). Storing the data in a proprietary format aint gonna help this a lot....Or is the storage solution offered by QV open for other vendor's? 

Laugh or cry.....I cried

Read this: http://www.qlikviewvsolap.blogspot.com/

It stands as a symbol for declining rigor in our beautifull industry. Just copy and paste stuff from SAS website, make comparisons that can never be compared and finally write conclusions that have NOOOOO basis whatsoever.

I am not even waisting my time to explain why this is an abomination......

***edit: join the 'Business Intelligence Group' which is a linkedin Group if you wanna add something to this discussion. It's becoming very interesting since I have - once again - stepped on the toes of a vendor's marketing machine. They are to lazy/cheap to buy some decent advertisment space so they spam these groups. And when critical remarks are made, they get pissed. Any help is appreciated here!!

Science meets DWH and BI.....and vice versa.

I have just finished reading several papers that describe the history of DSS (Decision Support Systems)-research [1,2,3,4,5]. Most of these papers are so-called meta-analysis studies.

First of all; I am doing research in the field of Information Management. Decision Support Systems (DSS) are part of that field of research. For this blog post I will use the categorisation of Arnott [4,5] within DSS:
1 - Personal DSS (PDSS)
2- Group Support Systems
3- Enterprise reporting and analysis (includes BI)
4- Data Warehouse
5- Intelligent DSS
6- Knowledge management-based DSS
7- Negotiation Support Systems
8- others

Interesting quote from Arnott [4];
'PDDS remains an important aspect of the IT-based management support in contemporary practice. Modern PDSS can source data from data warehouses and deploy powerfull modelling approaches from management science/operations research. The current industry term for the later class of PDSS is 'Analytics'.'

I like this quote because it de-hypes the phrase 'analytics' a bit.....a bit more grounding which is needed!!

Three things strike me when I read the categorisation of these DSS types:
1- Arnott puts Business Intelligence in category three. Let me quote Arnott [4] again:
'Business Intelligence is a poorly defined terms and it's industry origin means that different software vendors and consulting organizations have defined it to suit their products; some even use 'BI' for the entire range of DSS approaches. We use Business Intelligence as the contemporary term for both model-oriented and data-oriented DSS that focus on management reporting, that is, BI is a contemporary term for EIS.'

I very much agree with Arnott and would like to go that extra mile here; Business Intelligence, as a term, is dead. I would like to summarize the above quote into the fact that Business Intelligence in the idustry is badly grounded in any form of theory. For the sake of this post I will use the term Business Intelligence as defined by Arnott.

2- Arnott distinguishes between data warehousing and business intelligence as two different types of DSS.

Again; something I teach (and preach) for a long time. It will make a more rigor discussion possible if we would all seperate DWH from BI and vice versa.

3- Data warehousing is still being pushed into the decision support category of IS research. I see a DWH more as a prerequisite - in some cases - for decision support. A data warehouse by itself can not support decision making.....and it should be noted that modern day data warehouse not only should be able to support decision making, but also other processes.

I won't go deep into the articles, but let me give you some conclusions:
- There is an extreme low proportion of scientific papers on DSS types 3 and 4
- There is huge indication that DSS Types 3 and 4 are largely responsible for commercial expenditures in DSS.
- In general; the DSS research field is facing a crisis of relevance. More than half of all DSS research was assessed as having low or no practial relevance
- Only type 3 and 4 have reasonable relevance scores
- But...less than 10% of all DSS papers/research are in this area and data warehousing is only at a staggering 1.3%
- In most of the papers there is no reference to any judgement-or decision making methodology.

What do you guys think of this? Aint it weird that companies are investing (generalizing a bit here) hugely in data warehousing and Business Intelligence and their is virtually no scientific attention to these DSS types?  Data warehousing and Business Intelligene are very poorly grounded in science and yet companies seem to throw billions at it.....

Two things;
- The scientific community needs to wake up. They need to forget about the low relevance DSS Types and focus on the ones that seem to matter.....
- Practitioners in data warehousing and Business Intelligence need to strenghten the theoretical foundations a hell of a lot.

Science meets DWH and BI.....and vice versa. I got my work cut out for the years to come.

Edit may 27^th: Partner in crime - Wouter van Aerle - made a good post on his blog regarding this post

[1] Harvard Business Review - J.Dearden - MIS is a mirage - 1970
[2] Decision Support Systems - S.EOM, S.Lee - Leading US universities and most influential contributors in decision support systems research (1971 -1989)
[3] Decision Support Systems - I.Benbasat, B.Nault - An evaluation of Empirical research in Managerial Support Systems - 1990
[4] Journal of Information Technology - D.Arnott, G.Pervan - A critical analysis of decision support systems research - 2005
[5] Decision Support Systems - D.Arnott, G.Pervan - Eight key issues for the decision support systems discipline.

The ability of an organisation to absorb Business Intelligence

At the moment I am reading an article of Wesley Cohen and Daniel Levinthal called 'A New Perspective on Learning and Innovation' [1].

The article describes Absorptive Capacity; The ability of an organisation to recognize the value of new, external information, assimilate it, and apply it to commercial ends. The authors claim that this ability is critical for the firm's innovative capability.

For the sake of this blog post, 'External information' should not be defined to strict. Let's say we expose a firm for the first time to Business Intelligence. The ability of a firm to absorp the new possibilities that come with Business Intelligence depends roughly on the individuals- and organizational capacity to absorp Busines Intelligence.

The individual capacity to absorp Business Intelligence does not only depend on prior knowlege (on business processes,software use, statistical and analytical techniques etc..) but also on intensity of effort. Or, to put in more simple; trials and practice till you drop. Learning is more difficult in novel domains according to the authors and individual's expertise will change only incrementally. So introducing something new (like BI) to an individual will probably be more succesfull if the user posesses a broad and deep knowledge diversity. This user will be better equiped to make novel associations and linkages, thus improving the innovative process that BI can instantiate....

But what about the Organizational Absorptive Capacity? It's not only the sum of all all individual's absoptive capacities....It also depends on the ability of the organization to exploit absorptive capacity. And this in turn is heavily dependent on the structure of communications. For now, I won't elaborate on this one in this post.

The authors eventually generalize big time (but I like that) by saying; on the individual- and organizational level, prior knowledge permits the assimulation and exploitation of new knowledge. Some portion of that knowledge should be very closely related to the new knowledge to facility assimilation.

Example; if Microsoft invested timely in knowlegde regarding search techniques on the Internet....Google might not be as big....

Example; if a consulting organization wants to enter the health market, it must invest heaviliy in prior knowledge concerning this market if it wants to increase it's chances on succes.

Sounds logical aint it?

In terms of Business Intelligence; there are at least three interesting perspectives here.
1 - It pays off for a company BEFORE it embarks on a BI adventure (and contracting the builders) to invest in knowlegde regarding the mechanics of Business Intelligence, it's purpose, it's means (to an end) and off course; practise till you drop. It also pays off to assign the users with the most diversed scope of prior knowledge to the BI scene. They are able to associate and discover linkages faster that generate added value and thus increasing acceptance faster.

2 - Companies should be patient in demanding short term ROI from BI, especially when the degree of 'newness' (damn; is that a word?) is higher. The individuals as well as the organization needs time to learn from it by first doing the simple stuff (which do not deliver the most value most of the time) over and over again and slowly go to the more advanced stuff and possibly yield more return.

3- Business Intelligence should serve the exploration process in companies. Analysis of data, interpreting the data, being confronted with data, outcomes etc...These should all benefit the learning process. Business Intelligence has got the potential to be an instrument to increase the Absorptive Capacity.....

Let's be honest; the first two perspectives are not very well fulfilled in BI practise nowadays, aint it? And the third one is mostly a problem because the ROI for BI is quite intangible. On the other hand, research shows that if companies fail to invest in exploration they will increase the chance of being self-destructive [2].

[1] Absorptive Capacity; A new Perspective on Learning and Innovation. Wesley M. Cohen; Daniel A. Levinthal. Administrative Science Quaterly vol.35 no1. (Mar 1990).

[2] March, J.G. Exploration and Exploitation in Organizational Learning, Organization Science Vol.2 No.1  (1991)

The next generation EDW 3/3 is published in Database Magazine

The final article of ' The next generation EDW' was published this week in Database Magazine (Dutch). This article adresses the development proces in a next generation EDW. 

As always; in 6 weeks I will have the article available for download on my site. Some weeks later it will be available in English. Part 1 and 2 are already available for download in Dutch and in English.

Dutch post: information hiding and source extraction

A tip, although it's Dutch, is to read a blog post of Lidwine van As which is kind of sequal to a post I once wrote. It's about the concept of using information hiding as a prime source extraction method. It's good, I also responded to the post.

Next Generation EDW 2/3 - available in English now

Our second article - published in Database Magazine (Dutch) november 2008 - is now also available in English. Download available here.

The third one is to be published in March 2009 in Database Magazine. It will hanlde deployment in the Next generation EDW.

Now online available: The Next Generation EDW 2/3 - Datamart Strategy

The second article Lidwine van As and I wrote about ' The Next Generation EDW' is now online downloadable. It was publiced november 2008 in Database Magazine in the Netherlands.

Unfortunately the article is - for now- only available in Dutch. English is coming up.

Just a head up; the final article in 'The Next Generation EDW' is nearly done and will be publiced end of february 2009.

Gartner prediction: "Business units will control BI budget instead of IT"

Gartner publiced it's five predictions on BI the coming year. Coming days I will try to give my opinion on a few of them. Let me begin with the second prediction of Gartner:

Gartner's second prediction is that by 2012 business units, rather than IT, will control at least 40 percent of the total budget for BI. The problem here, Gartner wrote, is that "business users have lost confidence in the ability of [IT] to deliver the information they need to make decisions."

So business units will increase spending on packaged analytics, including corporate performance management, predictive analytics, and online marketing BI wares. In so doing, however, they "risk creating silos of applications and information, which will limit cross-function analysis, add complexity, and delay corporate planning and execution of changes," said Nigel Rayner, research vice president at Gartner, in a prepared statement.

I support the analysis of the problem by Gartner. The business has lost confidence in the ability of IT to deliver.....And since we live in a 'every-problem-needs-a-quick-fix' - world, I also support the risc Gartner states of business units, that will go shop and wil find vendors that sell pretty pictures (see my fairytales blogpost). The industry teaches it wrong.......

So...what now?

Well, first of all I wonder whether this issue Gartner is rising, is really new. I mean, everywhere I come, decentralized BI packages/solutions have been implemented mainly because 'IT could not deliver'. Whether or not that's true is not really the issue. I just do not see how it's a prediction?? Can somebody help me here? 

Now, let's embark on some kind of direction we - as an industry, as a trade - need to communicate. Business needs to be in control of the BI function.  But...

A metaphor....Imagine the automotive industry (I know - bad one atm). Organisation's in that industry need to construct a (BI) factories with production lines (processes), robots, vendor (sources) agreements delivering parts (data), stock (CDW) etc..Now, this factory exists simply because it's efficient and cost effective, but most of all, because it is able to deliver on demand (coming from customers).  Now, when these customers wanna order a car, do you think they wanna build a factory in their garden? Well, offcourse they don't....they are not even capable of doing that. Are they capable of formulating their demand - yes they are. How? Well, they have a catalog showing all kinds of characteristics (non-functionals) and options (functionality and content/metadata).

Setting up a centralized BI factory is an organisation's corporate concern. It's a basic service the organisation needs to build. Does this factory make pre-build cars? Maybe - but only when there's actual demand for it. Does this factory do custom builds? Well, to a certain extend where it remains efficient and cost effective.

Who payes what? Here the metaphor breaks a bit. Off course, customer's are paying for their 'cars and it's maintenance' - they know what kind of car they need for their particular purpose (work-related, private related, home-school etc..). It would be sooooo weird when the factory is paying for the cars and gives it to their customer saying.....'This is what you need'....

So - Business units are paying for their (BI) informational products. Nothing new I suppose? Duh....

But who is paying for the factory? Here the metaphor breaks a bit. All though - let's assume there are no car factories anymore (not unrealistic). Customers will still need car's. Like I said - it's not realistic that every customer will build there own car. So they will probably organise themselves (maybe in a cooperation or so) and let them be represented by a few of their own that are capable of setting up car factory. This 'cooperation'  is going to build a factory with a budget the members have contributed.

You still with me?

What I am trying to say is that the BI central function, the BI factory, need to be financed by business. The execution and governance however, needs to be as centralized as possible to achieve certain economies of scale. 

So - Business units need to finance the BI centralized function/factory.

And THIS IS THE REAL PROBLEM. Because, that is often not the case. Furthermore, the organisational structure of Business units with there own accountability is failing miserably at constructing/financing a centralized BI function. In the Netherlands a professor wrote  very good book about this: from unit organisation to multi dimensional organisation (unfortunately in Dutch) by H.Strikwerda. Setting up centralized functions in a business unit oriented organisation is not possible.....Read this book!

So the problem is much deeper and more fundamentally grounded in organisational theory. It therefor extends BI big time!

Pfff....in writing this I embarked on something I should make a thesis on and go underground for some years. It's a huge subject.....I might do that....

Those who read this and still understand where I am going....let me know what you think. It's a complex subject, but sooo interesting to explore even more.

"SOA" is dead, long live the data?

Finally, the "SOA" bubble is beginning to burst. It's like something - I believe - we all knew but did not say out loud. The recent crisis helped it a bit I believe....

This blog post is spot on......and to extrapolate it into Data Management I would suggest 'the industry' is going to focus itself on getting the data in control - rationalising your organisation's data architecture. I sincerely believe that it's in the area of data management where we can achieve value in terms of growth, improving margin and get into control. The latter for example in reducing complexity. Maybe, just maybe.....that's one of the underlying problems of " SOA"; they forgot about the data.

Well...thank God we are here to save the day

**edit** IBM got somewhat agitated here (lol):  Just to be precise here....I am talking about the word "SOA", the marketing hype, the product, the fantasy of implementing SOA in large organisations (give me an example!!!), It's often just too much to ask for large organisations in terms of governance, discipline etc...But vendors keep on pushing SOA as something "we all must do".....and I challenge that. I am NOT saying all services are bad..On the contrary! I sincerely believe - like the artcile - that it's a necessary path. But please, I believe that the majority of organisations are not yet mature enough to implement a Service Oriented Architecture. It would take a great man/vendor to help organisations, in reaching this conclusion and advising them on the (tiny/baby/small) steps an organisation can make to improve it's readiness for services........

**edit again*** IBM just gave me the definition of an information service, and I quote:

Information as a service is an entry point to service oriented architecture (SOA) that offers information access to complex, heterogeneous data sources within your company as reusable services. These services may be available both within the enterprise and across your value chain. By entering SOA from an information vantage, you may improve the availability and consistency of information, while simultaneously removing traditional barriers to information sharing. Drive innovation by better understanding your organization’s operational, transactional, analytical, and unstructured information and making it available in new ways through SOA.

This is exactly what is driving me mad. Vendors keep on telling, repeating the same (fantasy) story over and over again. They do that so many times that I start to believe 'I am missing out on something here'.....That's how advertising works as well. I must be stupid because I do not understand this. And just to be even more clear.....information services as described in the above definition of IBM is great - really, it is! BUT......it's got a huge level of 'green-field-thinking'. And that's not my reality. 

So please, can someone explain to me how this would work technically in a large organisation with complex processes and various heterogeneous sources? Please....explain this to me like I am a 4 year old. And do not....DO NOT..throw brochures at me, or glossy folders, or analysts reports. Just give it to me straight. How will this technically work?