Data Management & Decision Support

Wendbaar wetgeven; en nu de data.......

Komende vrijdag 22 november 2019 geef ik een presentatie voor de Expertisegroep Regelbeheer, dit is een kennisuitwisselingsverband tussen Belastingdienst, SVB, DUO, IND en Rijkswaterstaat op het gebied van bedrijfsregelbeheer. Onderstaand een abstract van de sessie die ik aldaar
ga geven. 

Voor een meer generieke achtergrond verwijs ik graag naar een vorige blog (engels)

Data is helaas, in de wereld van nu, vaak een bijproduct van een applicatie of een te ondersteunen proces. Belangen die de applicatie of het proces overstijgen worden onder het tapijt geveegd en 'opgelost' in kunstmatige en gebrekkige opwerkingen zoals datawarehouses, datalakes of andere (vaak informele) kopieermechanismen. Deze 'dataschuld' zorgt voor enorme uitdagingen t.a.v. wendbaarheid van wetsveranderingen op het digitale landschap, actualiteit en kwaliteit in dienstverlening naar burger en bedrijf en risico's t.a.v. ongecoördineerde verspreiding van data. We moeten data & logica scheiden van proces & applicatie. Dit wordt ook wel de scheiding tussen 'know' en 'flow' genoemd. Op het gebied van logica, en deze netjes afleiden uit de wet- regelgeving, heeft de Belastingdienst reeds ruime ervaring. Het is tijd dat nu ook het grote 'databeest'  zich hierop aansluit.

Daartoe heeft de Belastingdienst een Verticale Datadefinitie Architectuur  vastgesteld waarbij elke vorm van vastlegging is afgeleid uit wet- en regelgeving en los staat van proces- of
applicatiebelangen. Waarmee de impact van wetswijzigingen transparant wordt en - vice versa - de data altijd terug kan worden herleid tot wet- en regelgeving. Kennis-, informatie en datamodelleren is het vakmanschap dat (weer) nieuw leven in moet worden geblazen. Denk aan cognitatie, feitmodelleren, natuurlijke taal, regelspraak, logisch modelleren, anchor-style modellering, modeltransformaties en ordentelijke  'separation of concerns'. En nee, dit is allesbehalve een IV of IT taak, dit is per definitie een innige samenwerking tussen wetschrijver, wetsanalist. modelleur en engineer. Dat deze datadefinitie architectuur een paradigmaverschuiving is in de wijze waarop informatiesystemen tot stand komen, begint langzaam door te dringen. De grote uitdaging zit hem dan ook in de transitie, de fundamenteel andere manier van denken, werken en opleiden en hoe e.e.a. realistisch kan kantelen in de uitvoering.  

Data en regels horen bij elkaar, ze zijn als het ware 'brothers in arms'. Gezamenlijk definiëren zij een kennisdomein die gescheiden moet zijn van uitvoering.  De nieuwe Referentiearchitectuur Gegevenshuishouding van de Belastingdienst is gebaseerd op drie ontwikkelingen; (1) Wendbaar wetgeven, (2) Common Ground en (3) Regie op Gegevens. Een van de belangrijkste strategische thema's, de Verticale Datadefinitie Architectuur, wordt in deze sessie toegelicht.

The Yin & Yang of data: 'Science of data' & Datascience

People familiar with my thinking know that I am a bit of a 'fundamentalist' when it comes to 'data'. I am the guy that pushes the non-sexy part of data; data quality, data governance, metadata, data protection, data integration, semantics, rules, etc..

It is hard to stand your ground in a time where shortermism, technology-fetish and data-populism is thriving. I see ‘data architectures’ in my industry that boils down to superdooper databases, ultrafast massively parallel hardware and of course huge amounts of software that seem to glorify ‘coding’ your way to the promised kingdom.

Call me old school, but I want (data) architectures to separate concerns on various levels (conceptual, logical and technical), dimensions (process, data, interaction) and aspects (law & regulation, people, organisation, governance, culture). Architectures should enable businesses to reach certain goals that (preferably) serve the customer, civilian, patient, student, etc..

Lately I have been studying the ‘datascience’ community, attempting to comprehend how they think, act & serve the common goals of an organisation. I have abandoned (just a bit :-)) my declarative nature of data modelling, semantics, dataquality or governance and I have drowned myself in coursera courses, learning to code in Python, Julia and R. Dusting off my old Statistics books I learned in Uni, installed Anaconda, Jupyter notebook, Rstudio, Git, etc..

And oh my, it is soooo cool. Give me some data, I don’t care what, where it comes from and what it exactly means, but I can do something cool with it. Promise!

Now my problem…

• (1) It seems to me that the ‘science’ in ‘datascience’ is on average extremely low to non-existent. Example; I have heard of ‘data science’ labs/environments where the code is not versioned at all & data is not temporal freezed, ergo; reproducibility is next to zero. Discovering any relationship between variables does not mean it is a proven fact, more is needed. Datasciene is not equal to data-analysis with R (or whatever), is it?  
• (2) There seems to be a huge trust in relevance and quality of data wherever it comes from, whatever its context and however it is tortured. Information sits at the fabric of our  universe¹, it’s life, it’s the real world. Data is the ‘retarded little brother’ of this ‘information’, it is an attempt of humankind to capture information in a very poor way. Huge amounts of contexts are lost in this capturing. Attempting to retrofit ‘information’ from this ‘retarded brother’ called ‘data’ is dangerous and should be done with great care. Having these conversations with data scientists is hard and we simply seem to completely disconnect.
• (3) There seems to be little business-focus, bottom-line focus. Datascientists love to ‘play’, they call it experiment or innovate. I call it ‘play’ (if you are lucky they call their environment ‘sandbox’, wtf?). Playing on company resources should be killed. Experiments (or innovations) start with a hypothesis, something you wanna proof or investigate. You can fail, you can succeed, but you serve the bottom-line (and yes, failing is serving the bottom-line!) and the purpose/mission of an organisation. Datascientists seem to think they are done when they’ve made some fascinating machine learning-, predictive- or whatever-model in their sandbox or other lab-kind-of-environment. Getting this model deployed on scale in a production environment for ‘everyone’ to use, affecting the real world…..that is where the bottom-line value really shines, you are not done until this is achieved.
• (4) There seems to be little regard for data protection aspects. The new GDPR (General Data Protection Regulation) is also highly relevant for datascience. Your ‘sandbox’ or your separated research environment needs to be compliant as well! The penalties for non-compliance are huge.

There is huge value in datascience, its potential is staggering and it is soo much fun. But please, stop fooling around. This is serious business with serious consequences and opportunities for everyone and for probably every domain you can think of, whether it be automotive, banking, healthcare, poverty, climate control, energy, education, etc…

The ‘science of data’ and ‘datascience’ are the yin & yang of fulfilling the promise of being truly datadriven. Both are needed.

For my Data Quadrant Model followers; it is the marriage  between quadrant I & II versus quadrant III & IV.

¹ Increasingly, there is more and more 'evidence' originating from theoretical physics that this statement hold some truth to it, link [Dutch]. I would also like to give attention to this blogpost by John O'Gorman, very insightful, putting DIKW to rest....

 

 

Make data management a live issue for discussion in your organization

Last month an interview with me was published in the SAS Future Bright Magazine. There quarterly theme was 'A Data Driven Reality' [Link to Dutch version] - right up my alley. This magazine was published in French, English and Dutch.

At the event that launched this quarterly magazine I had the honour to speak about the core of my interview; The Data Quadrant Model.

Thanks SAS Institute for having me, for publishing a great magazine and for giving data management the attention it so desperately needs. 

For those interested in the interview that attempts to explain the Data Quadrant Model in layman's terms - here are the downloads:

Download Dutch - Het Data Kwadranten Model, Interview Ronald Damhof

Download English - The Data Quadrant Model, interview Ronald Damhof

Download French - modèle de quadrant de données, Ronald Damhof

 

 

Data Quadrant Model: decreasing entropy

With regard to 'order' and the 2^nd law of thermodynamics I have discussed the four quadrants of my Data Quadrant Model:

1. The order of quadrant I
2. The chaos of quadrant III
3. The un-order of quadrant IV
4. The "order" of quadrant II

Entropy in the Data Quadrant Model is lowest in quadrant I, higher in II, even higher in IV and highest in III. If we do not actively decrease it, we tend to loose the value of (the) data(-platform) and we will find ourselves investing huge amounts of Euro's to do the same thing over and over again (like Groundhog Day) or spending huge amounts of Euro's to control an 'out-of-control-beast'. Unfortunately, I have seen and still see this a lot.

In an ultimate open system¹, entropy can not decrease. In the universe the entropy will only increase (and eventually we all die). However, the Data Quadrant Model is a closed system. And in a closed system we can decrease entropy. How?

There are roughly three important directions in which entropy is to be decreased actively: 

1. Decreasing entropy from III to I
2. Decreasing entropy from IV to II
3. Decreasing entropy from II to I
   (I describe these in the details of this post. Warning it is not for the faint-hearted)

Important message: like in physics, decreasing entropy costs energy. The higher the difference in entropy between two systems/quadrants, the higher the energy needed. And yes, you can replace 'energy' with 'costs'. 

We now enter the field of Data Management. A prime directive of Data Management is to reduce entropy in data and to keep the data-platform in a sustainable modus where it serves the data-driven and data-centric organisation. It is hard, not cheap and still mostly unknown territory, but if you can make it work, the rewards - in the era of datafication - are huge. 

------------------------------

Data Quadrant II: the battle against entropy

A word of warning; after writing this post I realise that truly understanding it, requires a more than basic understanding of the Data Quadrant Model. And of course; if you still don't understand where I am going with it, then it's save to say that I failed you, my apologies. :-)

A while ago I wrote a blog about the Data Quadrant Model I developed. I use this model in my consultancy and speaking engagements. Increasingly I receive great feedback from organisations that are applying it, which is great.

A week ago I wrote about the un-order of quadrant IV. A few days ago I posted the blog about the order of quadrant I as well as the chaos of quadrant III. It's time to describe the battle against the ever increasing entropy of quadrant II.

Imagine quadrant I as data where the molecules are set in a fixed position, neatly managed, ordered & governed. Now, in quadrant II the molecules are beginning to move. They can be rearranged in a vast amount of different positions, the number of positions can be huge, infinite even. Now replace these molecules with data and take into account that data can be copied and re-used over and over again. There is a vast array of possible positions for data.....

Back to the Data Quadrant Model...

Quadrant II reflects the contexts¹ in which data is used, like I said before - in quadrant II we have multiple versions of the truth.

Lets, for argument's sake, assume that the number of contexts/truths is potentially large. It is save to say that the number of ways in which data can (and probably must) be re-arranged, aggregated, calculated, inferred or otherwise manipulated to serve a vast array of contexts, will equally be (exponentially) large. 

What quadrant II aims to achieve is to control the entropy as long as possible against justifiable costs. It is known that entropy, by definition, will only increase. But there all kinds of tools, technologies and mitigation measures to extend the time before entropy reaches an un-manageable (or non-cost-effective) state. If the latter happens we can always return to the 'zero-state' - still present in quadrant I - and start again. 

That's the cool thing about data. Entropy in the universe is ever increasing and we can never reset (unless we break some fundamental rules of physics or with divine intervention), with data we can....

Summarise
Quadrant II is trying to maintain order (same as quadrant I). The entropy makes it harder though and subsequently more energy (=costs) is needed. Where the systems in quadrant I are characterised by its obvious state, the systems in quadrant II are considered to be complicated².

 

¹ It would be interesting to research what the independent variables for # of contexts could be. Size of the organisation, # of internal & external stakeholders, # differentiation of products and services, dynamics of the (world)market, complexity of business processes, managerial effectiveness, organisation hierarchy...?

² A system can be very complicated but not complex at all. A system is complex when it has emergent behaviour (quadrant IV)

The chaos of Data Quadrant III

A while ago I wrote a blog about the Data Quadrant Model I developed. I use this model in my consultancy and speaking engagements. Increasingly I receive great feedback from organisations that are applying it, which is great.

A week ago I wrote about the un-order of quadrant IV. A few days ago I posted the blog about the order of quadrant I. Now it's time to describe the most misunderstood quadrant, quadrant III. 

So far, quadrant III has received little mention, even though it is incredibly important. It is the quadrant of data sources which are not under governance, like an ad hoc download which you obtain from an open data provider, a list in Excel that you want to use, or a set of verification data which you have received on a CD. But you might also use it to dump huge amounts of data that you want to explore or experiment on. 

Quadrant III exists (implicitly) in every organisation. Only think of the huge amounts of Excel sheets or MS Access databases currently stored on your fileservers. 

Entropy in quadrant III is at its height. There is data chaos. As Stephen Brobst (chief technology officer of Teradata) so eloquently put it when he described a data lake - slightly paraphrased; 'the only people that can use this data are the people that  have put it in'.

And yes - the popular 'data lake' is an artefact that is situated in quadrant III. 

Governance, control, checks and balances in quadrant III are virtually non-existent. People using this data (the peeps in quadrant IV) need to discover the structure/schema, make inferences and be extremely careful in drawing conclusions. The field of statistics is important in this regard. 

Let me be absolutely clear; I am not against a data lake. I am just against the notion that a data lake is viable as a quadrant I artefact (and that is how the industry is trying to sell it). As a quadrant III artefact the notion of a data lake could be valuable. Is is a cost-effective artefact that has the potential to drive innovation.

Although I described quadrant III as data chaos which is unmanaged, ungoverned, etc... It is possible to manage and govern the infrastructure. Current technology is promising in offering a managed and governed infrastructure on top of which one can innovate with data in an unmanaged and ungoverned manner.

The cool thing of this technology is that quadrant III is being made explicit, from the shadows into the light. In quadrant III an organisation might want to offer its users an infrastructure-as-a-service. 

Oh, one more thing; if you consciously architect/configure a quadrant III you are engaged with innovation. If you do not, you are just messing about. ;-)

The big danger of quadrant III (and quadrant IV) is that promising discoveries are never productised (I unfortunately see that a lot!). They remain to be high potentials that never ever see the light of day (promotion to quadrant I and II). The high level of entropy enforces this risks evenmore, since discoveries are by definition often related to the brilliance of individuals. Governing mechanisms are need to counter this risk, otherwise you are just playing with data.....a waste I'd say.

I have now discussed three quadrants (I, III and IV) in terms of various levels of order. I have one quadrant to go. Before I move on, let me again stress the following:

1. None of the four quadrants is more desirable that any other; there are no implied value axes. All quadrants are needed in every organisation that wants to be more data-centric, data-driven, adaptable to changes, innovative, etc..
2. The Data Quadrant Model is NOT a blueprint implementation model. It can however be used as a guide in how to organise, how to manage, how to govern, what technology choices to make, what people to hire, etc.. 

 

 

The order of quadrant I

A while ago I wrote a blog about the Data Quadrant Model I developed. I use this model in my consultancy and speaking engagements. Increasingly I receive great feedback from organisations that are applying it, which is great.

A week ago I wrote about the un-order of quadrant IV, this post will be about it's (diagonal) mirror image, quadrant I. The quadrant of Facts. 

It is in all its aspects a mirror image of quadrant IV. It is the quadrant with the lowest entropy. There is order, governance, rules, standards and automation. Quadrant I is constantly searching for efficiency or optimisation gains. The 'systems' (I use the term loosely here) in this quadrant are characterised by (mostly linear) cause and effect relations that are repeatable, perceivable and predictable. Checks and balances are strict in terms of the adherence to standard operating procedures.

Quadrant I is the domain of methodology (seeks to identify cause-effect relationships through the study of properties which appear to be associated with qualities). Methodology can be translated to architecture as in (information) system development. Put it In other words; if you have no architecture for quadrant I.....

There is a tremendous effort necessary to keep the 'systems' in quadrant I low in terms of entropy, it needs to be controlled and it needs to be governed fiercely. Organisational-wise, quadrant I benefits hugely of organisational centralisation (as opposed to quadrant IV that benefits a lot from decentralisation).

What it all boils down to is that in quadrant I there is no fooling around with data. People working in quadrant I have a data-mindset combined with an engineering heart. Data Architecture, Data & Information modeling, Databases (using the terms loosely), Data processing (sets & transactions), etc.. They are infused with the will to improve the consistency, availability, reliability and traceability of data. If there is a science of data, these people are living it and doing it. If anyone wants to know the pedigree of a certain data element....

And no - they are not 'IT' - they are the glue that binds business with IT¹ and they are scarce. You either got to buy them or train them - there is no middle ground, especially since universities are dropping the ball here big time. They seem to be pre-occupied by offering 'Data Science' masters now, riding the waves of opportunity and short-termism.

I want 'Science of Data' masters.....

Ah well, at least there is job security for quadrant I peeps. ;-)

 ^{1 In terms of Business- and IT alignment I am loosely referring to the generic framework for Information Management - the middle column of the Information management model. Warning: Organisations are using this framework as a blueprint for organizational structure. I can not begin to explain how utterly wrong that is. It is a framework that assists in thinking and designing, nothing more.} 

The un-order of Data Quadrant IV

A while ago I wrote a blog about the Data Quadrant Model I developed. I use this model in my consultancy and speaking engagements. Increasingly I receive great feedback from organisations that are applying it, which is great. 

In this post I want to elaborate a bit more on quadrant IV (see figure).
This quadrant is often present implicitly in most organisations.Just think of all the Excel users in your organisation! Nowadays, quadrant IV is supercharged by all the stuff that is offered under the flag of 'Data Science'. And yes, I put quotes around it....the word 'science' bugs me a lot...

But that is another blogpost.

Quadrant IV is often misunderstood, especially by my fellow architects and management. They have an inert need to establish order in this quadrant. And there is no order....there is also no dis-order.

Huh?

Dave Snowden is actually describing it correctly; there is un-order: "Un-order is not the lack of order, but a different kind of order, one not often considered but just as legitimate in its own way". He uses a great story to exemplify un-order:

"....a group of West Point graduates were asked to manage the playtime of a kindergarten as a final year assignment. The cruel thing is that they were given time to prepare. They planned; they rationally identified objectives; they determined backup and response plans. They then tried to “order” children’s play based on rational design principles, and, in consequence, achieved chaos. They then observed what teachers do. Experienced teachers allow a degree of freedom at the start of the session, then intervene to stabilize desirable patterns and destabilize undesirable ones; and, when they are very clever, they seed the space so that the patterns they want are more likely to emerge. "

Order might eventually emerge.....

Now, lets take this notion of un-order a bit further into quadrant IV. If something is ordered, management (and architects alike) like to focus on efficiency (e.g. control, standards, predictable/repetitiveness, best practices etc..) a lot. In an un-ordered system (like quadrant IV) this behaviour is killing. Management and architect need to allow a lot of sub-optimal behaviour in order to reap the benefits of quadrant IV. 

Put it more popular (and scientifically wrong, but what the hell); In quadrant IV we need to 'architect' a bit of chaos. We need to un-manage.....un-control....experiment & learn...Trust

So, we gotta free the experts/knowledge workers (yep, not calling them "data-scientists") in quadrant IV.

Will order eventually emerge? Yes and no. Order and un-order will exists both in the same timeframe in quadrant IV. 

 

 

The biology of data, Conformers & Regulators

Imagine 'Data' being an organism, a living being existing in an environment we tend to call 'organization'. In the field of biology there are two types of organisms in such an environment; conformers and regulators.

These two types have different mechanisms as to how they regulate their life system parameters (e.g. temperature, pH, blood glucose), in other words, how they maintain homeostasis.

Those organisms that conform allow the environment to determine several important life system parameters. A reptile for example uses a sun-heated rock in the morning to raise its body temperature. So - conformers needs to change its behavior to maintain homeostasis.

Those that regulate, maintain life system parameters at a constant level over possibly wide ambient environmental variations. It has the capability to adjust its metabolism to maintain homeostasis. In other words; we - human beings - are regulators.

Now, imagine 'Data' being a regulator. This would mean that 'Data' is capable of adapting, self-sufficiently, to different environmental variations. Now, in several discussions regarding data - especially Big Data - I encounter an assumption that data by itself can have value, meaning and relevance. And if there are lots of it, its value is even bigger!

That is just so wrong….

'Data' is definitely a conformer - its environment determines its life system parameters. Its value, it’s meaning, its relevance is - by definition - determined by the environment.

Still with me? Lets continue.

What is left of 'Data' - being a conformer - when it cannot correctly detect its environment? Or even worse, what happens to 'Data' when the environment is void? The answer is simple; it can not sustain, even worse, it can not exist. 'Data' is profoundly in existential trouble. 

Data needs context
Data needs metadata
Data needs definitions 

TDWI EU - Agile Data Warehousing; What is the buzz about

June 18th, just after the demise of the Dutch soccer team, I gave a seminar at TDWI-EU in Munich. I think me, Jos van Dongen and Frank Buytendijk where the only Dutch guys (speakers as well as attendee) at a succesful conference with over 700 attendees. 

I had a blast and was impressed by the seminars given by Jos van Dongen, Barry Devlin and Mike Ferguson. I learned a hell of a lot and had great laughs with these guys as well. 

To top things of I had some vegan food in Munich with a bunch of Pentaho folks that almost convinced me to join an even darker side (instead of SAS....lol). If you guys are reading this; thx for a great evening and showing us some architecture in Munich. 

Oh and please - there is no correlation between Pentaho and vegan food. How a meat-loving guy like me ends up in a vegan restaurant deserves a separate blogpost. Suffice it to say that Barry Devlin was to be blamed ;-). The food was great though!

The seminar regarding Agile Data Warehousing had something like 50 attendees I think. Which is pretty ok, considering my competition as well as considering the fact that Data Vault is not very known (yet) in Germany. I had some great questions, especially from the guys/gallls who 'know' and have 'been in the trenches'. Lots of recognition and good questions and discussions were asked after the session and the days that followed.

It amazed me that large companies in Germany experimented already with Data Vault and data warehouse automation or were familiar with it and wanted to start a pilot. Beside the Scandinavian countries, Holland and Belgium; I think Germany will soon join and contribute to the community, which would be great.