Data Management & Decision Support

Wendbaar wetgeven; en nu de data.......

Komende vrijdag 22 november 2019 geef ik een presentatie voor de Expertisegroep Regelbeheer, dit is een kennisuitwisselingsverband tussen Belastingdienst, SVB, DUO, IND en Rijkswaterstaat op het gebied van bedrijfsregelbeheer. Onderstaand een abstract van de sessie die ik aldaar
ga geven. 

Voor een meer generieke achtergrond verwijs ik graag naar een vorige blog (engels)

Data is helaas, in de wereld van nu, vaak een bijproduct van een applicatie of een te ondersteunen proces. Belangen die de applicatie of het proces overstijgen worden onder het tapijt geveegd en 'opgelost' in kunstmatige en gebrekkige opwerkingen zoals datawarehouses, datalakes of andere (vaak informele) kopieermechanismen. Deze 'dataschuld' zorgt voor enorme uitdagingen t.a.v. wendbaarheid van wetsveranderingen op het digitale landschap, actualiteit en kwaliteit in dienstverlening naar burger en bedrijf en risico's t.a.v. ongecoördineerde verspreiding van data. We moeten data & logica scheiden van proces & applicatie. Dit wordt ook wel de scheiding tussen 'know' en 'flow' genoemd. Op het gebied van logica, en deze netjes afleiden uit de wet- regelgeving, heeft de Belastingdienst reeds ruime ervaring. Het is tijd dat nu ook het grote 'databeest'  zich hierop aansluit.

Daartoe heeft de Belastingdienst een Verticale Datadefinitie Architectuur  vastgesteld waarbij elke vorm van vastlegging is afgeleid uit wet- en regelgeving en los staat van proces- of
applicatiebelangen. Waarmee de impact van wetswijzigingen transparant wordt en - vice versa - de data altijd terug kan worden herleid tot wet- en regelgeving. Kennis-, informatie en datamodelleren is het vakmanschap dat (weer) nieuw leven in moet worden geblazen. Denk aan cognitatie, feitmodelleren, natuurlijke taal, regelspraak, logisch modelleren, anchor-style modellering, modeltransformaties en ordentelijke  'separation of concerns'. En nee, dit is allesbehalve een IV of IT taak, dit is per definitie een innige samenwerking tussen wetschrijver, wetsanalist. modelleur en engineer. Dat deze datadefinitie architectuur een paradigmaverschuiving is in de wijze waarop informatiesystemen tot stand komen, begint langzaam door te dringen. De grote uitdaging zit hem dan ook in de transitie, de fundamenteel andere manier van denken, werken en opleiden en hoe e.e.a. realistisch kan kantelen in de uitvoering.  

Data en regels horen bij elkaar, ze zijn als het ware 'brothers in arms'. Gezamenlijk definiëren zij een kennisdomein die gescheiden moet zijn van uitvoering.  De nieuwe Referentiearchitectuur Gegevenshuishouding van de Belastingdienst is gebaseerd op drie ontwikkelingen; (1) Wendbaar wetgeven, (2) Common Ground en (3) Regie op Gegevens. Een van de belangrijkste strategische thema's, de Verticale Datadefinitie Architectuur, wordt in deze sessie toegelicht.

Make data management a live issue for discussion in your organization

Last month an interview with me was published in the SAS Future Bright Magazine. There quarterly theme was 'A Data Driven Reality' [Link to Dutch version] - right up my alley. This magazine was published in French, English and Dutch.

At the event that launched this quarterly magazine I had the honour to speak about the core of my interview; The Data Quadrant Model.

Thanks SAS Institute for having me, for publishing a great magazine and for giving data management the attention it so desperately needs. 

For those interested in the interview that attempts to explain the Data Quadrant Model in layman's terms - here are the downloads:

Download Dutch - Het Data Kwadranten Model, Interview Ronald Damhof

Download English - The Data Quadrant Model, interview Ronald Damhof

Download French - modèle de quadrant de données, Ronald Damhof

 

 

IT does not matter

This may look like a career-killing title for a blog post, but let me explain.

Oh - just to correctly name sources; I stole the title from an article Nicholas Carr wrote in the Harvard Business Review in 2003. 

In all the hype and buzz surrounding #BigData #InternetOfThings #MachineLearning #Digitization and so on, the tech is portrayed as a huge differentiator in competitive advantage. Well...it's not. The tech is highly irrelevant in terms of competitive advantage (not saying that it is not a prerequisite!). Some companies might have an edge for some time, purely based on some innovative use of tech, but it will not last. It will be copied extremely fast over and over again by others. "Tech will be commoditized.....'. 

So if tech is not a differentiator, what is? Because there is obviously something buzzing....

You know what is a differentiator? Data.
You know what is a differentiator? How you use data to delight your customers
You know what is a differentiator? How you use data to improve healthcare, balance in-equality, fight crime or supervise vital infrastructure
You know what is a differentiator? How you respect privacy of data not only because you want to avoid litigation, but because you want to be trusted and valued. 

Data must be nurtured, managed and governed aggressively and - most of all- must be enabled to be exploited to the max in order to get a good return or help the common cause. It is by far the ultimate proprietary asset. 

So if you want to change strategy, embark on a road of datafication, be more data-centric or data-driven, you might want to spend your investment Euros or Dollars not in tech right away - resist that urge. Invest it in people first; data-strategists, data architects, data modellers, data analysts (ok ok...data 'scientists'), data quality experts, data privacy experts and so on. The tech will come...

Some statements about c-level:
1.Tech is not a differentiator
2.Tech should therefor not be represented on the c-level
3.Data is a differentiator
4.Data should therefor be represented on the c-level
5.If your CIO is about the Information (not the tech!)? Yee - good for you!
6.If your CIO is about the tech? Well, in my opinion you don't have a CIO - you have a Technology Manager

A Chief Data Officer should not be necessary if you have a true CIO. If you don't, get yourself a Chief Data Officer. Someone seasoned in the field of (respectfully) exploiting data and information for the ambitions and goals of your organization.

 

 

Data is a company-wide concern, guts needed

The point am trying to make in this blogpost is that the classic distinction between operational data environment and informational data environment (where the data warehouse resides) is fading, thank god! Data is a company-wide concern...but guts is needed to actually achieve it. 

For me to be able to make this point, I need to convey 5 important statements first,clarifying where I am coming from a bit more:

1. I am going to elaborate a bit more on the Data Quadrant Model I wrote about some time ago. Remember quadrant I - the facts? A quadrant where data is factually registered conforming to several non-functionals (temporal, historical, standardized, etc..). 
2. It is important to emphasize that the data models in quadrant I are driven by centrally managed logical data models (preferably these are derived from a conceptual- or information model) and governed by means of Data Delivery Agreements. Physical models are preferably derived as much as we possibly can from the logical model. This is not trivial, but decades of science and more and more practitioners are actually doing it, (im)proving it.
3. A data warehouse is an architectural construct and by such should be separated from its implementation or its technology. This is a distinction I rarely see, but is vital for the point I am trying to make. 
4. Data warehouses are traditionally fed with data coming from various sources, either internal systems (batch, services, whatever) or external systems. 
5. In the Netherlands (my work area) we evolved data warehouses the last decade by making a very sharp distinction between facts (quadrant I) and context (quadrant II). This distinction is being made on all levels; architecture, data modeling, technology, management, organization, people, processes, etc..

If we take a closer look to the fact-part (first quadrant) of the data warehouse, something interesting is going on. Although originally setup to accommodate feeds from various sources, it slowly evolved into an integrated data environment.

How so? Suppose I wanna build me a new order-intake system. We make use cases, do the sprints and just 'build me an app'. The datamodel is often something that 'organically' grows with the app.....brrrr...Data is a company-wide concern not an app-concern.

Classically the data warehouse guys came only into play the moment the app is finished and people needed to do some sophisticated analytics or just some management reporting. A feed was constructed and data was to be deployed to the data warehouse. Slowly and gradually this has changed in my current work environment using the Data Quadrant Model and some very smart co-workers. The app guys are still doing the use cases, but all the modeling is now done by the data modelers of quadrant I. 

These data modelers first construct a logical model that the app guys can use as a stub to build upon all their 'app-stuff'. How can they? We do not want these app-guys to be concerned with stuff like auditability, traceability, temporality, extensibility, standardization of data definitions and types, etc.. Remember in quadrant I we have a 'fundamentalistic' opinion on how the data is to be managed. 

Now, the data modelers construct the physical model by using the full force of temporal- fact based modeling and hide the complexity of the physical model by virtualizing/decoupling the data back to the original logical data model. The app is build and users can now insert, update, delete data. 

The data is decoupled from the apps, other apps might be interested.....

The data is centrally managed and governed on at least the logical level....

The non-functionals regarding data are being guarded fiercely....

Interestingly, we still make a data delivery agreement. In this case with the functional owner of the system and the governor of quadrant I. We do a handshake on the logical data model, the validations we do before the data is registered and several other stuff we expect from each other regarding the data and its quality. 

The result: data that orginally came from an operational enviroment and was fed to a data warehouse, is now directly registered in the fact-part (quadrant I) of the data warehouse. This part of the data warehouse has now faded into an operational environment with the non-functionals we learned to implement in the first quadrant. The first quadrant governers are now in charge of a company wide integrated data-environment.....

What about the data warehouse? Quadrant II and IV (stuff like reporting, dasbhboarding, analytics, visualisation) are still fed from the first quadrant and 100% demand oriented. Since we model and store all data in an integrated manner adhering to important non-functionals we are able to serve these quadrants (II en IV) cheaper, faster and with a (to some degree) guaranteed level of quality. 

Two more things;

1. What I do not say is that data is stored in one location. That can and probably will be hugely federated within the company or outside the company (e.g cloud). What I do say is that the logical model AND the metadata are crucial to govern in an integral and central manner.
2. With the above in mind I strongly urge senior management to take the next step as well. Organize the data-competency; the people, governance (e.g. protection), architecture, design and implementation of the data needs firm management support. Centralize it....pls....A centralized shared data center preferably managed directly by a true CIO (Information officer, not technology officer) or even better, a Chief Data Officer...With means in terms of expertise and budgets.

If there is an organization outthere with the pain, the guts and the patience to really go for it, to assign and mandate a CDO; call me, I am game. 

Thou cannot buy your way out of the data misery you are in

In an interview with Gerald Weinberg, he was asked the question ‘In your experience, what is the most notable “constant” in the world of software?’. His answer:

“The infatuation with the latest fad which is supposed to “increase productivity” by some arbitrary amount, with no clue as to how that “productivity” is measured. For the most part, these fads are usually a new set of names for old practices. Yet at the same time a segment of the developer population goes ga-ga over the fad, a much larger segment doesn’t even attempt to learn what is good about the fad. The majority of developers are working the same primitive way their predecessors did 50 years ago, just using more machine power to do it.”

I think these are wise words and we should all try to imprint whenever we ‘buy’ or are infatuated with the newest ‘fad’.

In the data-world I am in, this seems to be particularly true, although that might be because I am not very deep into other ‘worlds’. Examples are Big Data, data virtualization, data visualization, analytics, cloud, in-memory, mobile, data integration hubs, etc..

The promises are immense. Either an increase in productivity, a decrease in costs or a fantastic super-dooper increase of customer satisfaction. Another characteristic is the rebelling against the ‘old’ or the ‘traditional’ way of doing stuff. ‘That era is over, they say, we now have <fill-in tech trend>’ or 'xxx is dead, now we have <fill-in tech trend>'. 

Basically, ‘we ’ get the message that we must be totally out of our minds not to use this <fill-in techtrend>.

From an architectural point of view it is shocking how analysts/experts or whatever can not seem to understand the difference between logical and technical views. They are constantly mixed; utter confusion being the result.

Now, do not get me wrong, I am a fan of progress in technology and I deem the above mentioned ‘trends’ extremely valuable……when applied appropriately….

In the end this technology needs to fit into a context. A context of people, their skills and experiences, a context of architecture, a context of legacy systems and a context of strategy and objectives.

Proper software development in data management is not about technology. It is about proper business analysis, information modeling, business rules, architecture and above all…it is about people, people and people. These are the constants over time - whatever tech you use. If you want to actually yield benefits, do the work. Invest in people first.....tech comes later..really...

Even better formulated; tech evolution will accelarate more and more, it's not about this tech (IT does not matter), it is about how well you prepare your organization in adopting technology for the benefit of your customer, patient, student. It is about the 'degree of adoption' that we need to worry about, instead of following blindly the next fad, promise or tech brochure. 

Thou cannot buy your way out of the data misery you are in - it takes blood, sweat and tears. Or in other words; stamina, discipline, Trust (!!!!!) and courage (especially by management!). 

[Dutch] Full circle; ‘geef mij de data, de software en de machine’

De afgelopen 15 jaar hebben veel organisatie veel geïnvesteerd in de uitnutting van gegevens. Enerzijds door middel van de aanschaf van technologie, zowel hardware als software, anderzijds door middel van projecten met concreet op te leveren informatieproducten en de daarbij behorende training van ontwikkelaar tot gebruiker.

Dit zijn significante investeringen die vanzelfsprekend om een gezonde ROI vragen. En devraag die ik hier stel is of we de ROI op al gedane investeringen kunnen verhogen?

Het antwoord van een goede consultant zou moeten zijn ‘dat hangt van uw context af’. En daar heeft die consultant helemaal gelijk in, maar met dit schrijven wil ik een scenario schetsen dat al dan niet bij uw organisatie past. Ik zie dit scenario in toenemende mate langskomen – aan de lezer de vraag of hij dit scenario herkend in haar eigen context.

Organisatie X heeft veel geïnvesteerd in het voortbrengen van gegevens; ofwel gegevens geschikt maken voor velerlei gebruik; marketing automation, risk management, financieel management,  in-externe verantwoording en sturing, etc.. De vorm van dit gebruik uit zich vaak in de vorm van rapportages, dashboards of geleide analyses.

Om bovenstaande daadwerkelijk uit te nutten heeft organisatie X ook de kwaliteit van haar gegevens moeten verbeteren, processen moeten aanpassen, rollen en verantwoordelijkheden moeten herdefinieren en belanghebbenden moeten trainen.

De stijl van ‘ontwikkeling’ wordt nog vaak gekenmerkt door een hoge mate van systeem ontwikkeling. Ofwel; er moet bekend zijn wat er gemaakt moet worden, hoe het eruit ziet, welk gegevens nodig zijn, hoe die gegevens aangepast moeten worden, wie er toegang moet hebben etc.. In deze stijl van ontwikkelen is er een sterk onderscheid tussen diegenen die ‘maken’ en diegenen die ‘gebruiken’.  Bovendien krijg je te maken met de typische kenmerken van applicatie ontwikkeling zoals de gevreesde ‘voortbrengingsstraat’ (ontwikkel, test, acceptatie, productie) en versiebeheer.

Organisatie X heeft deze stijl van systematische ontwikkeling goed onder controle, maar ondervond enkele jaren terug toch dat de gebruiker niet echt heel tevreden was en dat de voorzieningen maar mager werden gebruikt. Eén van de hoofd oorzaken die werd genoemd is dat de gebruiker maar moeilijk in staat was om zijn wensen snel te kunnen realiseren.  De gebruiker had eigenlijk de behoefte om zelf voor een deel de ontwikkelaar te zijn. Organisatie X besloot (achteraf bezien) om een andere ontwikkelstijl te introduceren, namelijk die van de ‘gedelegeerde ontwikkeling’. Een ander type gebruiker ontstond; de gebruiker die vanuit zijn domein zelf rapportjes in elkaar ging klikken en zijn achterban voorzag.

Al klinkt dit eenvoudig, deze nieuwe stijl van ontwikkeling bracht met name voor de IT-afdeling grote hoofdbrekens; hoe zit het met versiebeheer? Wie test de rapportages? Wie beheert de rapportages? Wie zorgt ervoor dat de omgeving niet omver wordt getrokken? Maar ook daar is organisatie X goed uitgekomen.

Op dit moment voelt organisatie X nog steeds dat het rendement op de investeringen te laag zijn, dit wordt nog eens versterkt door de steeds toenemende dynamiek in de omgeving die vereist dat medewerkers van organisatie X sneller moeten kunnen reageren, meer vooruit moeten kunnen kijken en schaarse middelen steeds beter moeten inzetten.

Er ontstaat een groep gebruikers met eisen die helemaal in lijn zijn met bovenstaande dynamiek:

‘Ik weet nog niet wat ik nodig heb om mijn probleem op te lossen. Ik moet exploreren, experimenteren en leren. Dus geef mij data, geef mij software, geef mij machines’.

Wederom een nieuwe ontwikkelstijl, of toch niet? Het is namelijk de ontwikkelstijl waarbij de gebruiker gelijk is aan de ontwikkelaar. En waar hebben we dat eerder gezien? Juist – in de glorietijd van spreadsheets en lokale databases. Ook toen had de gebruiker zijn eigen data, zijn eigen software en zijn eigen machine. De cirkel is rond? Deze ‘opportunistische ontwikkelstijl’ vraagt een kanteling in denken van vele partijen binnen organisatie X.

Zo is er besloten om, naast de omgeving die systematisch (en gedelegeerd) informatie producten kan voortbrengen, ook een aparte omgeving te creëren:

• Waar snel gegevens ter beschikking gesteld kunnen worden die qua volume niet of nauwelijks grenzen kent;
• Waar gegevens overal vandaan kunnen komen;of het nu een data warehouse is, gegevens van een ketenpartner of een twitterfeed;
• Waar geavanceerde visualisaties op uitgevoerd kunnen worden;
• Waar geavanceerde analytische mogelijkheden zijn;
• Waar de resultaten van deze analyses eenvoudig (mobiel) gedeeld kunnen worden;
• Waar de omgeving op kosten-efficiënte en transparante wijze opgeschaald kan worden.

Organisatie X beseft ter dege dat deze opportunistische ontwikkelstijl governance uitdagingen kent, veel vragen van de vaardigheden van de gebruiker en zeker niet de systematische en gedelegeerde ontwikkelstijlen vervangen, laat staan het data warehouse – ze zijn allemaal nodig.

Heel bewust is er voor gekozen om hier achter te komen gedurende de reis waaraan men is begonnen. Soms kan de technologie helpen om te fungeren als breekijzer om veranderingen teweeg te brengen of te leren aan de hand van experimenten.

 

The democracy of data

There is something going on for some time now, decades even. It all started with the arrival of the Internet where people voluntarily contributed data to, well, everyone who was interested. Data about themselves, their relationships, their adventures, their careers. This data was shared with consent of the owner of the data - although not everyone knew what the data was used for. So one might say that there was consent, but not informed consent.

Lets take it a step further and imagine....

What if our data  - generated by others - was given back to us and we could consent in an informed manner, that we wish to share this data for the greater good? For example; my tax information is my data, it is about me and I want to decide whether or not others can use this data. Or, suppose I can get a hold of my location/GPS data, showing all my movements. Or my point of sale data from the grocery store, showing my eating patterns. Suppose I can even get a hold of the data of the last MRI I took, my genome data or the data of my last blood test or even the data of a clinical test I was in?

Imagine....

What if I could decide to contribute this data (consensually) for the public good, where my privacy was still being honoured? What if dozens of people would decide that? What if millions of people would decide that? Clinical research would never be the same again. We would be able to scan for patterns in seas of data consisting of environmental data and healthcare data. No more clinical trials with just 2000 people and ever-increasing smarter statistics. In this setting the healthcare specialists, the quants, the sociologists and the behavioural scientists would have an unprecedented test bed of data. Is there a correlation (or even causality) between aspects of travelling, career, eating patterns, social status and cancer? Suppose even several generations would contribute their data; what would that mean for clinical research? Mindblowing.... 

In the above I discussed data that was about myself and so I should be the one who should decide whether or not to share. But what about data that is ours? The government heavily sponsors research in many countries. Research on biology, behavioural science, economic science, climate science etc.. Shouldn’t the data generated by this research be public domain? I think it should...

What about data created by government - which is us. Data about im- and export movements, data regarding employment, schooling, law enforcement, crime, etc.. 

Imagine....

What would this democratization of data mean with regard to innovation? I think it would truly ignite a burst of possibilities and a huge potential for our general wellbeing. And no - I am not referring to the challenge of marketing handbags to middle age ladies (quote somewhat paraphrased from Neil Raden).

No, set the data free to go for the real challenges we face; decreasing poverty, climate control, improving healthcare, scarcity of resources, economic stability and decreasing crime.

This blogpost is hugely inspired by John Wilbanks - google the guy (!) -, all the Open Data initiatives of the world where governmental agencies free up their data, the technological possibilities of data storage, data deployment, data enrichment, data visualization and advanced analytics and finally...this blog is inspired by a deeply felt wish and conviction that our field of knowledge (data management and data utilisation) can make a contribution to a better place for us to live in.

The biology of data, Conformers & Regulators

Imagine 'Data' being an organism, a living being existing in an environment we tend to call 'organization'. In the field of biology there are two types of organisms in such an environment; conformers and regulators.

These two types have different mechanisms as to how they regulate their life system parameters (e.g. temperature, pH, blood glucose), in other words, how they maintain homeostasis.

Those organisms that conform allow the environment to determine several important life system parameters. A reptile for example uses a sun-heated rock in the morning to raise its body temperature. So - conformers needs to change its behavior to maintain homeostasis.

Those that regulate, maintain life system parameters at a constant level over possibly wide ambient environmental variations. It has the capability to adjust its metabolism to maintain homeostasis. In other words; we - human beings - are regulators.

Now, imagine 'Data' being a regulator. This would mean that 'Data' is capable of adapting, self-sufficiently, to different environmental variations. Now, in several discussions regarding data - especially Big Data - I encounter an assumption that data by itself can have value, meaning and relevance. And if there are lots of it, its value is even bigger!

That is just so wrong….

'Data' is definitely a conformer - its environment determines its life system parameters. Its value, it’s meaning, its relevance is - by definition - determined by the environment.

Still with me? Lets continue.

What is left of 'Data' - being a conformer - when it cannot correctly detect its environment? Or even worse, what happens to 'Data' when the environment is void? The answer is simple; it can not sustain, even worse, it can not exist. 'Data' is profoundly in existential trouble. 

Data needs context
Data needs metadata
Data needs definitions 

TDWI EU - Agile Data Warehousing; What is the buzz about

June 18th, just after the demise of the Dutch soccer team, I gave a seminar at TDWI-EU in Munich. I think me, Jos van Dongen and Frank Buytendijk where the only Dutch guys (speakers as well as attendee) at a succesful conference with over 700 attendees. 

I had a blast and was impressed by the seminars given by Jos van Dongen, Barry Devlin and Mike Ferguson. I learned a hell of a lot and had great laughs with these guys as well. 

To top things of I had some vegan food in Munich with a bunch of Pentaho folks that almost convinced me to join an even darker side (instead of SAS....lol). If you guys are reading this; thx for a great evening and showing us some architecture in Munich. 

Oh and please - there is no correlation between Pentaho and vegan food. How a meat-loving guy like me ends up in a vegan restaurant deserves a separate blogpost. Suffice it to say that Barry Devlin was to be blamed ;-). The food was great though!

The seminar regarding Agile Data Warehousing had something like 50 attendees I think. Which is pretty ok, considering my competition as well as considering the fact that Data Vault is not very known (yet) in Germany. I had some great questions, especially from the guys/gallls who 'know' and have 'been in the trenches'. Lots of recognition and good questions and discussions were asked after the session and the days that followed.

It amazed me that large companies in Germany experimented already with Data Vault and data warehouse automation or were familiar with it and wanted to start a pilot. Beside the Scandinavian countries, Holland and Belgium; I think Germany will soon join and contribute to the community, which would be great.