Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)

DataDevOps:
CREATING A DATA &  
ANALYTICS CULTURE
AT SCALE

PL
S
RUS
UA
RO
CZ
D
NL
B
F
A
HR
I
E
BG
TR
18
COUNTRIES
2.4M+
CARS & MOTOS
10M+
USERS PER
MONTH

ROAD TO
MICROSERVICE
ARCHITECTURE

DataDevOps | Sean Gustafson & Arif Wider
HOW WE STARTED IN 2007
BI TOOL
2007
MIDDLE TIERWEB CORE DB
CRM
DWH
ANALYST
BI DEV
STAGING

APP
HOW THINGS GOT COMPLICATED IN 2011
BI TOOL
2011
$$$
APP
API
MIDDLE TIER
WEB
CORE DB
CRM
DWH
MYSQL
ANALYST
BI DEV
STAGING
2007

HADOOP
REST API
APP
MYSQL
BI TOOL
CORE DB
CRM
DWH
MYSQL
ANALYST
BI DEV
STAGING
HOW WE SLICED THE MONOLITH IN 2013
2013
APP
APP
WEB
SEA
API
API
API
API
SYNC
ELASTIC
DE
2011
2007

AWS
APP
HADOOP
REST API
BI TOOL
CORE DB
CRM
DWH
ANALYST
BI DEV
STAGING
APP
WEB
EXP SEA
API
API
API
API
SYNC
MONGO ELASTIC
DE
HOW A CENTRAL DATA TEAM DOESN’T SCALE
APP APP
2015
2013
2011
2007
APP
APP
APP
MYSQL
MYSQL
MYSQL
MYSQL APP

AWS
CENTRAL DATA LAKE ON S3
AWS
HOW WE REARCHITECTURED OUR DATA LANDSCAPE
2017
BI DEV
DE
CORE DB
CORE DBAPI
API
API
CRM
APP
APP
APP
APP
APP
APP
APP
APP
REST API
2015
2013
2011
2007
BI TOOL
DWH
ANALYST

SCOUT24 WANTS  
TO BECOME A TRULY
DATA-DRIVEN COMPANY

AC
T
PL
AN
D
O
CH
ECK
Fast & easy data-driven 
product development…
…supported by  
Data & Analytics

Everywhere in the company... ...without bloating up D‘n‘A
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
AN
D
O
CH
ECK
AC
T
PL
ANDO
CH
ECK

SCOUT24 
DATA LANDSCAPE MANIFESTO
ROLES, RESPONSIBILITIES, AND VALUES  
FOR A DATA-DRIVEN COMPANY AT SCALE

SCOUT24 DATA LANDSCAPE MANIFESTO
#1 Preamble
Data is a key asset of our company.

#2 Central Data Team’s Responsibility
We, Data & Analytics, are responsible for
providing a solid Data Platform as well
as clear guidelines and training how to
participate in the Data Landscape.  
D’n’A
DATA PLATFORM
DATA LANDSCAPE

#3 Data Autonomy, Not Anarchy
Data autonomy puts data producers &
data consumers in control of their data
& of their metrics and thereby allows
us to be data-driven at scale, but this
comes with responsibility.
M
ETRIC
CONSUMER
D’n’A
DATA PLATFORM
DATA LANDSCAPE
DATA
PRODUCER

AWSCENTRAL DATA LAKE ON S3
ROLES & RESPONSIBILITIES
DATA CATALOG
D’n’A
CHECKOUT 
SERVICE
PRODUCER
SPECIAL 
OFFER 
SERVICE
CONSUMER

#4 Producer’s Responsibility
Data producers are responsible for
publishing data to the central Data
Lake, for the data's quality, and for
publishing metadata that makes it
easy to find and consume the data.
D’n’A
DATA PLATFORM
DATA LANDSCAPE
DATA
METADATA
PRODUCER

DATA CATALOG
D’n’A
ORDER EVENTS
EVENT METADATA
CHECKOUT 
SERVICE
PRODUCER
SPECIAL 
OFFER 
SERVICE
CONSUMER

ORDER EVENTS
EVENT METADATA
CHECKOUT 
SERVICE
DATA CATALOG
PRODUCER
D’n’A
INGESTION TEMPLATE
SPECIAL 
OFFER 
SERVICE
CONSUMER

#5 Consumer’s Responsibility
Data consumers are responsible for the
definition & visualization of metrics and
for driving the implementation and
maintenance of these metrics.
M
ETRIC
CONSUMER
D’n’A
DATA PLATFORM
DATA LANDSCAPE
PRODUCER

ORDER EVENTS
EVENT METADATA
CHECKOUT 
SERVICE
DATA CATALOG
PRODUCER
SPECIAL 
OFFER 
SERVICE
CONSUMER
D’n’A
INGESTION TEMPLATE VIEW: ORDER HISTORY BY USER

#6 Transparency Over Continuity
We value data transparency over data
continuity, which means we may
break metric comparability if it is for
the cause of enabling better insights.
M
ETRIC
CONSUMER
D’n’A
DATA PLATFORM
CORE
METRIC
DATA LANDSCAPE
PRODUCER

The Ultimate Goal
A federal landscape of data producers
and consumers with just enough norms
to ensure seamless co-operation
without severely impeding autonomy. M
ETRIC
CONSUMER
D’n’A
DATA PLATFORM
DATA LANDSCAPE
DATA
METADATA
PRODUCER

Consequences for Product Teams
‣ Think about data & reporting
‣ Deliver your data to the lake
‣ Provide meta data
‣ Eat your own dog food: Consume your own data

Benefits for Product Teams
‣ Independently work with data
‣ No dependencies to data teams
‣ It’s easy to consume data produced by other teams
‣ Faster product & measurement iterations

How to convince everyone to go along?
‘Nudge’ them to participate
Promote the platform
Refuse new use cases in Data Warehouse

Design ‘nudges’ into the Platform
Make Data Lake easier than something else:
‣ automatic table publishing, partition detection
‣ backup and disaster recovery
‣ access control for restricted data
‣ optimize file formats (e.g. parquet) for efficiency

Learnings and lessons
‣ Change needs to be technological, organizational and cultural
‣ Build features to give benefits that counteract resistance
‣ Communication is the key

MOST IMPORTANTLY:
Have a strong opinion about how the
company should use data and build a
platform that pushes toward that vision.

Dr. Sean Gustafson & Dr. Arif Wider
@seangustafson @arifwider
THANKS

Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)

Similar to Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live) (20)

More from Thoughtworks

More from Thoughtworks (20)

Recently uploaded

Recently uploaded (20)

Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)