It is often not intuitive to use agile and lean startup thinking in the world of big data and analytics. You can feel far removed from users and customers and your products often have to enable people to answer open-ended questions to help solve complex business problems. Often you are the engine room of very large enterprises. We will discuss techniques and practices that we can use to stay agile under these conditions. Please note, this talk focuses more on ways of working than technical aspects. Target Audience: People in data-centric teams (warehousing, BI, analytics) or anyone with an interest in agile principles being applied.

1. You’ve Got No UI!?
(Agile Data Teams)
Mark Barber, Agile Coach
@mark_barbs

3. A (not so) long time ago…
• Previously I have been a Delivery Lead / Coach
for data engineering teams (warehousing, BI,
analytics)
• Mythbusting that agile and lean startup
principles are just for the teams with buttons
on a screen

4. A (not so) long time ago…
• Solving an open ended problem to learn about
our customers using data
• With data no one really knows about
• And no flashy UI to impress your friends

5. Obstacle One: Where are we going?

6. What was the problem?
• New team, new problems, new technology
• Large number of untested ideas
• What does success look like?!
Obstacle One: Where are we going?

7. What We Did
• Embedded product manager
• Team inception (weeks, not hours)
• Experiments in production
Obstacle One: Where are we going?

8. Why It Worked
• Shared vision, owned by the team
• Measurable success criteria
• Validated ideas with little investment
Business people and developers must work together daily
throughout the project.
Build projects around motivated individuals. Give them the
support they need and trust them to get the job done.
Obstacle One: Where are we going?

9. Obstacle Two: But we want BIG data!

10. What was the problem?
• Exciting new tech, Netflix is doing it!
• We’ve got data, it MUST be big
• Preconceived implementation leads to poor
decision making
Obstacle Two: But we want BIG data!

11. What We Did
• Simplest solution first (YAGNI)
• Not-So-Big Data
• Queries over My SQL before Apache Spark
• Squeeze all we could out of R
Obstacle Two: But we want BIG data!

12. Why it worked
• Chose the best toolset for the job
• No overcomplicated solutions
• Failed fast and learnt fast
Simplicity, the art of maximising the amount of work not
done, is essential
Obstacle Two: But we want BIG data!

13. Obstacle Three: Data processing is ops heavy

14. What was the problem?
• Experimenting quickly requires short-lived
environments, we needed massive memory
allocations before good design took hold, and
we’d heard tales of hardware woe
• Relying on external ops teams would slow us
down a lot
Obstacle Three: Data processing is ops heavy

15. What We Did
• Hired for devops without exception
• Team owned the end-to-end AWS envs
• Open source over proprietary software
Obstacle Three: Data processing is ops heavy

16. Why it worked
• The team owned the infrastructure and
treated it like all code
• No enterprise software licencing kept us free
from technical obligations
Continuous attention to technical excellence and good
design enhances agility
Barber’s Law: External dependencies will slow you down
Obstacle Three: Data processing is ops heavy

17. Obstacle Four: Mysterious algorithms

18. What was the problem?
• Black box algorithms make testing and
adapting difficult
• External dependencies on data scientists and
analysts when skills aren’t in the team
Obstacle Four: Mysterious algorithms

19. What We Did
• Formed cross-functional teams with data and
statistical analysts (the “Frankenstein Data
Scientist” and the 7 people you need on your
data team by Ian Thomas)
• Devs and analysts paired on modelling
Obstacle Four: Mysterious algorithms

20. Why it worked
• Removed external dependencies
• Analysts got immediate feedback
• Developers learnt about modelling
The best architectures, requirements, and designs
emerge from self organising teams
Obstacle Four: Mysterious algorithms

21. Obstacle Five: Too much up front infrastructure

22. What was the problem
• Too much time and effort before putting
something in front of a customer.
• “Big Upfront Infrastructure”
• Early optimisation, potential waste
Obstacle Five: Too much up front infrastructure

23. What We Did
• Collaborative story mapping
• Tracer bullet releases
Obstacle Five: Too much up front infrastructure

24. Why it worked
• Story mapping visualised we were focusing
efforts in the wrong places
• Thin releases with fast feedback loops helped
us build the right thing
Our highest priority is to satisfy the customer through early
and continuous delivery of valuable software.
Working software is the primary measure of progress.
Obstacle Five: Too much up front infrastructure

25. Obstacle Six: Data quality is difficult to monitor

26. What was the problem?
• With billions of rows of calculations do you
want to assert on every one?
• Without monitoring for every user, how
confident could we be that the data was
correct? It reduced confidence when making
changes
Obstacle Six: Data quality is difficult to monitor

27. What We Did
• Monitor TRENDS at the GRANULARITY that
matters
• Visualise and put them on screens
• Monitor upstream and downstream
• Testing face-to-face with users
Obstacle Six: Data quality is difficult to monitor

28. Why it worked
• Visualisations in the team space put quality as
a focal point for the team
• Baselines gave us data around how much we
were impacting customers with changes
• Team learnt about our users
Continuous attention to technical excellence and good
design enhances agility.
Welcome changing requirements, even late in development
Obstacle Six: Data quality is difficult to monitor

29. Obstacle Seven: Everybody wants the data!

30. What was the problem?
• Great insights lead to great demand on the
teams generating them
• Becoming an operational system will lead to
strict SLAs and reluctance to change
• Constant prioritisation, long lead time in the
value stream, more failure demand work
Obstacle Seven: Everybody wants the data!

31. What We Did
• Built platforms that allowed teams to build
insights without breaking other systems
• Batched data generation and let downstream
consumers take on operational SLAs
Obstacle Seven: Everybody wants the data!

32. Why it worked
• Freed the team up to focus on delivering on
our own goal and allowed other teams to
deliver more value to customers
• Focus on value demand work
Continuous attention to technical excellence and good
design enhances agility
Obstacle Seven: Everybody wants the data!

33. Key take aways
• Set direction early, and collaborate closely
with product, analytics, development and
anyone else needed to solve the problem
• Validate ideas with minimal investment in time
and effort and TALK to your CUSTOMERS
• Actively monitor quality over reactive alerts
• Build a platform for other teams to extend
• Work to remove external dependencies
• Keep it simple

34. Thank You!
Mark Barber
Agile Coach @ MYOB (we’re hiring)
@mark_barbs