Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
You’ve Got No UI!?
(Agile Data Teams)
Mark Barber, Agile Coach
@mark_barbs
A (not so) long time ago…
• Previously I have been a Delivery Lead / Coach
for data engineering teams (warehousing, BI,
an...
A (not so) long time ago…
• Solving an open ended problem to learn about
our customers using data
• With data no one reall...
Obstacle One: Where are we going?
What was the problem?
• New team, new problems, new technology
• Large number of untested ideas
• What does success look l...
What We Did
• Embedded product manager
• Team inception (weeks, not hours)
• Experiments in production
Obstacle One: Where...
Why It Worked
• Shared vision, owned by the team
• Measurable success criteria
• Validated ideas with little investment
Bu...
Obstacle Two: But we want BIG data!
What was the problem?
• Exciting new tech, Netflix is doing it!
• We’ve got data, it MUST be big
• Preconceived implementa...
What We Did
• Simplest solution first (YAGNI)
• Not-So-Big Data
• Queries over My SQL before Apache Spark
• Squeeze all we...
Why it worked
• Chose the best toolset for the job
• No overcomplicated solutions
• Failed fast and learnt fast
Simplicity...
Obstacle Three: Data processing is ops heavy
What was the problem?
• Experimenting quickly requires short-lived
environments, we needed massive memory
allocations befo...
What We Did
• Hired for devops without exception
• Team owned the end-to-end AWS envs
• Open source over proprietary softw...
Why it worked
• The team owned the infrastructure and
treated it like all code
• No enterprise software licencing kept us ...
Obstacle Four: Mysterious algorithms
What was the problem?
• Black box algorithms make testing and
adapting difficult
• External dependencies on data scientist...
What We Did
• Formed cross-functional teams with data and
statistical analysts (the “Frankenstein Data
Scientist” and the ...
Why it worked
• Removed external dependencies
• Analysts got immediate feedback
• Developers learnt about modelling
The be...
Obstacle Five: Too much up front infrastructure
What was the problem
• Too much time and effort before putting
something in front of a customer.
• “Big Upfront Infrastruc...
What We Did
• Collaborative story mapping
• Tracer bullet releases
Obstacle Five: Too much up front infrastructure
Why it worked
• Story mapping visualised we were focusing
efforts in the wrong places
• Thin releases with fast feedback l...
Obstacle Six: Data quality is difficult to monitor
What was the problem?
• With billions of rows of calculations do you
want to assert on every one?
• Without monitoring for...
What We Did
• Monitor TRENDS at the GRANULARITY that
matters
• Visualise and put them on screens
• Monitor upstream and do...
Why it worked
• Visualisations in the team space put quality as
a focal point for the team
• Baselines gave us data around...
Obstacle Seven: Everybody wants the data!
What was the problem?
• Great insights lead to great demand on the
teams generating them
• Becoming an operational system ...
What We Did
• Built platforms that allowed teams to build
insights without breaking other systems
• Batched data generatio...
Why it worked
• Freed the team up to focus on delivering on
our own goal and allowed other teams to
deliver more value to ...
Key take aways
• Set direction early, and collaborate closely
with product, analytics, development and
anyone else needed ...
Thank You!
Mark Barber
Agile Coach @ MYOB (we’re hiring)
@mark_barbs
You've Got No UI?! (Agile Data Teams)
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Measuring for team effectiveness (NEW)
Next
Download to read offline and view in fullscreen.

Share

You've Got No UI?! (Agile Data Teams)

Download to read offline

It is often not intuitive to use agile and lean startup thinking in the world of big data and analytics. You can feel far removed from users and customers and your products often have to enable people to answer open-ended questions to help solve complex business problems. Often you are the engine room of very large enterprises. We will discuss techniques and practices that we can use to stay agile under these conditions.

Please note, this talk focuses more on ways of working than technical aspects.

Target Audience: People in data-centric teams (warehousing, BI, analytics) or anyone with an interest in agile principles being applied.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

You've Got No UI?! (Agile Data Teams)

  1. 1. You’ve Got No UI!? (Agile Data Teams) Mark Barber, Agile Coach @mark_barbs
  2. 2. A (not so) long time ago… • Previously I have been a Delivery Lead / Coach for data engineering teams (warehousing, BI, analytics) • Mythbusting that agile and lean startup principles are just for the teams with buttons on a screen
  3. 3. A (not so) long time ago… • Solving an open ended problem to learn about our customers using data • With data no one really knows about • And no flashy UI to impress your friends
  4. 4. Obstacle One: Where are we going?
  5. 5. What was the problem? • New team, new problems, new technology • Large number of untested ideas • What does success look like?! Obstacle One: Where are we going?
  6. 6. What We Did • Embedded product manager • Team inception (weeks, not hours) • Experiments in production Obstacle One: Where are we going?
  7. 7. Why It Worked • Shared vision, owned by the team • Measurable success criteria • Validated ideas with little investment Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the support they need and trust them to get the job done. Obstacle One: Where are we going?
  8. 8. Obstacle Two: But we want BIG data!
  9. 9. What was the problem? • Exciting new tech, Netflix is doing it! • We’ve got data, it MUST be big • Preconceived implementation leads to poor decision making Obstacle Two: But we want BIG data!
  10. 10. What We Did • Simplest solution first (YAGNI) • Not-So-Big Data • Queries over My SQL before Apache Spark • Squeeze all we could out of R Obstacle Two: But we want BIG data!
  11. 11. Why it worked • Chose the best toolset for the job • No overcomplicated solutions • Failed fast and learnt fast Simplicity, the art of maximising the amount of work not done, is essential Obstacle Two: But we want BIG data!
  12. 12. Obstacle Three: Data processing is ops heavy
  13. 13. What was the problem? • Experimenting quickly requires short-lived environments, we needed massive memory allocations before good design took hold, and we’d heard tales of hardware woe • Relying on external ops teams would slow us down a lot Obstacle Three: Data processing is ops heavy
  14. 14. What We Did • Hired for devops without exception • Team owned the end-to-end AWS envs • Open source over proprietary software Obstacle Three: Data processing is ops heavy
  15. 15. Why it worked • The team owned the infrastructure and treated it like all code • No enterprise software licencing kept us free from technical obligations Continuous attention to technical excellence and good design enhances agility Barber’s Law: External dependencies will slow you down Obstacle Three: Data processing is ops heavy
  16. 16. Obstacle Four: Mysterious algorithms
  17. 17. What was the problem? • Black box algorithms make testing and adapting difficult • External dependencies on data scientists and analysts when skills aren’t in the team Obstacle Four: Mysterious algorithms
  18. 18. What We Did • Formed cross-functional teams with data and statistical analysts (the “Frankenstein Data Scientist” and the 7 people you need on your data team by Ian Thomas) • Devs and analysts paired on modelling Obstacle Four: Mysterious algorithms
  19. 19. Why it worked • Removed external dependencies • Analysts got immediate feedback • Developers learnt about modelling The best architectures, requirements, and designs emerge from self organising teams Obstacle Four: Mysterious algorithms
  20. 20. Obstacle Five: Too much up front infrastructure
  21. 21. What was the problem • Too much time and effort before putting something in front of a customer. • “Big Upfront Infrastructure” • Early optimisation, potential waste Obstacle Five: Too much up front infrastructure
  22. 22. What We Did • Collaborative story mapping • Tracer bullet releases Obstacle Five: Too much up front infrastructure
  23. 23. Why it worked • Story mapping visualised we were focusing efforts in the wrong places • Thin releases with fast feedback loops helped us build the right thing Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Working software is the primary measure of progress. Obstacle Five: Too much up front infrastructure
  24. 24. Obstacle Six: Data quality is difficult to monitor
  25. 25. What was the problem? • With billions of rows of calculations do you want to assert on every one? • Without monitoring for every user, how confident could we be that the data was correct? It reduced confidence when making changes Obstacle Six: Data quality is difficult to monitor
  26. 26. What We Did • Monitor TRENDS at the GRANULARITY that matters • Visualise and put them on screens • Monitor upstream and downstream • Testing face-to-face with users Obstacle Six: Data quality is difficult to monitor
  27. 27. Why it worked • Visualisations in the team space put quality as a focal point for the team • Baselines gave us data around how much we were impacting customers with changes • Team learnt about our users Continuous attention to technical excellence and good design enhances agility. Welcome changing requirements, even late in development Obstacle Six: Data quality is difficult to monitor
  28. 28. Obstacle Seven: Everybody wants the data!
  29. 29. What was the problem? • Great insights lead to great demand on the teams generating them • Becoming an operational system will lead to strict SLAs and reluctance to change • Constant prioritisation, long lead time in the value stream, more failure demand work Obstacle Seven: Everybody wants the data!
  30. 30. What We Did • Built platforms that allowed teams to build insights without breaking other systems • Batched data generation and let downstream consumers take on operational SLAs Obstacle Seven: Everybody wants the data!
  31. 31. Why it worked • Freed the team up to focus on delivering on our own goal and allowed other teams to deliver more value to customers • Focus on value demand work Continuous attention to technical excellence and good design enhances agility Obstacle Seven: Everybody wants the data!
  32. 32. Key take aways • Set direction early, and collaborate closely with product, analytics, development and anyone else needed to solve the problem • Validate ideas with minimal investment in time and effort and TALK to your CUSTOMERS • Actively monitor quality over reactive alerts • Build a platform for other teams to extend • Work to remove external dependencies • Keep it simple
  33. 33. Thank You! Mark Barber Agile Coach @ MYOB (we’re hiring) @mark_barbs
  • DeniseMangan1

    Aug. 2, 2016
  • Queenouassila

    Jul. 5, 2016

It is often not intuitive to use agile and lean startup thinking in the world of big data and analytics. You can feel far removed from users and customers and your products often have to enable people to answer open-ended questions to help solve complex business problems. Often you are the engine room of very large enterprises. We will discuss techniques and practices that we can use to stay agile under these conditions. Please note, this talk focuses more on ways of working than technical aspects. Target Audience: People in data-centric teams (warehousing, BI, analytics) or anyone with an interest in agile principles being applied.

Views

Total views

861

On Slideshare

0

From embeds

0

Number of embeds

38

Actions

Downloads

8

Shares

0

Comments

0

Likes

2

×