In a world of seamless deployments to auto-scaling cloud environments, a ThoughtWorks team found itself in a very different place - trying to deploy a RESTful pricing API to every one of a UK supermarket’s 40,000 tills in a reliable, repeatable fashion.

1. A Different Type of
Supermarket Delivery
Phil Jenkins

2. About Me
● Developer
● 14 years of coding
● Career motto: “Try anything once”
● When not coding, I like to cycle

3. Background

4. The Client
One of the largest supermarkets in the UK
● £Billions in revenue
● 1000s of UK stores
● Online / In store channels

5. Client Vision

6. Enterprise APIs
● Take ownership of business logic
● Standardise
● Replace the till
● Flexibility for the customer
Motivation

7. Enterprise APIs
Pricing / Quoting Identity Customer
Segregation by business function
BUSINESS CRITICAL SERVICES

8. Enter ThoughtWorks

9. Enterprise APIs
Cloud-based services
Pricing / Quoting Identity Customer

10. Enterprise APIs
Cloud-based services
Pricing / Quoting Identity Customer

11. Pricing API
Tech Stack

12. Pricing API
Initial solution
HTTP
Events
Store

13. However...
Store

14. However...
● Network performance is poor
○ 60 mins downtime per store per month
● Most stores have a single point of failure
● The customer must ALWAYS be able to check out
THE ONLY PLACE FOR BUSINESS CRITICAL SERVICES
IS THE TILL ITSELF

15. Plan
● Till ALWAYS talks to a local Pricing API
● Events are stored in embedded DB
● Background sync to central
● Run in parallel

17. Store
Offline

18. Online
Sync Events
Store

19. 40000

20. Rule of Tills
For 𝑖=1,…, 𝑛
40000i = A big number

21. Rule of Tills
● 15MB of logs per till per day
● 15MB x 40000 = ~ 600GB logs per day

22. Tills

26. Deployment

27. Methods
“Till Build”
OS and core applications installed by engineer on site Slow release cycle
Enterprise Software Management
CA DSM / Nolio etc. No standard tool
“Spider”

28. Methods
Written by Till Deployment team
Already running in till estate
P2P distribution in store
“Spider”

30. The End

31. Pain

32. App
Store Store
Store
Store
StoreStore
Store
Store
Store
Store

36. Every Morning

40. Rethink

43. Service
HTTP

44. Service

45. Service

46. Service

47. Guiding Principles
Central Service
● Logic goes here
● Scalable
● Cheap
Till Clients
● Low overhead
● Convention-based
● Bootstrap once
● Self-updating

48. Lambda
API Gateway
Apigee
S3HTTP Endpoint
Rate Limiting
Artefacts and
Manifest
Application Logic

49. Manifest
Till Type
Till
Store
Mapping application versions to
devices

50. Manifest
{
stores: {
A: 1.5,
B: 1.4
}
}
{
store: A,
till: 2,
version: 1.0
}
Service

51. 302
https://url/v1.5.zip
Manifest
{
stores: {
A: 1.5,
B: 1.4
}
}
Service

52. Manifest
{
stores: {
A: 1.5,
B: 1.4
}
}
{
store: A,
till: 2,
version: 1.5
}
Service

53. 304
Manifest
{
stores: {
A: 1.5,
B: 1.4
}
}
Service

56. Batch
scripts
Scheduled Task
Tools
(dunzip, fciv)
curl
HTTP Client
Orchestration
Execution
Unzipping,
Checksumming

57. Rule of Tills
For 𝑖=1,…,𝑛
40000i = A big number

58. Rule of Tills
● Pricing API: 40MB
● 40000 x 40MB = 1.6TB to distribute!
● Bandwidth considerations
○ 500 Mbps centrally
○ 6 Mbps in store

59. 03:00 -
04:00 -
05:00 -
06:00 -
07:00 -
08:00 -
18:00 -
19:00 -
20:00 -
21:00 -
22:00 -
23:00 -
Scheduling
00:00 -
01:00 -
02:00 -

60. Deltas
● Only send changes
● Explode JARs
● Diff in Lambda
● > 40x smaller!
V1.0 V1.1
V1.0 --> V1.1

62. Success!

63. Lessons

64. 1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous L Peter Deutsch
So true...
Fallacies of Distributed Computing

65. !??

66. Back to Basics
● Be pragmatic!
● Do one thing well
● Concepts over tools

67. Till the next time
Thank you