How can infrastructure engineers empower their product developers with easy-to-use systems and processes that abstract the complexity of core infrastructure? This talk focuses on Envoy configuration management, and how the networking team at Lyft builds on top of Envoy to allow Lyft engineers to focus on business logic.

1. Envoy @ ft
José Niño
jnino@lyft.com - @junr03

2. Who am I?
Envoy
Maintainer
Networking
Team @ Lyft
@junr03

3. My time at Lyft
1. Initial Envoy open sourcing: documentation, and docker sandbox examples
2. Create Envoyoutbound: enable developers to easily communicate with partners
over stable IPs
3. Open sourcing ratelimit, and a couple other golang libraries: provide ample
documentation for consumers
4. Expand Envoy’s outlier detection system, and build tooling (stats, logging) to
help developers understand anomalies in their services
5. xDS APIs and the future of Envoy configuration management at Lyft: how do we
make the control plane accessible and easy to use
@junr03

4. There is a pattern...
1. Open sourcing envoy: documentation, and docker sandbox examples
2. Create Envoyoutbound: enable developers to easily communicate with
partners over stable IPs
3. Open sourcing ratelimit, and a couple other golang libraries: provide ample
documentation for consumers
4. Expand Envoy’s outlier detection system, and build tooling (stats, logging) to
help developers understand anomalies in their services
5. xDS APIs and the future of Envoy configuration management at Lyft: how do we
make the control plane accessible and easy to use
The focus is on developer productivity!
@junr03

5. The Story
Envoy is a powerful and complex tool.
How does the Networking Team at
Lyft hide the complexity to allow
service developers to leverage the
power of Envoy?
@junr03

6. Why is this important?
• Lyft engineers are the Infra org’s customers
• Lyft is about to have a lot more engineers
• The number of services at Lyft is ever increasing
@junr03

7. Envoy @ Lyft Mesh
Front Envoy
Envoyoutbound
Tracing Collectors Ratelimit Discovery
Ancillary Services
> 200 services
> 20,000 Hosts
> 5 million RPS
@junr03

8. Frame of Reference - The Control Plane
• Proxy configuration is complicated: envoy is not the exception
• Operating the data plane should be reserved to a select few
• Configuring some options of the data plane via the control plane should be
open to all service owners
@junr03

9. Configuration Management - The Past
Initially static files
‒ Only two types: edge proxy, service sidecar
‒ Deployed on a deploy bundle out to the edge proxy, and to all services in the mesh
Human Static Files
“Deploy
Magic”
Proxies
@junr03

10. Configuration Management - The Past
As complexity grew we moved to templated files
‒ Jinja2 templates, and some python glue
‒ Expose certain “knobs” to the service engineers at Lyft
‒ At deploy time, create the configuration file
Human
Exposed
Knobs
“Deploy
Magic”
Proxies
Jinja2
Templates
+
@junr03

11. Use case: create a new public route
• Service developers manipulate edge proxy route table
• Deploying public routing changes was tied to an Envoy binary deployment
• Erroneous configuration could be deployed next to complex code
Front Envoy
/new/route
New Service
@junr03

12. Pain points
• Configuration deployment was tied to binary deployment
• UX is tedious and fragmented
The Complexity is in Plain Sight
@junr03

13. Configuration Management - The Present
Mid 2017: xDS APIs for configuration management.
• gRPC/protobuf based
• Bi-directional gRPC streaming
• Interacting with the control plane is separated from data plane operation
• Enable us to develop smart, robust control plane solutions
RDS - Route Discovery Service
CDS - Cluster DS
LDS - Listener DS
...
@junr03

14. Configuration Management - The Present
Envoymanager
/ /
service
deployment
envoy-static-config
service
“manifest”
Document
Cloud Storage
@junr03

15. Configuration Management - The Present
envoy-static-config
service
“manifest”
match:
path: /rider/
route:
cluster: pagelauncher
@junr03

16. Configuration Management - The Present
internal_hosts:
- jobscheduler
- roads
external_hosts:
- dynamodb_iad
- kinesis_iad
circuit_breaker:
default:
max_requests: 100
envoy-static-config
service
“manifest”
@junr03

17. Configuration Management - The Present
service
deployment
Document
Cloud Storage
@junr03

18. Caching
Configuration Management - The Present
Envoymanager
/ /
Data
processing
xDS
Server
@junr03

19. Configuration Management - The Present
Envoymanager
/ /
service
deployment
envoy-static-config
service
“manifest”
Document
Cloud Storage
@junr03

20. Envoy @ Lyft Mesh
Front Envoy
Envoyoutbound
Tracing Collectors Ratelimit Discovery Envoymanager
Ancillary Services
@junr03

21. How is the complexity
hidden?
@junr03

22. Use case: create a new public route
Envoymanager
envoy-static-config
Document
Cloud Storage
@junr03

23. Documentation
• Documentation built on top of the public Envoy documentation
• Clear step-by-step guides
• FAQs
• Video Tutorials
@junr03

24. Making a Change

25. Deployment
• Same mechanics as service deployments
• Easy to use deployment pipeline
• Canary is part of the deployment process
@junr03

26. Deployment
• Same mechanics as service deployments
• Easy to use deployment pipeline
• Canary is part of the deployment process
@junr03

27. Versioning
service
deployment
envoy-static-config
service
“manifest”
Document
Cloud Storage
• Leverage git as a versioning system
• Easy rollback and roll forward
• Git shas have semantic meaning
• Versions are used throughout the
system
• Used in monitoring tooling
@junr03

28. Stats - Envoymanager
@junr03

29. Stats - Front Envoy
@junr03

30. Stats - Per Service Metrics

31. Wins
• Allows service developers to own configuration changes all the way to
production
• Most configuration changes do not entail an envoy restart
• Most configuration changes do not entail an envoy binary deploy
• Opens up the world to more friendly UX for configuration changes
@junr03

32. The Future
@junr03

33. The networking team focuses on building
accessible and easy-to-use systems for
service developers to successfully
configure, operate, and debug Envoy
@junr03

34. Thanks - Q&A
jnino@lyft.com - @junr03