Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k1eZ00. Chris Roche and Christopher Burnett discuss how they extended the Protocol Buffer (PB) IDL to create unified APIs and data models. From validation logic to automatic logging and statistics, PBs allows them to speed up development across Go, Python, and PHP stacks. They also show how they leverage the Envoy proxy to transparently upgrade HTTP/1.1 services to speak gRPC on the wire. Filmed at qconnewyork.com. Chris Roche is a Core Libraries Engineer at Lyft focused on enriching the tools to create distributed systems. Christopher Burnett is Lead Core Libraries Engineer at Lyft, where he designs software for distributed systems.

1. TYPES ALL THE WAY DOWN
GENERATING APIS WITH PROTOBUFS AND GRPC

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
api-pb-grpc

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. HI!
Christopher Burnett - Lead Core Libraries
Formerly Twitter, VSCO
@twoism
Chris Roche - Core Libraries
Formerly VSCO
@rodaine

5. CORE LIBRARIES
A little about us…
• HTTP & gRPC Frameworks
• Tooling for Go, Python, and PHP
• Rolling out gRPC within Lyft

6. AGENDA
• Why choose RPC in the first place?
• Working with legacy systems
• Disrupting workflows, nicely :)
• What we’ve built

7. EVERY TEN YEARS…
A furious bout of language and protocol design takes place and a new
distributed computing paradigm is announced that is compliant with the
latest programming model.
- A Note On Distributed Computing, Waldo 1994

8. LOOK FAMILIAR?
• CORBA
• Thrift
• SOAP
• WDDX
• JSON-RPC
• XML-RPC
• Avro
• HyperMedia
• REST
• MessagePack

10. A LITTLE HISTORY
Like any good story we begin with a PHP
monolith…
• Active decomp efforts
• 100s of Python Microservices
- Flask HTTP/REST
And to keep things interesting…
• gRPC Core and Compositional Services

11. DEFINING A CORE SERVICE
• Organizational Primitives
- User, Rides, Vehicles
• Zero (Service) Dependencies
- Databases, caches, etc
• Highly Performant

12. SO, WHY GRPC?

13. LET’S TALK ABOUT REST

14. RESTful

15. RESTish

16. REST/JSON: S2S COMMUNICATION
POST /api/updateUser HTTP/1.0
Content-Type: application/json
{
"id": 18446744073709551615,
"username": “chris"
}

17. ALRIGHT, LET’S PAINT THAT SHED…
PUT /api/users HTTP/1.0
Content-Type: application/json
{
"id": 18446744073709551615,
"username": “chris"
}

18. PUTTING ON ANOTHER COAT…
PUT /api/users/18446744073709551615 HTTP/1.0
Content-Type: application/json
{
"username": “chris"
}

19. FINISHING TOUCHES…
PUT /api/v1/users/18446744073709551615 HTTP/1.0
Content-Type: application/json
{
"username": “chris"
}

20. IDLs are pretty great :)

21. IDLS ARE PRETTY GREAT
• Single Source of Truth
-Primitive definitions
• Code Generation
-APIs, Clients, Servers, Data Models, Docs, Observability
• Extensibility
-Plugins for everything else

22. IDL SERVICE DEFINITION
package lyft.service.users.v1
service Users {
rpc Update(UpdateRequest) UpdateResponse;
}
message UpdateRequest {
uint64 id = 1;
string name = 2;
}

23. What about existing
services?

24. IDL SERVICE DEFINITION — HTTP
package lyft.service.users.v1
service Users {
option (http.http_server_options).isHttpServer = true;
rpc Update(UpdateRequest) returns UpdateResponse {
// Override `path` for legacy URL support
option (http.http_options).path = "/api/v1/users/:id";
option (http.http_options).method = "PUT";
}
}

25. TYPES ON THE WIRE
• Simplified API I/O
- Structs In, Structs Out
• Safety
- Big wins for dynamic languages
• Transfer Cost
- Improved latencies

26. ENVOY TOPOLOGY IMAGE

27. Types on the wire eliminate
an entire class of errors

28. TypeError: unsupported operand
type(s) for -: 'NoneType' and 'float'

29. HTTP/2.0

30. HTTP/2.0
• Full Duplex Streaming
• Binary Transport
• Push
• Header Compression

31. WHAT’S NOT SO GREAT?

32. Introducing a new protocol or
language can be highly
traumatic for teams.

33. “How do I cURL this?”

34. WHAT CAN MAKE THIS BETTER?
• Incremental Adoption
-Allow teams to opt-in to the new shiny things
• Familiarity
-Tooling that feels welcoming
-Standardized framework patterns
• Roll Forward
-Wire format first, then the protocol and frameworks

35. How can we make protocol and
infra changes flexible and
transparent?

37. Service
Mesh
ENVOY TOPOLOGY
Internet
Clients
“Front” Envoy
Legacy PHP
Monolith
Envoy
Go Services
Python Services
Envoy
Envoy
MongoDB /
DynamoDB
Ratelimiting
Stats /
Tracing
Discovery

38. SIDECAR?

40. ENVOY: GRPC BRIDGE FILTER
Python HTTP
Envoy
Envoy
HTTP/1
POST /GetUser
200 OK
gRPC
gRPCBridge

41. ENVOY: JSON PROXY FILTER
Python HTTP
Envoy
Envoy
HTTP/1
POST /GetUser
200 OK
gRPC
JSONFILTER

42. With Envoy, services and clients
can evolve independently.

43. *Next Chris

44. How can we leverage
IDLs beyond the API?

45. ODIE: IDLS MEET THE DATASTORE
Type-Safe Repository
Driver-Agnostic Client
Expression
Engine
Decorator
Middleware
Database
Driver
IDL-Based
Model
Lifecycle
Events
MongoDB
DynamoDB
Spanner
BoltDB

46. ODIE: MODELS AS PROTOCOL BUFFERS
message User {
option (odie.mongo).enabled = true;
string id = 1 [(odie.mongo).primary = true,
(odie.type).object_id = true];
string name = 2 [(odie.mongo).name = "username"];
int64 date = 3 [(odie.type).datetime = true];
uint32 vers = 4 [(odie.locking).revision = true];
}

47. ODIE: MODELS AS PROTOCOL BUFFERS
type UserModel struct {
Id bson.ObjectId `bson:"_id"`
Name string `bson:"username"`
Date time.Time
Vers uint32
}
func (pb *User) ToModel() *UserModel
func (m *UserModel) ToProto() *User

48. ODIE: TYPE-SAFE REPOSITORIES
type UserRepo interface {
Events() *Events
Get(ctx context.Context, id bson.ObjectId) *GetBuilder
Put(ctx context.Context, m *UserModel) *PutBuilder
Delete(ctx context.Context) *DeleteBuilder
Update(ctx context.Context) *UpdateBuilder
Query(ctx context.Context) *QueryBuilder
}

49. ODIE: THE PLATONIC IDEAL
• Zero/Empty value ≅ Nil value
- False is semantically the same as null
• Fields are not polymorphic
- All attributes are singularly typed
• Models are rigidly typed
- Legacy data is already mappable

50. ODIE: THE REALITY
• Zero/Empty value ≅ Nil value
- False is semantically the same as null
• Fields are not polymorphic
- All attributes are singularly typed
• Models are rigidly typed
- Legacy data is already mappable
• true and false …and nil
- Allow proto2 syntax models
• Need to decomp from existing
PHP Doctrine models
- Move source of truth without breaking
legacy code.
• Covariant polymorphic
repeated discriminator maps
- Enough said.

51. “How do you change the
wheels of a moving train?”

52. MOVE THE SOURCE OF TRUTH
Entity
UserModel
Schema
Computed
Properties
Behavior
Entity
UserModel
Computed
Properties
Behavior
UserFields
Schema
Codegen
from
the IDL

53. MOVE THE SOURCE OF TRUTH
Users Service
Envoy
Legacy PHP
Envoy
m.ToProto()
pb.ToModel()
OdieDoctrine
Adapter
Odie
Repo
Legacy
ModelPB / gRPC
Fields
Class

54. Reduce trauma with
transitional tooling

55. PYTHON ENVOY-GRPC CLIENTS
// Envoy-gRPC Client
class UsersClient(…):
def get(self, request)
def multiget(self, request)
def update(self, request)
// Official gRPC client
class UsersServiceClient(…):
def get(self, request)
def multiget(self, request)
def update(self, request)

56. Listen to your customers

57. “Protobufs are really
hard to use”

58. OFFICIAL PYTHON MESSAGES
User = _reflection.GeneratedProtocolMessageType('User',
(_message.Message,), dict(
DESCRIPTOR = _USER,
__module__ = 'pb.lyft.user_pb2'
))
_sym_db.RegisterMessage(User)

59. PYTHON MESSAGE PROXIES
class UserProxy(object):
def __init__(self, base_pb):
self._base = base_pb
@property
def id(self):
return self._base.id
@id.setter
def id(self, value):
self._base.id = value

60. FURTHER CODE GENERATION
• Go/PHP Envoy-aware clients
• PB over HTTP clients/server
• Observability interceptors
• Ergonomics helpers
• Response caching
• CLI

61. That’s an awful lot of
codegen…

62. PROTOC-GEN-STAR (PG*)
Code generation framework
• AST of Lyft primitives
• Simplifies code generation
• Highly testable
Package
File
Service
Method Enum
EnumValue
Message
OneOf
Field

63. PG*: WALK THE AST
type Visitor interface {
VisitPackage(Package) (v Visitor, err error)
VisitFile(File) (v Visitor, err error)
VisitMessage(Message) (v Visitor, err error)
VisitEnum(Enum) (v Visitor, err error)
VisitEnumValue(EnumValue) (v Visitor, err error)
VisitField(Field) (v Visitor, err error)
VisitOneOf(OneOf) (v Visitor, err error)
VisitService(Service) (v Visitor, err error)
VisitMethod(Method) (v Visitor, err error)
}

64. How far can we take this?

65. SERVICE GENERATION
Srv Framework
HTTP Server gRPC Server
Debug API
App-Level Config
Users Odie Gizmo Odie
HTTP Service
Config
Users Service ConfigModel Repositories
HTTP Handlers Users Handlers

66. FUTURE TOOLS
Linting & Static Analysis
• Enforce best practices
• Protect production code
• Core Libraries ≠ IDL Police
Mocks & Test Fixtures
• Scenarios of valid state
• Reduce reliance on integration tests
• Developer confidence
gRPC on Mobile
• Reduced payload size
• Leverage streaming APIs
• Global consistency

67. The incremental march
continues…

68. Having an ideal is good

69. Awareness of reality is
essential

70. THANKS!
Christopher Burnett - Lead Core Libraries
@twoism
Chris Roche - Core Libraries
@rodaine

71. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/api-pb-
grpc