Story about scaling custom resources on Kubernetes and Crossplane.

1. © 2023 Akamai
1
Power and protect
life online

2. © 2023 Akamai
2
Crossplane and a story
about scaling Kubernetes
custom resources
Richa’rd [mhmxs] Kova’cs

3. © 2023 Akamai
3
Boring Slide
• At work
○ Staff Kubernetes Engineer
○ @Akamai > Linode
○ Go microservices, Kubernetes controllers
• At IT space
○ Many years of DevOps and automation
○ Several years of Go, Java … who counts
○ OSS devotee
○ Known as @mhmxs

4. © 2023 Akamai
4
Long time ago on a cluster far, far away….
Agenda
• Shallow dive into Kubernetes
• What are custom resources good for?
• What the heck is a Kubernetes operator?
• Crossplane in a nutshell
• Scaling characteristics
• Scaling in action

5. © 2023 Akamai
5
Shallow dive into
Kubernetes

6. © 2023 Akamai
6
What is Kubernetes?
Kubernetes, is an open-source system for automating deployment, scaling, and
management of containerized* applications.
It’s like an operating system for usually more than one computers.
● Regular “processes” are groups of containers.
*Containers are lightweight, portable, and isolated execution environments that encapsulate applications and their dependencies.

7. © 2023 Akamai
7
K8OS
Resource Management
Both allocate and manage resources like CPU, memory, and storage to ensure optimal performance.
Process Isolation
Both systems provide a level of process isolation to prevent conflicts and ensure applications run
independently.
Scheduling
Both systems implement scheduling algorithms to efficiently manage the execution of
processes or containers.
Security and AuthZ
Both focus on security measures, controlling access to resources, and maintaining the integrity of
the system.
Lifecycle Management
Both Kubernetes and traditional operating systems oversee the complete lifecycle of applications.
Similarities

8. © 2023 Akamai
8
Declarative
Kubernetes follows a declarative approach, where users specify the desired state of their applications.
The platform then automatically works to ensure the actual state aligns with this defined configuration.
Controllers are control loop processes that continuously work to maintain the desired state of resources by
reconciling the actual state with the specified configuration.

9. © 2023 Akamai
9
Architecture
src https://platform9.com/blog/kubernetes-enterprise-chapter-2-kubernetes-architecture-concepts/

10. © 2023 Akamai
10
Architecture
src https://platform9.com/blog/kubernetes-enterprise-chapter-2-kubernetes-architecture-concepts/

11. © 2023 Akamai
11
Architecture
src https://platform9.com/blog/kubernetes-enterprise-chapter-2-kubernetes-architecture-concepts/

12. © 2023 Akamai
12
Average backend service
Managed Resources
Pod

13. © 2023 Akamai
13
Average backend service
Managed Resources
Deployment
Pod

14. © 2023 Akamai
14
Average backend service
Managed Resources
Service
Deployment
Pod

15. © 2023 Akamai
15
Average backend service
Managed Resources
Gateway
Service
Deployment
Pod

16. © 2023 Akamai
16
What are custom
resources good for?

17. © 2023 Akamai
17
Custom Resources
● Custom Resources (CRs) are extensions to the Kubernetes API.
● Explaining the structure of a Custom Resource Definition (CRD).
○ Basic validation
○ Validation webhook
● The Kubernetes API Server primarily functions as a data store.
○ Allowing users to manage their own custom objects (CRUD).
○ Only takes care on desired state.

18. © 2023 Akamai
18
Custom Resource Definitions

19. © 2023 Akamai
19
Custom Resource Definitions

20. © 2023 Akamai
20
Custom Resource Definitions

21. © 2023 Akamai
21
Custom Resource Definitions

22. © 2023 Akamai
22
Custom Resource Definitions

23. © 2023 Akamai
23
What the heck is a
Kubernetes operator?

24. © 2023 Akamai
24
Kubernetes Operators
● Group of custom controllers to ensure desired state of Core nor Custom
Resources.
● A Kubernetes Operator is a method of packaging, deploying, and
managing applications on top of Kubernetes.
● It extends the Kubernetes API to create, configure, and manage
instances of complex applications.
● Actively watches resources via the Kubernetes API server for real-time
monitoring of changes and deviations in the system's state.

25. © 2023 Akamai
25
Crossplane in a nutshell

26. © 2023 Akamai
26
Crossplane
in a nutshell: Crossplane is an operator framework on steroids.
Some random points:
● Operator for external resources.
● Uses Kubernetes as backend.
● Provides CRs as API for everything.
● Able to bundle complex applications into one package.
● Combine multiple CRDs into one.
● Advanced RBAC support by design.
● Examples: Kubernetes, Helm, Linode (beta), Terraform, Argo CD, CAPI, …
○ Marketplace: https://marketplace.upbound.io/

27. © 2023 Akamai
27
Architecture

28. © 2023 Akamai
28
Architecture
Crossplane in a nutshell
https://github.com/linode/provider-ceph

29. © 2023 Akamai
29
Scaling characteristics

30. © 2023 Akamai
30
Crossplane
It is a plain old controller

31. © 2023 Akamai
31
Kubernetes
No more than 150,000 total pods
No more than 300,000 total containers
No more than 5,000 nodes
No more than 110 pods per node

32. © 2023 Akamai
32
Dimensions
Number of CRDs
Each Custom Resource Definitions (CRD) contributes to
the overall complexity of the API server, potentially
impacting its performance and responsiveness.
Expanding the variety of CRDs often entails the creation
of more controllers, adding to the management
overhead within the Kubernetes cluster.
API discovery becomes pivotal to manage the growing
complexity and ensure streamlined access to extended
functionalities.
Number of CRs
As the quantity of custom resources grows,
considerations must be made regarding resource
utilization, such as storage, network overhead,
memory usage, and processing power.
The larger the number of controller instances, the more
significant the impact on the cluster's operational
aspects, including scheduling, monitoring, and
event handling.
Throughput

33. © 2023 Akamai
33
Horizontal
Number of CRDs
● Problems tend to start happening once you’re over - very approximately - 500 CRDs.
● Some Crossplane providers should have 800+ of CRDs.
○ Crossplane have a new feature for partial deployment of CRDs.
● Take care on client side rate limits, most importantly burst, to avoid overloading Kubernetes API server.
● Some clients [kubectl] are designed that there won’t be more than 50-100 CRDs installed on a cluster.
● Use Kubernetes v1.26.0+ [https://blog.upbound.io/scaling-kubernetes-to-thousands-of-crds]
● Count with longer startups and heavy CPU and memory peaks of Kube API server.

34. © 2023 Akamai
34
Vertical
Number of CRs
ETCD data store
● Storage speed is critical.
● How to go further:
○ Kine, a storage plugin designed for providing an efficient and flexible storage backend for container orchestration.
■ Only a few databases are supported jet …
■ Implement your own database integration.
■ Implement routing logic to distribute load.
● ETCD cluster per namespace
● ETCD cluster per resource

35. © 2023 Akamai
35
Vertical
Number of CRs
Operator side
● Memory would be the main bottleneck.
● Configure rate limits, bursts and timeouts of clients.
● Retry failed actions to reduce CPU load.
● Find and schedule best cache sync periods.
● Filter watch resources by label. - PAUSE
● Count with longer startups.
● Forget leader elected operators.
● Ensure best reconciliation concurrency.

36. © 2023 Akamai
36
Vertical
Number of CRs
Kubernetes API server
● Network bandwidth
○ # of active connections. - watchers everywhere
● CPU and Memory - infinite if possible.
● Only a few configuration options available.
○ watch-cache [on/off]
○ event-ttl [1h]
○ endpoint-reconciler-type [lease]
○ enable-aggregator-routing [EP IP]
○ http2-max-streams-per-connection
○ max-[mutating-]requests-inflight

37. © 2023 Akamai
37
Vertical
Number of CRs
Kubernetes API server
● How to go further:
○ Bring your own cluster.
○ Scale vertically, there is only one leader.
○ Running a proxy in front of the API server.
○ Delegating load via Aggregation API.

38. © 2023 Akamai
38
Scaling in action

39. © 2023 Akamai
39
Overall memory footprint

40. © 2023 Akamai
40
Overall memory footprint

41. © 2023 Akamai
41
Overall memory footprint

42. © 2023 Akamai
42
Overall memory footprint

43. © 2023 Akamai
43
This is sample text. This is sample text. This is sample text. This is sample text.
The main reason for watch cache
is to speed up list operations …

44. © 2023 Akamai
44
Summary
● Kubernetes API server leader is SPOF.
● Kubernetes API server doesn’t scale well.
● Object reconciliation pause logic is mandatory.
● Millions of custom resources needs ~infinite resources.
● Some clients may not tolerate huge datasets or large timeouts.
● Pretty easy to kill Kubernetes API server via unfiltered list operations.
● Middle tier performance
○ requires lots of care and understanding.
○ needs complex architecture.
○ is impossible without custom components and business logic.

45. © 2023 Akamai
45