Building a Replicated Logging System with Apache Kafka
•Transferir como PPTX, PDF•
10 gostaram•3,669 visualizações
Apache Kafka is a scalable publish-subscribe messaging system with its core architecture as a distributed commit log. It was originally built as its centralized event pipelining platform for online data integration tasks. Over the past years developing and operating Kafka, we extend its log-structured architecture as a replicated logging backbone for much wider application scopes in the distributed environment. I am going to talk about our design and engineering experience to replicate Kafka logs for various distributed data-driven systems, including source-of-truth data storage and stream processing.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Building a Replicated Logging System with Apache Kafka
Semelhante a Building a Replicated Logging System with Apache Kafka (20)
Mais de Guozhang Wang
Mais de Guozhang Wang (8)
Último
Último (20)
Building a Replicated Logging System with Apache Kafka
- 1. Building a Replicated Logging System with Apache Kafka Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam Mammad Zadeh, Neha Narkhede, Jun Rao, Jay Kreps, Joe Stein
- 2. We All Love Logs!
- 6. Apache Kafka • A distributed messaging system ..that store messages as a log!
- 7. Example: LinkedIn back in 2010 Point-to-Point Pipelines What We Want: A Centralized Data Pipeline
- 8. Log-centric Data Flow • Logical Ordering • Persistent Buffering • “Source-of-Truth”
- 9. Store Messages as a Log 4 5 5 7 8 9 10 11 12... Producer Write Consumer1 Reads (offset 7) Consumer2 Reads (offset 10) Messages 3
- 10. Partition the Log across Machines Topic 1 Topic 2 Partitions Producers Producers Consumers Consumers Brokers
- 11. Apache Kafka Example: Kafka at LinkedIn
- 12. “Source-of-Truth” should not be lost even when..
- 14. Consensus for Log Replication Logs Broker-1 Logs Logs Broker-2 Broker-3 Write Consensus Protocol Consensus Protocol
- 15. Key Idea Separate membership configuration from data replication
- 16. Primary-backup Replication Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write
- 17. Conventional Quorum Commits Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write
- 18. Conventional Quorum Commits Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write
- 19. Conventional Quorum Commits Logs Broker-1 * Logs Logs Broker-2 Broker-3
- 20. Conventional Quorum Commits Logs Broker-1 * Logs Logs Broker-2 Broker-3
- 21. • Leader maintains in-sync-replicas (ISR) • Failed / slow follower => drop from ISR • Caught-up follower => re-join ISR • Producer specifies required ACK based on ISR Configurable ISR Commits
- 22. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 23. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 24. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 25. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 26. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 27. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 28. Example: ACK with all ISRs Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 29. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 30. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 31. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 32. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 33. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 34. Example: ACK with Leader-only Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“leader”) ISR {1, 2, 3}
- 35. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 36. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 37. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 38. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 39. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2, 3}
- 40. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2}
- 41. Example: Slow Follower Logs Broker-1 * Logs Logs Broker-2 Broker-3 Write (ack=“all”) ISR {1, 2}
- 42. Configurable ISR Commits ACK mode Latency On Failures “no" no network delay some data loss “leader" 1 network roundtrip a few data loss “all" ~2 network roundtrips no data loss
- 43. • Use an embedded controller • Detect broker failure via ZooKeeper • Leader failure => elect new leader from ISR • Leader and ISR persisted in Zookeeper • For Controller fail-over Membership Management
- 44. Example: Broker Failure Logs Broker-1 * Logs Logs Broker-2 Broker-3 ISR {1, 2}
- 45. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3
- 46. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3
- 47. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3
- 48. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3 ISR {2}
- 49. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3 ISR {2}
- 50. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 Broker-3 ISR {2}
- 51. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 * Broker-3 ISR {2}
- 52. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 * Broker-3 ISR {2}
- 53. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 * Broker-3 ISR {2}
- 54. Example: Broker Failure Logs Broker-1 Logs Logs Broker-2 * Broker-3 ISR {2, 3}
- 55. • Overview: Logs and Kafka • Log Replication in Kafka • Kafka Usage at LinkedIn • Conclusion Agenda
- 57. Apache Kafka Example: Kafka at LinkedIn
- 58. Example: Espresso • A distributed document store • Primary online data serving platform at LI • Member profile, homepage, InMail, etc [SIGMOD 2013]
- 59. Old Espresso Replication Data Center-1 Storage Node Storage NodeMySQL Replication MySQL MySQL Search Index Hadoop … …Databus Cross-DC Replicator Data Center-1 Storage Node Storage NodeMySQL Replication MySQL MySQL Search Index Hadoop … DatabusCross-DC Replicator
- 60. Problems with MySQL Replication Master Storage Node P1 Slave Storage Node P2 P3 P4 P5 P6 P1 P2 P3 P4 P5 P6 Binary Log Shipping
- 61. Replicate Logs with Kafka Storage Node Kafka Logs P1 Storage Node P2 P3 P4 P5 P6 P1 P2 P3 P4 P5 P6 Kafka Producer Kafka Consumer Kafka Consumer Kafka Producer
- 62. Key-based Log Compaction ... Partition Messages Segment-3 Segment-4 Segment-6 *
- 63. Key-based Log Compaction d: 3 f: 8 b: 0 c: null... Partition Messages c: 3 a: 5 a: 6 a: 5 f: 9 ... Segment-3 Segment-4 b: 2 d: 4a: 1
- 64. Key-based Log Compaction ... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 f: 8 b: 0 a: 5 f: 9 New Segment Partition Messages
- 65. Key-based Log Compaction ... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 6 d: 3 f: 8 b: 0 c: null a: 5 f: 9 New Segment Partition Messages
- 66. Key-based Log Compaction ... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 b: 0 a: 5 f: 9 New Segment Partition Messages
- 67. Key-based Log Compaction ... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ... Segment-3 Segment-4 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 b: 0 a: 5 f: 9 New Segment Partition Messages
- 68. New Espresso Replication Data Center-1 Storage Node Storage Node Storage Node Kafka Logs MySQL MySQL MySQL Data Center-n Storage Node Storage Node Storage Node Kafka Logs MySQL MySQL MySQL Kafka MirrorMaker Search Index Hadoop … … Search Index Hadoop … * In Progress
- 70. Apache Kafka Example: Kafka at LinkedIn
- 71. • Data flow streaming on Kafka and YARN • Stateful processing • Re-processing • Failure Recovery Example: Samza [CIDR 2015]
- 73. Kafka Kafka Samza StateProces s Protoc ol StateProces s Protoc ol StateProces s Protoc ol Samza Processing Kafka Changelog
- 74. Kafka Kafka Samza StateProces s Protoc ol StateProces s Protoc ol StateProces s Protoc ol Samza Processing Kafka Changlog
- 75. Kafka Kafka Samza StateProces s Protoc ol StateProces s Protoc ol StateProces s Protoc ol Samza Processing Kafka Changlog StateProces s Protoc ol
- 76. Take-aways • Log-centric data flow helps scaling your systems • Kafka: replicated log streams for real-time platforms
- 77. We are Hiring
- 78. Take-aways • Log-centric data flow helps scaling your systems • Kafka: replicated log streams for real-time platforms THANKS!
Notas do Editor
- Thank you. And good morning, today I am going to talk about Kafka, and how it can be built as a general replicated log streams for a wide use of scalable systems. This is a joint work from the Apache Kafka community.
- First of all, being in this room, I think it is safe to say “we all love logs”. Logs have been around almost as long as this research community.
- No-overwrite in POSTGRES ARIES: Write-Ahead-Logging in the 80’s Today, reading the 50 page Aries pager has been the must-to-do for every single database graduate student including myself.
- Similarly, Log-Structured storage architecture.
- Replicated State Machine And in all these examples, the log is used as the source of truth data change log to scale the systems while providing durability and consistency.
- So that is all good stuff about logs, but where is Kafka is this big picture. Well, Kafka is an Apache open sourced distributed messaging system that stores messages as a commit log.
- Data-serving websites, LinkedIn has a lot of data We have this variety of data and and we need to build all these products around such data. Messaging: ActiveMQ User Activity: In house log aggregation Logging: Splunk Metrics: JMX => Zenoss Database data: Databus, custom ETL
- This idea of using logs for data flow has been floating around LinkedIn, log-centric fashion. Take all the organization's data and put it into a central log for real-time subscription. Data integration, replication, real-time stream processing.
- Disks are fast when used sequentially File system caching
- Topic = message stream Topic has partitions, partitions are distributed to brokers
- higher availability and durability
- evenly distributed
- replicated log => replicated state machine
- One of the replicas is leader, leader evenly spread All writes go to leader Leader propagates writes to followers in order Leader decides when to commit message
- The size of the ISR is decoupled from the size of the replica set, hence the number of replicas and acknowledgements are independent.
- ack=3
- committed messages to consumer messages are committed is independent of the ack chosen by the producer.
- ack=1
- ack=1
- ack=3, follower slow
- under replicated partitions
- ack=3, broker failure
- load balancing cluster expansion
- load balancing cluster expansion
- This is a major initiative and will put Kafka on the critical path for site latency sensitive data paths which also require much higher message delivery guarantees.
- Data standardization, site monitoring
- Data flow graph. Flow rate may overwhelm query processor: batch processing, sampling, synopsis, etc In-memory storage constraints: single-pass algorithms, no stream backtracking
- WAL
- Streaming on Message Pipes