Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Confidential
Architecture of Highly Loaded Geo-
Distributed Applications
Andrey Antilikatorov
Solutions Architect
19.08....
2
Confidential
3
Confidential
4
Confidential
5
Confidential
6
Confidential
New Content On-Boarding
SAS / Signed URL
7
Confidential
File Chunker
Content Filtering and
Tagging (ML-based)
Transcoding & Quality
Conversion
Distribution to Cont...
8
Confidential
New Content :: Transcoding & Quality Conversion
9
Confidential
File Chunker
Content Filtering and
Tagging (ML-based)
Transcoding & Quality
Conversion
Distribution to Cont...
10
Confidential
CDN
11
Confidential
CDN :: Content Distribution Optimization
12
Confidential
CDN :: Content Distribution Optimization
13
Confidential
CDN
14
Confidential
Metadata Storage
15
Confidential
Consistency
Partition
Tolerance
Availability
Metadata Storage
16
Confidential
Metadata Storage
CP AP
CP/AP
17
Confidential
Over 50 Clusters
Over 500 Nodes (The largest cluster has 72 nodes)
Over 500K writes/second
Over 30TB per d...
18
Confidential
Storage Optimization
19
Confidential
Big Data Storage Approach
20
Confidential
Data Lake vs Data Warehouse
Structured/ Semi-Structured/
Unstructured/Binary
On-Read / Post Processing
Cos...
21
Confidential
ETL vs ELT
* Picture from https://big2smart.com/data-lake-vs-data-warehouse-whats-the-difference-and-which...
22
Confidential
Data Lake vs Data Warehouse
Education & Science
IoT & Automotive
Healthcare
1
2
3
Finance and Banking
Publ...
23
Confidential
Monolithic data storage. Data team needs to know
about all services.
High dependency on data engineers and...
24
Confidential
Data Mesh Approach
25
Confidential
Service 1
Service 2 Service 3
Service 5
Service 6
Service 4
Caching
26
Confidential
Service 6
Caching at Netflix
https://github.com/Netflix/EVCache
27
Confidential
Caching at Netflix
28
Confidential
Caching at Netflix
29
Confidential
1 2
Endpoint
Service Service
Service
A
Handling Failures & Services Degradation
Service Service Service
30
Confidential
Gracefully handle timed-out calls
Prevent execution of requests which cannot be handled
Disconnect failed ...
31
Confidential
Cascading Failures Handling
https://github.com/Netflix/Hystrix
https://github.com/alibaba/Sentinel
32
Confidential
Critical Functionality and Analytics
33
Confidential
Use me! I’m free!!!
34
Confidential
Geo-Optimization :: Taxi Map
35
Confidential
Optimization based on Geolocation
36
Confidential
Uber RingPop
37
Confidential
Automatic redistribution of region IDs.
Seamless adding/removal of nes nodes
Automatic redirects to a serv...
38
Confidential
39
Confidential
39
Thank You!
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Сloud Webinar #1 “Architecture of Highly Loaded Geo-Distributed Applications”

Download to read offline

This online Сloud Webinar “Architecture of Highly Loaded Geo-Distributed Applications” was delivered by Andii Antilikatorov (Senior Solution Architect, Technology) on August 19, 2021

During this event, the speaker shared his proven experience in creating solutions for systems that work in different geographic regions all over the world.

More details and presentation you can find here: https://bit.ly/3A2JuoK

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Сloud Webinar #1 “Architecture of Highly Loaded Geo-Distributed Applications”

  1. 1. 1 Confidential Architecture of Highly Loaded Geo- Distributed Applications Andrey Antilikatorov Solutions Architect 19.08.2021
  2. 2. 2 Confidential
  3. 3. 3 Confidential
  4. 4. 4 Confidential
  5. 5. 5 Confidential
  6. 6. 6 Confidential New Content On-Boarding SAS / Signed URL
  7. 7. 7 Confidential File Chunker Content Filtering and Tagging (ML-based) Transcoding & Quality Conversion Distribution to Content Delivery Network New Content On-Boarding Process
  8. 8. 8 Confidential New Content :: Transcoding & Quality Conversion
  9. 9. 9 Confidential File Chunker Content Filtering and Tagging (ML-based) Transcoding & Quality Conversion Distribution to Content Delivery Network New Content On-Boarding
  10. 10. 10 Confidential CDN
  11. 11. 11 Confidential CDN :: Content Distribution Optimization
  12. 12. 12 Confidential CDN :: Content Distribution Optimization
  13. 13. 13 Confidential CDN
  14. 14. 14 Confidential Metadata Storage
  15. 15. 15 Confidential Consistency Partition Tolerance Availability Metadata Storage
  16. 16. 16 Confidential Metadata Storage CP AP CP/AP
  17. 17. 17 Confidential Over 50 Clusters Over 500 Nodes (The largest cluster has 72 nodes) Over 500K writes/second Over 30TB per day Totally Denormalized Data Model Cassandra Under High Load
  18. 18. 18 Confidential Storage Optimization
  19. 19. 19 Confidential Big Data Storage Approach
  20. 20. 20 Confidential Data Lake vs Data Warehouse Structured/ Semi-Structured/ Unstructured/Binary On-Read / Post Processing Cost-effective Big Data storage Analysts, engineers, data scientists Big Data analytics, deep learning, real-time analytics Type of data Processing Purpose Primary Users Tasks Structured and normalized On-Write / Pre-Processing Analytics for business decisions Business analysts Read-only queries for aggregating and summarizing data insights
  21. 21. 21 Confidential ETL vs ELT * Picture from https://big2smart.com/data-lake-vs-data-warehouse-whats-the-difference-and-which-is-the-best-data-architecture/
  22. 22. 22 Confidential Data Lake vs Data Warehouse Education & Science IoT & Automotive Healthcare 1 2 3 Finance and Banking Public sector Travel
  23. 23. 23 Confidential Monolithic data storage. Data team needs to know about all services. High dependency on data engineers and analysts. Data team needs to understand entire business model Single point of failure. High DevOps & Engineering effort. Problem One Problem Two Problem Three Problem Four 1 2 3 4 Common Problems of ‘Traditional Approach’
  24. 24. 24 Confidential Data Mesh Approach
  25. 25. 25 Confidential Service 1 Service 2 Service 3 Service 5 Service 6 Service 4 Caching
  26. 26. 26 Confidential Service 6 Caching at Netflix https://github.com/Netflix/EVCache
  27. 27. 27 Confidential Caching at Netflix
  28. 28. 28 Confidential Caching at Netflix
  29. 29. 29 Confidential 1 2 Endpoint Service Service Service A Handling Failures & Services Degradation Service Service Service
  30. 30. 30 Confidential Gracefully handle timed-out calls Prevent execution of requests which cannot be handled Disconnect failed services Configures default responses or fallback scenarios Gathers custom metrics Cascading Failures Handling
  31. 31. 31 Confidential Cascading Failures Handling https://github.com/Netflix/Hystrix https://github.com/alibaba/Sentinel
  32. 32. 32 Confidential Critical Functionality and Analytics
  33. 33. 33 Confidential Use me! I’m free!!!
  34. 34. 34 Confidential Geo-Optimization :: Taxi Map
  35. 35. 35 Confidential Optimization based on Geolocation
  36. 36. 36 Confidential Uber RingPop
  37. 37. 37 Confidential Automatic redistribution of region IDs. Seamless adding/removal of nes nodes Automatic redirects to a server which owns particular region High reliability and performance due to possibility to apply various optimizations Services can communicate to each other using RPC calls. RingPop Requests Processing
  38. 38. 38 Confidential
  39. 39. 39 Confidential 39 Thank You!

This online Сloud Webinar “Architecture of Highly Loaded Geo-Distributed Applications” was delivered by Andii Antilikatorov (Senior Solution Architect, Technology) on August 19, 2021 During this event, the speaker shared his proven experience in creating solutions for systems that work in different geographic regions all over the world. More details and presentation you can find here: https://bit.ly/3A2JuoK

Views

Total views

123

On Slideshare

0

From embeds

0

Number of embeds

91

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×