Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ Reducing large S3 API costs using Alluxio at Datasapiens Juraj Pohanka & Koen Michiels, Datasapiens About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack

1. DATA ORCHESTRATION SUMMI
T
Reducing AWS S3 API costs using Alluxio
Koen Michiels | CEO @ datasapiens, Juraj Pohanka | CTO
@ datasapiens

2. DATA ORCHESTRATION
SUMMIT
2020
datasapiens introduction

3. DATA ORCHESTRATION SUMMIT
About datasapiens

4. DATA ORCHESTRATION SUMMIT
Our tools help to turn data into profit

5. DATA ORCHESTRATION SUMMIT
Some users of our products

6. DATA ORCHESTRATION
SUMMIT
2020
Our infrastructure journey

7. DATA ORCHESTRATION SUMMIT
▪simple to setup and use
▪storage-compute coupled architecture
▪long query compilation times
▪vendor-locking
Data platform: 1st generation

8. DATA ORCHESTRATION SUMMIT
▪storage-compute separation benefits
▪faster performance due to reduced compilation times
▪cost decrease below expectations due to increase of S3 API costs
Data platform: 2nd generation version 1

9. DATA ORCHESTRATION SUMMIT
▪co-located Alluxio-Presto cluster
▪faster and more consistent performance
▪S3 API costs decreased to negligible level
Data platform: 2nd generation version 2

10. DATA ORCHESTRATION
SUMMIT
2020
Our observations

11. DATA ORCHESTRATION SUMMIT
▪Docker Swarm cluster as a Docker stack
deployment
▪various levels of usage concurrency
▪cluster setup
• master node (1x c5.4xlarge)
• worker nodes (10x c5.4xlarge)
• Docker stack services:
• PrestoSQL v317
• Apache Hive v2.3.5
Performance tests on a Presto cluster

12. DATA ORCHESTRATION SUMMIT
▪S3 API costs during the performance tests
Drastic increase of S3 API costs

13. DATA ORCHESTRATION SUMMIT
▪query costs
Performance tests – per day stats
Date Total query count Total S3 costs ($) Total S3 request
count
Avg. costs per
query ($)
Avg. S3 requests
per query
2019-08-28 15,086 172.73 197,549,860 0.01145 13,095
2019-08-29 29,997 124.87 138,210,969 0.00416 4,607
2019-08-30 6,394 26.27 22,638,580 0.00411 3,541
2019-08-31 36,815 80.47 66,406,798 0.00219 1,804
2019-09-01 13,939 62.54 56,195,976 0.00449 4,032
2019-09-03 752 4.72 3,331,228 0.00628 4,430
2019-09-06 6,806 28.59 18,005,207 0.00420 2,645
2019-09-09 75,687 231.74 175,443,269 0.00306 2,318
Total 185,476 731.93 677,781,887 0.00395 3,654

14. DATA ORCHESTRATION SUMMIT
▪infrastructure costs
Performance tests – per day stats
Date Total S3 costs ($) Total EC2 costs ($) S3 API costs - % of
total costs
2019-08-28 172.73 45.25 79.24%
2019-08-29 124.87 107.21 53.80%
2019-08-30 26.27 27.18 49.15%
2019-08-31 80.47 29.42 73.23%
2019-09-01 62.54 23.69 72.53%
2019-09-03 4.72 11.47 29.15%
2019-09-06 28.59 73.05 28.13%
2019-09-09 231.74 102.97 69.24%
Total 731.94 420.24 63.53%

15. DATA ORCHESTRATION SUMMIT
▪co-located Alluxio-Presto cluster
▪faster and more consistent performance
▪S3 API costs dropped to below $1/day
Alluxio as a remedy

16. DATA ORCHESTRATION
SUMMIT
2020
A reproducible example using TPC-DS

17. DATA ORCHESTRATION SUMMIT
▪Alluxio-Presto cluster
• hardware setup:
• master node (1x m5.2xlarge)
• worker nodes (6x c5.4xlarge)
• software deployment setup:
• Alluxio 2.2.0
• PrestoSQL 337
▪EMR-Presto cluster
• hardware setup:
• master node (1x m5.2xlarge)
• worker nodes (6x c5.4xlarge)
• software deployment setup:
• PrestoDB 0.227
Cluster setup

18. DATA ORCHESTRATION SUMMIT
▪dataset
• TPC-DS dataset with scale factor
100
• stored in a S3 bucket
▪query execution
• set of queries:
• TPC-DS suite excl. Query no. 72
• query execution:
• number of repeates: 10
• concurrency level: 1
Test setup
▪measurements
• Alluxio:
• logical operations: 'File Infos Got'
• RPC invocations: 'GetFileInfo'
• S3:
• total request counts per request type
• total request costs per request type

19. DATA ORCHESTRATION SUMMIT
▪10 most API requests-expensive queries
Results from the Alluxio-Presto cluster
Query name File Infos Got - avg GetFileInfo Operations - avg
q14_1 159,200.1 127,576.9
q09 137,031.0 109,669.0
q14_2 110,933.8 88,732.6
q75 101,468.4 81,166.3
q64 75,148.3 60,099.4
q88 73,224.0 58,584.0
q23_1 61,313.6 49,054.3
q23_2 60,566.2 48,457.6
q95 56,518.0 45,212.0
q28 54,810.0 43,866.0

20. DATA ORCHESTRATION SUMMIT
▪cumulative request counts
Results from the Alluxio-Presto cluster
Operation type Cumulative count
File Infos Got 24,089,740
GetFileInfo Operations 19,287,627
▪S3 API costs for caching the dataset into Alluxio
Request type Cumulative count Cumulative cost ($)
ListBucket 28,324 0.14
GetObject 24,033 0.01
HeadObject 44,581 0.02
Total 96,938 0.17

21. DATA ORCHESTRATION SUMMIT
▪Cumulative request counts
Results from the EMR-Presto cluster
Request type Cumulative count Cumulative cost ($)
ListBucket 5,771,219 28.86
GetObject 29,254,280 11.70
HeadObject 133,888 0.05
Total 35,159,387 40.61

22. DATA ORCHESTRATION SUMMIT
▪10 most API requests-expensive queries
Per-query cost estimations
Query name S3 API cost ($)
q14_1 0.2684
q09 0.2310
q14_2 0.1870
q75 0.1711
q64 0.1267
q88 0.1234
q23_1 0.1034
q23_2 0.1021
q95 0.0953
q28 0.0924

23. DATA ORCHESTRATION SUMMIT
▪Infrastructure costs and S3 API costs
Costs comparions
▪S3 API costs form 0.58% of total costs when using Alluxio
▪S3 API costs form 48.83% of total costs when not using Alluxio
Cluster Infrastructure costs ($) S3 API costs ($)
Alluxio+Presto cluster 29.02 0.17
EMR+Presto cluster 42.55 40.61

24. DATA ORCHESTRATION
SUMMIT
2020
General implications for cloud data lake
architectures

25. DATA ORCHESTRATION SUMMIT
▪pricing for storage across cloud providers
is similar
▪common analytical workloads are far
more storage/compute intense than in
our example
▪no intermediate data storage layer will
lead to higher costs
Use an intermediate storage layer

26. DATA ORCHESTRATION SUMMIT
▪GitHub repository with complete test results:
• https://github.com/datasapiens/alluxio-s3-costs-test
▪DZone article link:
• https://dzone.com/articles/reducing-large-s3-api-costs-using-alluxio
▪Company website
• https://www.datasapiens.co.uk
Links

27. DATA ORCHESTRATION
SUMMIT
2020
Q&A