Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ Powering Interactive Analytics with Alluxio and Presto Dmytro Dermanskyi, Data Engineering Lead, WalkMe About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack

1. 1
Powering interactive analytics
with Alluxio and Presto
Dmytro Dermanskyi
DATA ORCHESTRATION SUMMIT 2020

2. 22
™
© All Rights Reserved
™
WalkMe: Digital Adoption Platform

3. 33
™
© All Rights Reserved
™
Insights: WalkMe’s web analytics solution

4. 44
™
© All Rights Reserved
™
Funnels analysis

5. 55
™
© All Rights Reserved
User must perform these steps to submit a support ticket
1 2 3
Click “Contact
support” link
Visit “Submit a ticket”
page
Fill in the details and
open the ticket1 2 3
Funnel example: “Submit support ticket” journey

6. 66
™
© All Rights Reserved
™
Funnel example: UI

7. 77
™
© All Rights Reserved
™
Funnel example: SQL behind the scenes

8. 88
™
© All Rights Reserved
™
Initial architecture

9. 99
™
© All Rights Reserved
™
Challenges on the way to Alluxio
1. Glue tables data location paths are fixed to s3:// “filesystem”.
2. Alluxio metadata needs to be periodically refreshed to sync changes in
S3 since some S3 files are written not through Alluxio.
1. In a collocated deployment we need to carefully distribute memory
allocations between Alluxio and Presto processes. By default EMR gives
Presto JVMs most of available RAM.

10. 1010
™
© All Rights Reserved
™
Problem 1: Pointing Glue tables to Alluxio
Solution:
1. Export Glue catalog into a classic Hive metastore.
2. During the export fix table location paths to use alluxio:// instead of
s3://.
3. Connect EMR Presto to the Hive metastore instead of Glue catalog.
4. Repeat the steps 1 and 2 periodically to sync changes in Glue(e.g. new
partitions, new tables) to the Hive metastore.

11. 1111
™
© All Rights Reserved
™
Problem 1: Pointing Glue tables to Alluxio
Disadvantages of the solution:
● Glue-to-Hive export job is really complex.
● Necessary to keep two table catalogs: Glue and Hive metastore.
Resolving and syncing schema changes is difficult.
● The export job depends on proprietary Glue API which can change with
time and break the job.

12. 1212
™
© All Rights Reserved
™
Problem 1: Pointing Glue tables to Alluxio
Possible alternative solution: Alluxio Catalog Service
Just delegate the problem to Alluxio
https://docs.alluxio.io/os/user/stable/en/core-services/Catalog.html

13. 1313
™
© All Rights Reserved
™
Problem 2: Alluxio metadata sync for S3 changes
Solution:
alluxio fs ls [-R|-f]
Can be executed:
● Upon file change events from S3
● Periodically with crontab

14. 1414
™
© All Rights Reserved
™
Problem 3: Presto vs Alluxio memory contention
By default EMR gives Presto JVM processes most available RAM:
EC2 node type RAM EMR Presto -Xmx
c5.2xlarge 16GB 12GB
r5.4xlarge 128GB 100GB
c5.4xlarge 32GB 24.5GB
c4.8xlarge 60Gb 47GB
c5.9xlarge 72GB 55GB
Remaining memory may not be enough for Alluxio processes.
Especially if you want Alluxio to cache data in RAM.

15. 1515
™
© All Rights Reserved
™
Problem 3: Presto vs Alluxio memory contention
EMR doesn’t provide means to adjust default Presto JVM config. But we need such
control to appropriately distribute memory allocation for processes.

16. 1616
™
© All Rights Reserved
™
Problem 3: Presto vs Alluxio memory contention
Solution 1:
Don’t use EMR Presto but deploy our own Presto distribution:
● This brings additional complexity and more work.
● Though also gives more flexibility for Presto version choice and
configuration.

17. 1717
™
© All Rights Reserved
™
Problem 3: Presto vs Alluxio memory contention
Solution 2:
Use EMR bootstrap script to setup a crontab script that fixes EMR Presto JVM
config as soon as it provisioned by EMR:
NOTE: EMR runs bootstrap scripts before installing Presto
● Allows reusing EMR Presto distribution (no need to deploy own Presto).
● “Hacky” solution.

18. 1818
™
© All Rights Reserved
™
Alluxio-based architecture

19. 1919
™
© All Rights Reserved
™
Performance improvement
Average query execution time has dropped from ~20 sec to ~1 sec.

20. 2020
™
© All Rights Reserved
™
Lessons learned: Memory cannot be overcommitted
● Make sure to not use default EMR Presto JVM config since there might be
not enough memory left for Alluxio processes. Otherwise beware OOM
crashes.
● Chose appropriate caching medium for Alluxio:
○ To cache in RAM use memory optimized instances.
○ Consider caching on SSD. Both NVMe and EBS gp2 disks can give
significant performance boost.

21. 2121
™
© All Rights Reserved
™
Lessons learned: Data locality matters
We tried deploying Alluxio as a separate from Presto cluster:
+ This design allows decoupling of storage (S3), caching layer (Alluxio) and
compute (Presto). You can give Alluxio as much resources (RAM/SSD) as
necessary.
- Query performance is at least 2-3 times worse comparing to the collocated
deployment.

22. 2222
™
© All Rights Reserved
™
Lessons learned: Alluxio metastore can grow large
When the number of files in UFS is becoming huge (e.g. hundreds of
thousand) consider the following:
● Alluxio journal will have gigabytes of size. Make sure the disk where it’s
kept is large enough. Or use UFS for it (though it’s not so simple).
● It may be worth switching from HEAP to RocksDB-based metastore. This
way gigabytes of master RAM taken by metastore can be freed up.

23. 2323
™
© All Rights Reserved
™
Lessons learned: Monitor everything!
What can go wrong:
● Presto workers can crash
● Alluxio workers can crash
● Presto/Alluxio masters can crash
● EC2 instance can run out of disk space
● Queries can start to fail because of lots of different reasons
● Queries can start to just hang and timeout
● You name it ...

24. 24
Thank you!
Any questions?