Introduction to Apache Mesos Cluster Manager Framework By: Morteza Zakeri

1. Introduction to MESOS
By: Morteza Zakeri
Cluster and Grid Computing Course
Instructor: Dr. Mohsen Sharifi
School of Computer Engineering
Iran University of Science and Technology
Winter 2017

2. Outline
• Cluster Computing
• Applications
• Backgrounds and Terminology
• Problems
• Sharing Frameworks
• Mesos
• Goals to Design
• Key Elements
• Architecture
10 March 2017 Intro. to Mesos - M.Zakeri Page 2 of 48

3. Outline
• Resource Offer
• Implementation and API
• Evaluation (Benchmarks)
• Limitations
• Related Works
• Conclusion
• References
10 March 2017 Intro. to Mesos - M.Zakeri Page 3 of 48

4. Cluster computing main applications
• High Performance Computing (HPC)
Large internet services
• E.g. social networks, online shopping, …
Data-Intensive scientific applications
• E.g. physics, astronomy,
molecular sciences, …
10 March 2017 Intro. to Mesos - M.Zakeri Page 4 of 48
M4 globular cluster

5. Terminology
• Cluster
• A collection of distributed machines that collaborate to present a
single integrated computing resource to the user.
• Node
• Each computing element (single machine itself) within cluster.
• Framework
• A software system that manages and executes one or more jobs
on a cluster.
10 March 2017 Intro. to Mesos - M.Zakeri Page 5 of 48

6. Terminology
• Job
• A unit of work run on the cluster nodes.
• Slot
• Nodes are subdivided into “slots”.
• (In some framework such as Hadoop and Dryad)
• Task
• Jobs are composed of short tasks that are matched to slots.
• (In some framework such as Hadoop and Dryad)
10 March 2017 Intro. to Mesos - M.Zakeri Page 6 of 48

7. Terminology
• Elastic framework
• Can scale its resources up and down during execution tasks
(dynamically).
• Such as Hadoop and Dryad.
• Rigid framework
• Can start running its jobs only after it has acquired a fixed quantity
of resources.
• Can not scale (up/down) dynamically.
• Such as MPI.
10 March 2017 Intro. to Mesos - M.Zakeri Page 7 of 48

8. Background
• Diverse array of cluster computing frameworks:
• Hadoop/ Map-Reduce (multi-terabyte data-sets)
• Dryad (machine learning)
• Pregel (graph computation)
• MPI
• …
10 March 2017 Intro. to Mesos - M.Zakeri
Dryad
Page 8 of 48
Pregel

9. Problem
• Rapid innovation in cluster computing frameworks.
There is no single framework that optimal for all
applications.
Clusters typically run single framework at a time.
10 March 2017 Intro. to Mesos - M.Zakeri Page 9 of 48

10. Problem
• Hadoop data warehouse at Facebook:
• Include 2000-node Hadoop cluster,
• Uses a fair scheduler for Hadoop,
• Takes advantage of the fine-grained nature of the workload,
• Allocate resources at the level of tasks and to optimize data
locality.
↓
• This cluster can only run Hadoop jobs!
• Can it run an MPI program efficiently?
10 March 2017 Intro. to Mesos - M.Zakeri Page 10 of 48

11. Problem
10 March 2017 Intro. to Mesos - M.Zakeri
CDF of job and task durations in Facebook’s Hadoop data warehouse
Page 11 of 48
Duration (s)
NumberofJobsandTasks

12. Motivation to Sharing
• We want to run multiple frameworks in a single cluster.
• Sharing improves cluster utilization,
• Allows applications to share access to large datasets.
• may be too costly to replicate across distinct clusters!
10 March 2017 Intro. to Mesos - M.Zakeri Page 12 of 48

13. Common solutions for sharing a cluster
1. Statically partition the cluster and run one framework per
partition.
2. Allocate a set of VMs to each framework.
• Disadvantages:
• No high utilization,
• No efficient data sharing.
10 March 2017 Intro. to Mesos - M.Zakeri Page 13 of 48

14. Better solution: Mesos!
10 March 2017 Intro. to Mesos - M.Zakeri Page 14 of 48
Hadoop
Pregel
MPI
Shared cluster
Today: static partitioning Mesos: dynamic sharing

15. What is Mesos?
• A platform for sharing clusters between multiple diverse
cluster computing frameworks.
• E.g. Hadoop + Pregel + MPI + …
10 March 2017 Intro. to Mesos - M.Zakeri
Mesos
Node Node Node Node
Hadoop Pregel …
Node Node
Hadoop
Node Node
Pregel
…
Page 15 of 48

16. Goals to Design Mesos
• High utilization
• Efficient data sharing
• Support diverse frameworks (current and future)
• Scalability to 10,000’s of nodes
• Reliability in face of failures (robust to failures)
10 March 2017 Intro. to Mesos - M.Zakeri Page 16 of 48

17. Design Philosophy
• Small microkernel-like core
• Implements fine-grained sharing
• Allocation at the level of tasks within a job
• Simple, scalable application-controlled scheduling
mechanism.
10 March 2017 Intro. to Mesos - M.Zakeri Page 17 of 48

18. Fine-Grained Sharing
10 March 2017 Intro. to Mesos - M.Zakeri
Framework 1
Framework 2
Framework 3
321
654
987
Storage System (e.g. HDFS)
321
654
987
Storage System (e.g. HDFS)
Fw. 1
Fw. 1Fw. 3
Fw. 3 Fw. 2Fw. 2
Fw. 2
Fw. 1
Fw. 3
Fw. 2Fw. 3
Fw. 1
Fw. 1 Fw. 2Fw. 2
Fw. 1
Fw. 3 Fw. 3
Fw. 3
Fw. 2
Fw. 2
Page 18 of 48

19. Scheduling Mechanism: Challenges
• How to build a scalable and efficient system that supports a
wide array of both current and future frameworks?
1. Each framework have different scheduling needs.
2. Scheduling system must scale to large number of cluster
nodes.
• Running hundreds of jobs with millions of tasks.
3. System must be fault-tolerant and highly available.
10 March 2017 Intro. to Mesos - M.Zakeri Page 19 of 48

20. Scheduling Mechanism: Approaches
• Centralized Scheduling
• Frameworks express needs in a specification language,
• Global scheduler matches them to resources.
• + Can make optimal decisions.
• – Complexity,
• – Difficult to scale and to make robust,
• – Future frameworks may have unanticipated needs.
10 March 2017 Intro. to Mesos - M.Zakeri Page 20 of 48

21. Scheduling Mechanism: Approaches
• Mesos Approach: Resource Offer
• Offer available resources to frameworks,
• Delegating control over scheduling to them.
• + Keeps Mesos simple, lets it support future frameworks,
• - Decentralized decisions might not be optimal!
10 March 2017 Intro. to Mesos - M.Zakeri Page 21 of 48

22. Resource Offer
10 March 2017 Intro. to Mesos - M.Zakeri Page 22 of 48

23. Mesos Architecture
10 March 2017 Intro. to Mesos - M.Zakeri
Mesos architecture diagram, showing two running frameworks (Hadoop and MPI).
Page 23 of 48

24. Mesos Architecture: Resource Offer
• The master implements fine-grained sharing across
frameworks using resource offers.
• Each resource offer is a list of free resources on multiple
slaves.
• The master decides how many resources to offer to each
framework according to an organizational policy.
• Mesos lets organizations define their own policies via a
pluggable allocation module.
10 March 2017 Intro. to Mesos - M.Zakeri Page 24 of 48

25. Mesos Architecture: Resource Offer
• Each framework running on Mesos consists of two
components:
• A scheduler that registers with the master to be offered resources.
• An executor process that is launched on slave nodes to run the
framework’s tasks.
• Frameworks’ schedulers select which of the offered
resources to use.
10 March 2017 Intro. to Mesos - M.Zakeri Page 25 of 48

26. Resource Offer: Example
10 March 2017 Intro. to Mesos - M.Zakeri Page 26 of 48

27. Optimization: Filters
• Let frameworks short-circuit rejection by providing a
predicate on resources to be offered
• E.g. “nodes from list L” or “nodes with > 8 GB RAM”
• Could generalize to other hints as well
10 March 2017 Intro. to Mesos - M.Zakeri Page 27 of 48

28. Isolation
• Isolate resources using OS container technologies
• Linux Containers (LXC)
• Solaris Projects
• Limit the CPU, memory, network bandwidth, and (in new
Linux kernels) I/O usage of a process tree.
• Not perfect, but much better than no isolation!
10 March 2017 Intro. to Mesos - M.Zakeri Page 28 of 48

29. Fault Tolerance
• Master’s only state is the list of active slaves, active
frameworks, and running tasks.
• Run multiple masters in a hot-standby configuration
• Using ZooKeeper.
• To deal with scheduler failures, Mesos allows a framework to
register multiple schedulers.
• when one fails, another one is notified by the Mesos master to
take over.
10 March 2017 Intro. to Mesos - M.Zakeri Page 29 of 48

30. Implementation
• About 10,000 lines of C++ code.
• Runs on Linux, Solaris and OS X.
• Frameworks ported: Apache Hadoop, MPICH2, Torque
• New specialized framework: Spark
• for iterative jobs
• up to 20× faster than Hadoop
• & Open Source :)
10 March 2017 Intro. to Mesos - M.Zakeri Page 30 of 48

31. Organizations using Mesos
• Twitter uses Mesos on > 100 nodes to run ~12 production
services (mostly stream processing).
• UCSF medical researchers are using Mesos to run Hadoop
and eventually non-Hadoop apps.
• Berkeley machine learning researchers are running several
algorithms at scale on Spark.
• Read more at:
• http://mesos.apache.org/documentation/latest/powered-by-
mesos/
10 March 2017 Intro. to Mesos - M.Zakeri Page 31 of 48

32. Mesos API
10 March 2017 Intro. to Mesos - M.Zakeri Page 32 of 48

33. Evaluation
• Mesos is evaluated through a series of experiments on the
Amazon Elastic Compute Cloud (EC2).
• 96 Nodes
• Nodes with 4 CPU cores and 15 GB of RAM.
• Four workloads in two scenarios:
 96-node Mesos cluster using fair sharing
V.S.
 4 static partitions of the cluster (24 nodes per partition)
10 March 2017 Intro. to Mesos - M.Zakeri Page 33 of 48

34. Dynamic Resource Sharing
10 March 2017 Intro. to Mesos - M.Zakeri Page 34 of 48

35. Static vs Mesos: Resource Allocation
10 March 2017 Intro. to Mesos - M.Zakeri Page 35 of 48

36. Static vs Mesos: Speedup
10 March 2017 Intro. to Mesos - M.Zakeri
Framework Speedup on Mesos
Facebook Hadoop Mix 1.14×
Large Hadoop Mix 2.10×
Spark 1.26×
Torque / MPI 0.96×
Page 36 of 48

37. Static vs Mesos: Utilization
10 March 2017 Intro. to Mesos - M.Zakeri Page 37 of 48

38. Mesos Overhead
• Ran two benchmarks using MPI and Hadoop on an EC2
cluster with 50 nodes.
• 2 CPU cores and 6.5 GB RAM
• The MPI job (LINPACK benchmark):
• ~50.9s without Mesos  ~51.8s with Mesos
• Hadoop job (WordCount job):
• ~160s without Mesos  ~166s with Mesos
• Result
• In both cases, the overhead of using Mesos was less than 4%
10 March 2017 Intro. to Mesos - M.Zakeri Page 38 of 48

39. Data Locality with Resource Offers
• Ran 16 instances of Hadoop on a shared HDFS cluster
• Using 93 EC2 nodes
• 4 CPU cores and 15 GB RAM
• Used delay scheduling in Hadoop to get locality (wait a short
time to acquire data-local nodes)
• Running the Hadoop instances on Mesos improves data
locality.
10 March 2017 Intro. to Mesos - M.Zakeri Page 39 of 48

40. Data Locality with Resource Offers
10 March 2017 Intro. to Mesos - M.Zakeri Page 40 of 48

41. Scalability
• Mesos only performs inter-framework scheduling (e.g. fair
sharing), which is easier than intra-framework scheduling
• Result:
• Scaled to 50,000 emulated slaves
• 200 frameworks
• 100K tasks (30s len)
10 March 2017 Intro. to Mesos - M.Zakeri Page 41 of 48

42. Scalability
10 March 2017 Intro. to Mesos - M.Zakeri Page 42 of 48

43. Results Analysis
• Resource offer works well when:
• Frameworks can scale up and down elastically
• Task durations are homogeneous
• Frameworks have many preferred nodes
• Otherwise Mesos may not be useful.
• E.g. in the case of Torque / MPI
• Programming against Mesos is not easy :)
10 March 2017 Intro. to Mesos - M.Zakeri Page 43 of 48

44. Related Work
• HPC schedulers (e.g. Sun Grid Engine, Torque, …)
• Coarse-grained sharing for inelastic jobs (e.g. MPI jobs).
• Virtual machine clouds
• Coarse-grained sharing similar to HPC schedulers.
• Condor
• Centralized scheduler based on matchmaking
• …
10 March 2017 Intro. to Mesos - M.Zakeri Page 44 of 48

45. Conclusion
• Mesos shares clusters efficiently among diverse frameworks.
• Two design elements:
• Fine-grained sharing, at the level of tasks,
• Resource offers, a scalable mechanism for application-controlled
scheduling.
• Enables co-existence of current frameworks and
development of new specialized ones.
• In use at Twitter, UC Berkeley, …
10 March 2017 Intro. to Mesos - M.Zakeri Page 45 of 48

46. Conclusion
• What do you think about this?

10 March 2017 Intro. to Mesos - M.Zakeri Page 46 of 48
Hardware
OS
MESOS

47. References
• [1] Hindman, B., Konwinski, A., Matei, Z., Ghodsi, A., D.Joseph, A., Katz, R., …
Stoica, I. (2011). Mesos: A platform for fine-grained resource sharing in the data
center. Proceedings of the …, 32. https://doi.org/10.1109/TIM.2009.2038002
• [2] Apache Mesos. (n.d.). Retrieved from http://mesos.apache.org/ ,2017
• [3] Apache Hadoop. (n.d.). Retrieved from https://hadoop.apache.org/, 2017
• [4] Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., & Stoica, I.
(2010). Delay scheduling. Proceedings of the 5th European Conference on
Computer Systems - EuroSys ’10, 265. https://doi.org/10.1145/1755913.1755940
10 March 2017 Intro. to Mesos - M.Zakeri Page 47 of 48

48. Any QUESTIONS?
 m-zakeri@live.com