SlideShare uma empresa Scribd logo
1 de 65
Baixar para ler offline
L E S S O N S L E A R N E D AT T W I T T E R
H A D O O P P E R F O R M A N C E
O P T I M I Z AT I O N AT S C A L E
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8
DATA PLATFORM @TWITTER
Develop, maintain, and support the core data processing
libraries used at Twitter
In a good position to make system-wide performance
improvements
Core Data Libraries Team
DATA PLATFORM @TWITTER
Idiomatic functional Scala library for writing Hadoop map reduce
Functional programming is a natural fit for map reduce
Compile time type checked
Core Data Libraries Team
github.com/twitter/scalding
DATA PLATFORM @TWITTER
Columnar storage format for the Hadoop ecosystem
Uses the Google Dremel column shredding and assembly
algorithm
Core Data Libraries Team
APACHE PARQUET
github.com/apache/parquet-mr
DATA PLATFORM @TWITTER
Streaming map reduce for hybrid realtime / batch topologies
Write once, execute in parallel on Storm / Heron (online) and
Scalding (offline)
Core Data Libraries Team
SUMMINGBIRD
github.com/twitter/summingbird
Hadoop at Twitter Scale
H A D O O P AT T W I T T E R
300+PETABYTES OF
DATA
100k MAP REDUCE
JOBS DAILY
MULTIPLES OF
1000+MACHINE
HADOOP
CLUSTERS
MULTIPLE
LARGEST
HADOOP
CLUSTERS IN
THE WORLD
AMONG THE
At this scale, even small system-wide
improvements can save significant
amounts of compute resources
C O S T AT S C A L E
What does your Hadoop cluster
spend most of its time doing?
W H AT T O I M P R O V E ?
Profile your cluster, you might be
surprised by what you find
M E A S U R E - D O N ' T G U E S S
ENABLE JVM PROFILING WITH -XPROF
Built into the JVM (HotSpot), so there's nothing to install
Xprof: a low overhead profiler built into the jvm
mapreduce.task.profile='true'
mapreduce.task.profile.maps='0-'
mapreduce.task.profile.reduces='0-'
mapreduce.task.profile.params='-Xprof'
ENABLE JVM PROFILING WITH -XPROF
Low overhead (uses stack sampling)
Surfaces the most expensive methods
Prints directly to task logs (stdout)
Xprof: a low overhead profiler built into the jvm
Flat profile of 412.48 secs (38743 total ticks): SpillThread
Interpreted + native Method
12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect
4.6% 0 + 822 java.io.FileOutputStream.writeBytes
...
19.4% 352 + 3082 Total interpreted (including elided)
Compiled + native Method
50.0% 8549 + 299 java.lang.StringCoding.decode
16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement
4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode
2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write
2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare
1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare
...
79.0% 13514 + 467 Total compiled
Thread-local ticks:
54.3% 21053 Blocked (of total)
HADOOP CONFIGURATION OBJECT
Looks and behaves a lot like a HashMap
Surprisingly expensive
Configuration conf = new Configuration()
conf.set("myKey", "myValue")
String value = conf.get("myKey")
HADOOP CONFIGURATION OBJECT
Constructor reads + unzips + parses an XML file from disk
Surprisingly expensive
public class KryoSerialization {
public KryoSerialization() {
this(new Configuration())
}
}
HADOOP CONFIGURATION OBJECT
get() method involves regular expressions, variable substitution
Surprisingly expensive
String value = conf.get("myKey")
HADOOP CONFIGURATION OBJECT
Calling these methods in a loop, or once per record, is
expensive
Some (non trivial) jobs were spending 30% of their time in
Configuration methods
Surprisingly expensive
It's hard to predict what needs to
be optimized without a profiler
L E S S O N L E A R N E D
If you don't profile, you could be
missing easy wins
L E S S O N L E A R N E D
Measure whether IO or CPU is your
biggest cost
L E S S O N L E A R N E D
INTERMEDIATE COMPRESSION
Xprof surfaced that compression + decompression in the spill
thread was taking a lot of time
Intermediate outputs are temporary
We now use lz4 instead of lzo level 3, which produces 30%
larger intermediate data that's faster to read
Made some large jobs 1.5X faster
Find the right balance
Record Serialization + Deserialization
can be the most expensive part of
your job
L E S S O N L E A R N E D
Record Serialization is CPU intensive,
and may overshadow IO
L E S S O N L E A R N E D
How to reduce costs due to record
serialization?
L E S S O N L E A R N E D
USE HADOOP'S RAW COMPARATOR API
Hadoop MR deserializes the map output keys in order to sort
them between the map and reduce phases
Don't make sorting more expensive than it already is
deserialize(keyBytes1).compare(deserialize(keyBytes2))
USE HADOOP'S RAW COMPARATOR API
This can cost a lot, especially for complex non-primitive keys,
which is fairly common
Don't make sorting more expensive than it already is
requests.groupBy { req => (req.country, req.client) }
USE HADOOP'S RAW COMPARATOR API
This can cost a lot, especially for complex non-primitive keys,
which is fairly common
Don't make sorting more expensive than it already is
Complex object
that requires sorting
requests.groupBy { req => (req.country, req.client) }
Flat profile of 412.48 secs (38743 total ticks): SpillThread
Interpreted + native Method
12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect
4.6% 0 + 822 java.io.FileOutputStream.writeBytes
...
19.4% 352 + 3082 Total interpreted (including elided)
Compiled + native Method
50.0% 8549 + 299 java.lang.StringCoding.decode
16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement
4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode
2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write
2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare
1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare
...
79.0% 13514 + 467 Total compiled
Thread-local ticks:
54.3% 21053 Blocked (of total)
USE HADOOP'S RAW COMPARATOR API
Hadoop comes with a RawComparator API for comparing
records in their serialized (raw) form
Don't make sorting more expensive than it already is
deserialize(keyBytes1).compare(deserialize(keyBytes2))
compare(keyBytes1, keyBytes2)
USE HADOOP'S RAW COMPARATOR API
Hadoop comes with a RawComparator API for comparing
records in their serialized (raw) form
Don't make sorting more expensive than it already is
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2);
}
USE HADOOP'S RAW COMPARATOR API
Unfortunately, this requires you to write a custom comparator
by hand
And assumes that your data is actually easy to compare in its
serialized form
Don't make sorting more expensive than it already is
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2);
}
SCALA MACROS FOR RAW COMPARATORS
Macros to the rescue!
A slightly more hipster API for Raw Comparators in Scala
And a handful of macros to generate implementations of this
API for tuples, case classes, thrift objects, primitives, Strings,
etc.
SCALA MACROS FOR RAW COMPARATORS
1 3 f o o 0 1 17 1 88 ...
Macros to the rescue!
First, creates a custom dense serialization format that's easy to
compare
1 3 f o o 0 1 22 0 ... ...
non-null String
null value
non-null int non-null int
null value
SCALA MACROS FOR RAW COMPARATORS
1 3 f o o 0 1 17 1 88 ...
Macros to the rescue!
Then, creates a compare method that takes advantage of this
format
1 3 f 0 o 0 1 22 0 ... ...
SCALA MACROS FOR RAW COMPARATORS
Macros to the rescue!
TotalComputeTime
Default Raw Comparators
1.5X
FASTER
How to reduce costs due to record
serialization?
L E S S O N L E A R N E D
COLUMN PROJECTION
Don't read or deserialize data that you don't need
struct User {
1: i64 id
2: Address address
3: string name
4 list<Interest> interests
}
COLUMN PROJECTION
Columnar file formats like Apache Parquet support this directly
Specialized record deserializers can skip over unwanted fields
in row oriented storage
Don't read or deserialize data that you don't need
APACHE PARQUET
Columnar storage for the people
In traditional row-oriented storage layout, an entire record is
stored sequentially
R1.A R1.B R1.C R2.A R2.B R2.C R3.A R3.B R3.C
APACHE PARQUET
Columnar storage for the people
In traditional row-oriented storage layout, an entire record is
stored sequentially
9903489083
"123 elm street"
"alice"
"columnar file formats"
9903489084
"333 oak street"
"bob"
"Hadoop"
Compressed with lzo / gzip / snappy
APACHE PARQUET
Columnar storage for the people
In columnar storage layout, an entire column is stored
sequentially
R1.A R2.A R3.A R1.B R2.B R3.B R1.C R2.C R3.C
APACHE PARQUET
Columnar storage for the people
All user ids stored together
In columnar storage layout, an entire column is stored
sequentially
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
"123 elm street"
"333 oak street"
"827 maple drive"
APACHE PARQUET
Columnar storage for the people
Schema aware storage can use specialized encodings
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
9903489083
+1
+1
-10
+3
-1
delta
"twitter.com/foo/bar"
"blog.twitter.com"
"twitter.com/foo/bar"
"twitter.com/foo/bar"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com/123"
"twitter.com/foo/bar": 0
"blog.twitter.com": 1
"blog.twitter.com/123": 2
0
1
0
0
1
1
1
2
dictionary
FILE SIZE COMPARISON
SizeinGB
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
2X
SMALLER
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
APACHE PARQUET
Columnar storage for the people
Collocating entire columns allows for efficient
column projection
Read off disk only the columns you need
Possibly more importantly: deserialize only
the columns you need
TotalComputeTime
1 column 10 columns 40 columns
Parquet Lzo Thrift
COLUMN PROJECTION WITH PARQUET
3X
FASTER
1.5X
FASTER
1.15X
FASTER
TotalComputeTime
1 column 10 columns 40 columns
Parquet Lzo Thrift
COLUMN PROJECTION WITH PARQUET
3X
FASTER
1.5X
FASTER
1.15X
FASTER
APACHE PARQUET
Columnar storage for the people
Parquet is often slower to read all columns than row oriented
storage
Parquet is a dense format, read performance scales with the
number of columns in the schema -- nulls take time read
Sparse, row oriented formats (thrift) scale with the number of
columns present in the data -- nulls take no time read
COLUMN PROJECTION FOR ROW ORIENTED DATA
Row oriented is a very common way to store Thrift, Avro,
Protocol Buffers, etc.
Specialized record deserializers can skip over unwanted fields
in these row oriented storage formats
Prototype implemented as a Scala macro that creates a custom
deserializer at compile time
Don't deserialize data that you don't need
COLUMN PROJECTION FOR ROW ORIENTED DATA
Don't deserialize data that you don't need
198 111 121 054 e l m _ s t r ... a l i c e ...
Decode User Id to Long
Skip over unwanted address field
Decode Name to String
COLUMN PROJECTION FOR ROW ORIENTED DATA
No IO savings
But only decodes the fields you care about into objects
CPU time spent decoding Strings can be huge compared to
time it takes to load + ignore the encoded bytes
Don't deserialize data that you don't need
TotalComputeTime
Number of Columns Selected
1 7 10 13 48
Parquet Thrift
Parquet Pig
Lzo Thrift + Projection
COLUMN PROJECTION: THRIFT VS. PARQUET
Parquet Thrift has a lot
of room for
improvement
Parquet faster than row
oriented until 13 columns
This schema is relatively
flat, and most columns
populated
APACHE PARQUET
Columnar storage for the people
Predicate push-down also allows parquet to
skip over records that don't match your filter
criteria
Parquet stores statistics about chunks of
records, so in some cases entire chunks of
data can be skipped after examining these
statistics
APACHE PARQUET
Columnar storage for the people
Combining both column projection and predicate push
down is a powerful combination
TotalComputeTime
Lzo Thrift Parquet + Filter Parquet + Filter + Project
FILTER PUSH DOWN WITH PARQUET
4.3X
FASTER
APACHE PARQUET
Columnar storage for the people
Predicate push-down performance depends on the nature
of the filter
Searching for rare records is the best case, entire chunks of
records are likely to not contain the records you are looking
for
Key take aways
I N S U M M A R Y
IN SUMMARY
Key takeaways
Profile!
Serialization is expensive, and Hadoop does a lot of it
Choose a storage format that fits your access patterns
Use column projection
Sorting is expensive -- use Raw Comparators
IO may not be your bottleneck -- more IO for less CPU may be
a good tradeoff
ACKNOWLEDGEMENTS
Thanks to everyone involved!
Dmitriy Ryaboy @squarecog
Gera Shegalov @gerashegalov
Julien Le Dem @J_
Katya Gonina @katyagonina
Mansur Ashraf @mansur_ashraf
Oscar Boykin @posco
Sriram Krishnan @krishnansriram
Tianshuo Deng @tsdeng
Zak Taylor @zakattacktaylor
And many more!
GET INVOLVED
Contributions always welcome!
github.com/twitter/scalding
github.com/twitter/algebird
github.com/twitter/chill
github.com/apache/parquet-mr
JOIN THE FLOCK
We're Hiring!
Work on data processing challenges at scale
Strong commitment to open source
jobs.twitter.com
Data Platform: (https://about.twitter.com/careers/positions?jvi=oipMYfwb,Job)
Q U E S T I O N S ?
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8

Mais conteúdo relacionado

Mais procurados

Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guideRyan Blue
 
Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data AnalyticsIceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data AnalyticsAlluxio, Inc.
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesDatabricks
 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark InternalsPietro Michiardi
 
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...HostedbyConfluent
 
Efficient Data Storage for Analytics with Apache Parquet 2.0
Efficient Data Storage for Analytics with Apache Parquet 2.0Efficient Data Storage for Analytics with Apache Parquet 2.0
Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.
 
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...StreamNative
 
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingIntroduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent
 
Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0Databricks
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc.
 
Smart Join Algorithms for Fighting Skew at Scale
Smart Join Algorithms for Fighting Skew at ScaleSmart Join Algorithms for Fighting Skew at Scale
Smart Join Algorithms for Fighting Skew at ScaleDatabricks
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemDatabricks
 
Apache Spark's Built-in File Sources in Depth
Apache Spark's Built-in File Sources in DepthApache Spark's Built-in File Sources in Depth
Apache Spark's Built-in File Sources in DepthDatabricks
 
Spark SQL Join Improvement at Facebook
Spark SQL Join Improvement at FacebookSpark SQL Join Improvement at Facebook
Spark SQL Join Improvement at FacebookDatabricks
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcachedJurriaan Persyn
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseDataWorks Summit
 
Apache Arrow: High Performance Columnar Data Framework
Apache Arrow: High Performance Columnar Data FrameworkApache Arrow: High Performance Columnar Data Framework
Apache Arrow: High Performance Columnar Data FrameworkWes McKinney
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Databricks
 

Mais procurados (20)

Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
 
Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data AnalyticsIceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data Analytics
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark Internals
 
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...
Keep Your Cache Always Fresh with Debezium! with Gunnar Morling | Kafka Summi...
 
Efficient Data Storage for Analytics with Apache Parquet 2.0
Efficient Data Storage for Analytics with Apache Parquet 2.0Efficient Data Storage for Analytics with Apache Parquet 2.0
Efficient Data Storage for Analytics with Apache Parquet 2.0
 
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
 
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingIntroduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processing
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
 
Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
Smart Join Algorithms for Fighting Skew at Scale
Smart Join Algorithms for Fighting Skew at ScaleSmart Join Algorithms for Fighting Skew at Scale
Smart Join Algorithms for Fighting Skew at Scale
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
 
Apache Spark's Built-in File Sources in Depth
Apache Spark's Built-in File Sources in DepthApache Spark's Built-in File Sources in Depth
Apache Spark's Built-in File Sources in Depth
 
Spark SQL Join Improvement at Facebook
Spark SQL Join Improvement at FacebookSpark SQL Join Improvement at Facebook
Spark SQL Join Improvement at Facebook
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data Warehouse
 
Apache Arrow: High Performance Columnar Data Framework
Apache Arrow: High Performance Columnar Data FrameworkApache Arrow: High Performance Columnar Data Framework
Apache Arrow: High Performance Columnar Data Framework
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
 

Destaque

Pysparkで始めるデータ分析
Pysparkで始めるデータ分析Pysparkで始めるデータ分析
Pysparkで始めるデータ分析Tanaka Yuichi
 
Pythonで入門するApache Spark at PyCon2016
Pythonで入門するApache Spark at PyCon2016Pythonで入門するApache Spark at PyCon2016
Pythonで入門するApache Spark at PyCon2016Tatsuya Atsumi
 
Redshift Spectrumを使ってみた話
Redshift Spectrumを使ってみた話Redshift Spectrumを使ってみた話
Redshift Spectrumを使ってみた話Yoshiki Kouno
 
AWS Black Belt Online Seminar 2017 Amazon Connect
AWS Black Belt Online Seminar 2017 Amazon ConnectAWS Black Belt Online Seminar 2017 Amazon Connect
AWS Black Belt Online Seminar 2017 Amazon ConnectAmazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon GameLift
AWS Black Belt Online Seminar 2017 Amazon GameLiftAWS Black Belt Online Seminar 2017 Amazon GameLift
AWS Black Belt Online Seminar 2017 Amazon GameLiftAmazon Web Services Japan
 
Docker入門: コンテナ型仮想化技術の仕組みと使い方
Docker入門: コンテナ型仮想化技術の仕組みと使い方Docker入門: コンテナ型仮想化技術の仕組みと使い方
Docker入門: コンテナ型仮想化技術の仕組みと使い方Yuichi Ito
 
AWS Black Belt Online Seminar 2017 Amazon EMR
AWS Black Belt Online Seminar 2017 Amazon EMR AWS Black Belt Online Seminar 2017 Amazon EMR
AWS Black Belt Online Seminar 2017 Amazon EMR Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハックAWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハックAmazon Web Services Japan
 
Amazon Athena 初心者向けハンズオン
Amazon Athena 初心者向けハンズオンAmazon Athena 初心者向けハンズオン
Amazon Athena 初心者向けハンズオンAmazon Web Services Japan
 

Destaque (13)

Pysparkで始めるデータ分析
Pysparkで始めるデータ分析Pysparkで始めるデータ分析
Pysparkで始めるデータ分析
 
Pythonで入門するApache Spark at PyCon2016
Pythonで入門するApache Spark at PyCon2016Pythonで入門するApache Spark at PyCon2016
Pythonで入門するApache Spark at PyCon2016
 
Redshift Spectrumを使ってみた話
Redshift Spectrumを使ってみた話Redshift Spectrumを使ってみた話
Redshift Spectrumを使ってみた話
 
Alibaba Cloud
Alibaba CloudAlibaba Cloud
Alibaba Cloud
 
AWS Black Belt Online Seminar 2017 Amazon Connect
AWS Black Belt Online Seminar 2017 Amazon ConnectAWS Black Belt Online Seminar 2017 Amazon Connect
AWS Black Belt Online Seminar 2017 Amazon Connect
 
AWS Black Belt - AWS Glue
AWS Black Belt - AWS GlueAWS Black Belt - AWS Glue
AWS Black Belt - AWS Glue
 
AWS Black Belt Online Seminar 2017 Amazon GameLift
AWS Black Belt Online Seminar 2017 Amazon GameLiftAWS Black Belt Online Seminar 2017 Amazon GameLift
AWS Black Belt Online Seminar 2017 Amazon GameLift
 
Docker入門: コンテナ型仮想化技術の仕組みと使い方
Docker入門: コンテナ型仮想化技術の仕組みと使い方Docker入門: コンテナ型仮想化技術の仕組みと使い方
Docker入門: コンテナ型仮想化技術の仕組みと使い方
 
AWS BlackBelt AWS上でのDDoS対策
AWS BlackBelt AWS上でのDDoS対策AWS BlackBelt AWS上でのDDoS対策
AWS BlackBelt AWS上でのDDoS対策
 
AWS Black Belt Online Seminar 2017 Amazon EMR
AWS Black Belt Online Seminar 2017 Amazon EMR AWS Black Belt Online Seminar 2017 Amazon EMR
AWS Black Belt Online Seminar 2017 Amazon EMR
 
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
 
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハックAWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
 
Amazon Athena 初心者向けハンズオン
Amazon Athena 初心者向けハンズオンAmazon Athena 初心者向けハンズオン
Amazon Athena 初心者向けハンズオン
 

Semelhante a Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson)

Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterHadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterDataWorks Summit
 
Apache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesApache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesWalaa Hamdy Assy
 
AWS Hadoop and PIG and overview
AWS Hadoop and PIG and overviewAWS Hadoop and PIG and overview
AWS Hadoop and PIG and overviewDan Morrill
 
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
Apache Spark Workshop, Apr. 2016, Euangelos LinardosApache Spark Workshop, Apr. 2016, Euangelos Linardos
Apache Spark Workshop, Apr. 2016, Euangelos LinardosEuangelos Linardos
 
2014 sept 26_thug_lambda_part1
2014 sept 26_thug_lambda_part12014 sept 26_thug_lambda_part1
2014 sept 26_thug_lambda_part1Adam Muise
 
OVERVIEW ON SPARK.pptx
OVERVIEW ON SPARK.pptxOVERVIEW ON SPARK.pptx
OVERVIEW ON SPARK.pptxAishg4
 
Katello on TorqueBox
Katello on TorqueBoxKatello on TorqueBox
Katello on TorqueBoxlzap
 
Transformation Processing Smackdown; Spark vs Hive vs Pig
Transformation Processing Smackdown; Spark vs Hive vs PigTransformation Processing Smackdown; Spark vs Hive vs Pig
Transformation Processing Smackdown; Spark vs Hive vs PigLester Martin
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupNed Shawa
 
Data science with spark on amazon EMR - Pop-up Loft Tel Aviv
Data science with spark on amazon EMR - Pop-up Loft Tel AvivData science with spark on amazon EMR - Pop-up Loft Tel Aviv
Data science with spark on amazon EMR - Pop-up Loft Tel AvivAmazon Web Services
 
Sparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With SparkSparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With SparkIan Pointer
 
(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data Platform(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data PlatformAmazon Web Services
 
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labs
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labsApache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labs
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labsViswanath Gangavaram
 
Running Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data PlatformRunning Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data PlatformEva Tse
 
SparkR: Enabling Interactive Data Science at Scale on Hadoop
SparkR: Enabling Interactive Data Science at Scale on HadoopSparkR: Enabling Interactive Data Science at Scale on Hadoop
SparkR: Enabling Interactive Data Science at Scale on HadoopDataWorks Summit
 
Strata NYC 2015 - Supercharging R with Apache Spark
Strata NYC 2015 - Supercharging R with Apache SparkStrata NYC 2015 - Supercharging R with Apache Spark
Strata NYC 2015 - Supercharging R with Apache SparkDatabricks
 

Semelhante a Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson) (20)

Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterHadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
 
Apache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesApache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & libraries
 
AWS Hadoop and PIG and overview
AWS Hadoop and PIG and overviewAWS Hadoop and PIG and overview
AWS Hadoop and PIG and overview
 
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
Apache Spark Workshop, Apr. 2016, Euangelos LinardosApache Spark Workshop, Apr. 2016, Euangelos Linardos
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
 
Osd ctw spark
Osd ctw sparkOsd ctw spark
Osd ctw spark
 
2014 sept 26_thug_lambda_part1
2014 sept 26_thug_lambda_part12014 sept 26_thug_lambda_part1
2014 sept 26_thug_lambda_part1
 
OVERVIEW ON SPARK.pptx
OVERVIEW ON SPARK.pptxOVERVIEW ON SPARK.pptx
OVERVIEW ON SPARK.pptx
 
Data Science
Data ScienceData Science
Data Science
 
Katello on TorqueBox
Katello on TorqueBoxKatello on TorqueBox
Katello on TorqueBox
 
Transformation Processing Smackdown; Spark vs Hive vs Pig
Transformation Processing Smackdown; Spark vs Hive vs PigTransformation Processing Smackdown; Spark vs Hive vs Pig
Transformation Processing Smackdown; Spark vs Hive vs Pig
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetup
 
Data science with spark on amazon EMR - Pop-up Loft Tel Aviv
Data science with spark on amazon EMR - Pop-up Loft Tel AvivData science with spark on amazon EMR - Pop-up Loft Tel Aviv
Data science with spark on amazon EMR - Pop-up Loft Tel Aviv
 
February 2014 HUG : Pig On Tez
February 2014 HUG : Pig On TezFebruary 2014 HUG : Pig On Tez
February 2014 HUG : Pig On Tez
 
Masterclass Live: Amazon EMR
Masterclass Live: Amazon EMRMasterclass Live: Amazon EMR
Masterclass Live: Amazon EMR
 
Sparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With SparkSparklife - Life In The Trenches With Spark
Sparklife - Life In The Trenches With Spark
 
(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data Platform(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data Platform
 
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labs
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labsApache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labs
Apache pig power_tools_by_viswanath_gangavaram_r&d_dsg_i_labs
 
Running Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data PlatformRunning Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data Platform
 
SparkR: Enabling Interactive Data Science at Scale on Hadoop
SparkR: Enabling Interactive Data Science at Scale on HadoopSparkR: Enabling Interactive Data Science at Scale on Hadoop
SparkR: Enabling Interactive Data Science at Scale on Hadoop
 
Strata NYC 2015 - Supercharging R with Apache Spark
Strata NYC 2015 - Supercharging R with Apache SparkStrata NYC 2015 - Supercharging R with Apache Spark
Strata NYC 2015 - Supercharging R with Apache Spark
 

Último

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 

Último (20)

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson)

  • 1. L E S S O N S L E A R N E D AT T W I T T E R H A D O O P P E R F O R M A N C E O P T I M I Z AT I O N AT S C A L E A L E X L E V E N S O N | I A N O ' C O N N E L L | @ T H I S W I L LW O R K @ 0 X 1 3 8
  • 2. DATA PLATFORM @TWITTER Develop, maintain, and support the core data processing libraries used at Twitter In a good position to make system-wide performance improvements Core Data Libraries Team
  • 3. DATA PLATFORM @TWITTER Idiomatic functional Scala library for writing Hadoop map reduce Functional programming is a natural fit for map reduce Compile time type checked Core Data Libraries Team github.com/twitter/scalding
  • 4. DATA PLATFORM @TWITTER Columnar storage format for the Hadoop ecosystem Uses the Google Dremel column shredding and assembly algorithm Core Data Libraries Team APACHE PARQUET github.com/apache/parquet-mr
  • 5. DATA PLATFORM @TWITTER Streaming map reduce for hybrid realtime / batch topologies Write once, execute in parallel on Storm / Heron (online) and Scalding (offline) Core Data Libraries Team SUMMINGBIRD github.com/twitter/summingbird
  • 6. Hadoop at Twitter Scale H A D O O P AT T W I T T E R
  • 8. 100k MAP REDUCE JOBS DAILY MULTIPLES OF
  • 11. At this scale, even small system-wide improvements can save significant amounts of compute resources C O S T AT S C A L E
  • 12. What does your Hadoop cluster spend most of its time doing? W H AT T O I M P R O V E ?
  • 13. Profile your cluster, you might be surprised by what you find M E A S U R E - D O N ' T G U E S S
  • 14. ENABLE JVM PROFILING WITH -XPROF Built into the JVM (HotSpot), so there's nothing to install Xprof: a low overhead profiler built into the jvm mapreduce.task.profile='true' mapreduce.task.profile.maps='0-' mapreduce.task.profile.reduces='0-' mapreduce.task.profile.params='-Xprof'
  • 15. ENABLE JVM PROFILING WITH -XPROF Low overhead (uses stack sampling) Surfaces the most expensive methods Prints directly to task logs (stdout) Xprof: a low overhead profiler built into the jvm
  • 16. Flat profile of 412.48 secs (38743 total ticks): SpillThread Interpreted + native Method 12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect 4.6% 0 + 822 java.io.FileOutputStream.writeBytes ... 19.4% 352 + 3082 Total interpreted (including elided) Compiled + native Method 50.0% 8549 + 299 java.lang.StringCoding.decode 16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement 4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode 2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write 2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare 1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare ... 79.0% 13514 + 467 Total compiled Thread-local ticks: 54.3% 21053 Blocked (of total)
  • 17. HADOOP CONFIGURATION OBJECT Looks and behaves a lot like a HashMap Surprisingly expensive Configuration conf = new Configuration() conf.set("myKey", "myValue") String value = conf.get("myKey")
  • 18. HADOOP CONFIGURATION OBJECT Constructor reads + unzips + parses an XML file from disk Surprisingly expensive public class KryoSerialization { public KryoSerialization() { this(new Configuration()) } }
  • 19. HADOOP CONFIGURATION OBJECT get() method involves regular expressions, variable substitution Surprisingly expensive String value = conf.get("myKey")
  • 20. HADOOP CONFIGURATION OBJECT Calling these methods in a loop, or once per record, is expensive Some (non trivial) jobs were spending 30% of their time in Configuration methods Surprisingly expensive
  • 21. It's hard to predict what needs to be optimized without a profiler L E S S O N L E A R N E D
  • 22. If you don't profile, you could be missing easy wins L E S S O N L E A R N E D
  • 23. Measure whether IO or CPU is your biggest cost L E S S O N L E A R N E D
  • 24. INTERMEDIATE COMPRESSION Xprof surfaced that compression + decompression in the spill thread was taking a lot of time Intermediate outputs are temporary We now use lz4 instead of lzo level 3, which produces 30% larger intermediate data that's faster to read Made some large jobs 1.5X faster Find the right balance
  • 25. Record Serialization + Deserialization can be the most expensive part of your job L E S S O N L E A R N E D
  • 26. Record Serialization is CPU intensive, and may overshadow IO L E S S O N L E A R N E D
  • 27. How to reduce costs due to record serialization? L E S S O N L E A R N E D
  • 28. USE HADOOP'S RAW COMPARATOR API Hadoop MR deserializes the map output keys in order to sort them between the map and reduce phases Don't make sorting more expensive than it already is deserialize(keyBytes1).compare(deserialize(keyBytes2))
  • 29. USE HADOOP'S RAW COMPARATOR API This can cost a lot, especially for complex non-primitive keys, which is fairly common Don't make sorting more expensive than it already is requests.groupBy { req => (req.country, req.client) }
  • 30. USE HADOOP'S RAW COMPARATOR API This can cost a lot, especially for complex non-primitive keys, which is fairly common Don't make sorting more expensive than it already is Complex object that requires sorting requests.groupBy { req => (req.country, req.client) }
  • 31. Flat profile of 412.48 secs (38743 total ticks): SpillThread Interpreted + native Method 12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect 4.6% 0 + 822 java.io.FileOutputStream.writeBytes ... 19.4% 352 + 3082 Total interpreted (including elided) Compiled + native Method 50.0% 8549 + 299 java.lang.StringCoding.decode 16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement 4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode 2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write 2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare 1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare ... 79.0% 13514 + 467 Total compiled Thread-local ticks: 54.3% 21053 Blocked (of total)
  • 32. USE HADOOP'S RAW COMPARATOR API Hadoop comes with a RawComparator API for comparing records in their serialized (raw) form Don't make sorting more expensive than it already is deserialize(keyBytes1).compare(deserialize(keyBytes2)) compare(keyBytes1, keyBytes2)
  • 33. USE HADOOP'S RAW COMPARATOR API Hadoop comes with a RawComparator API for comparing records in their serialized (raw) form Don't make sorting more expensive than it already is public interface RawComparator<T> { public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2); }
  • 34. USE HADOOP'S RAW COMPARATOR API Unfortunately, this requires you to write a custom comparator by hand And assumes that your data is actually easy to compare in its serialized form Don't make sorting more expensive than it already is public interface RawComparator<T> { public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2); }
  • 35. SCALA MACROS FOR RAW COMPARATORS Macros to the rescue! A slightly more hipster API for Raw Comparators in Scala And a handful of macros to generate implementations of this API for tuples, case classes, thrift objects, primitives, Strings, etc.
  • 36. SCALA MACROS FOR RAW COMPARATORS 1 3 f o o 0 1 17 1 88 ... Macros to the rescue! First, creates a custom dense serialization format that's easy to compare 1 3 f o o 0 1 22 0 ... ... non-null String null value non-null int non-null int null value
  • 37. SCALA MACROS FOR RAW COMPARATORS 1 3 f o o 0 1 17 1 88 ... Macros to the rescue! Then, creates a compare method that takes advantage of this format 1 3 f 0 o 0 1 22 0 ... ...
  • 38. SCALA MACROS FOR RAW COMPARATORS Macros to the rescue! TotalComputeTime Default Raw Comparators 1.5X FASTER
  • 39. How to reduce costs due to record serialization? L E S S O N L E A R N E D
  • 40. COLUMN PROJECTION Don't read or deserialize data that you don't need struct User { 1: i64 id 2: Address address 3: string name 4 list<Interest> interests }
  • 41. COLUMN PROJECTION Columnar file formats like Apache Parquet support this directly Specialized record deserializers can skip over unwanted fields in row oriented storage Don't read or deserialize data that you don't need
  • 42. APACHE PARQUET Columnar storage for the people In traditional row-oriented storage layout, an entire record is stored sequentially R1.A R1.B R1.C R2.A R2.B R2.C R3.A R3.B R3.C
  • 43. APACHE PARQUET Columnar storage for the people In traditional row-oriented storage layout, an entire record is stored sequentially 9903489083 "123 elm street" "alice" "columnar file formats" 9903489084 "333 oak street" "bob" "Hadoop" Compressed with lzo / gzip / snappy
  • 44. APACHE PARQUET Columnar storage for the people In columnar storage layout, an entire column is stored sequentially R1.A R2.A R3.A R1.B R2.B R3.B R1.C R2.C R3.C
  • 45. APACHE PARQUET Columnar storage for the people All user ids stored together In columnar storage layout, an entire column is stored sequentially 9903489083 9903489084 9903489085 9903489075 9903489088 9903489087 "123 elm street" "333 oak street" "827 maple drive"
  • 46. APACHE PARQUET Columnar storage for the people Schema aware storage can use specialized encodings 9903489083 9903489084 9903489085 9903489075 9903489088 9903489087 9903489083 +1 +1 -10 +3 -1 delta "twitter.com/foo/bar" "blog.twitter.com" "twitter.com/foo/bar" "twitter.com/foo/bar" "blog.twitter.com" "blog.twitter.com" "blog.twitter.com" "blog.twitter.com/123" "twitter.com/foo/bar": 0 "blog.twitter.com": 1 "blog.twitter.com/123": 2 0 1 0 0 1 1 1 2 dictionary
  • 47. FILE SIZE COMPARISON SizeinGB B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet 2X SMALLER B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
  • 48. APACHE PARQUET Columnar storage for the people Collocating entire columns allows for efficient column projection Read off disk only the columns you need Possibly more importantly: deserialize only the columns you need
  • 49. TotalComputeTime 1 column 10 columns 40 columns Parquet Lzo Thrift COLUMN PROJECTION WITH PARQUET 3X FASTER 1.5X FASTER 1.15X FASTER
  • 50. TotalComputeTime 1 column 10 columns 40 columns Parquet Lzo Thrift COLUMN PROJECTION WITH PARQUET 3X FASTER 1.5X FASTER 1.15X FASTER
  • 51. APACHE PARQUET Columnar storage for the people Parquet is often slower to read all columns than row oriented storage Parquet is a dense format, read performance scales with the number of columns in the schema -- nulls take time read Sparse, row oriented formats (thrift) scale with the number of columns present in the data -- nulls take no time read
  • 52. COLUMN PROJECTION FOR ROW ORIENTED DATA Row oriented is a very common way to store Thrift, Avro, Protocol Buffers, etc. Specialized record deserializers can skip over unwanted fields in these row oriented storage formats Prototype implemented as a Scala macro that creates a custom deserializer at compile time Don't deserialize data that you don't need
  • 53. COLUMN PROJECTION FOR ROW ORIENTED DATA Don't deserialize data that you don't need 198 111 121 054 e l m _ s t r ... a l i c e ... Decode User Id to Long Skip over unwanted address field Decode Name to String
  • 54. COLUMN PROJECTION FOR ROW ORIENTED DATA No IO savings But only decodes the fields you care about into objects CPU time spent decoding Strings can be huge compared to time it takes to load + ignore the encoded bytes Don't deserialize data that you don't need
  • 55. TotalComputeTime Number of Columns Selected 1 7 10 13 48 Parquet Thrift Parquet Pig Lzo Thrift + Projection COLUMN PROJECTION: THRIFT VS. PARQUET Parquet Thrift has a lot of room for improvement Parquet faster than row oriented until 13 columns This schema is relatively flat, and most columns populated
  • 56. APACHE PARQUET Columnar storage for the people Predicate push-down also allows parquet to skip over records that don't match your filter criteria Parquet stores statistics about chunks of records, so in some cases entire chunks of data can be skipped after examining these statistics
  • 57. APACHE PARQUET Columnar storage for the people Combining both column projection and predicate push down is a powerful combination
  • 58. TotalComputeTime Lzo Thrift Parquet + Filter Parquet + Filter + Project FILTER PUSH DOWN WITH PARQUET 4.3X FASTER
  • 59. APACHE PARQUET Columnar storage for the people Predicate push-down performance depends on the nature of the filter Searching for rare records is the best case, entire chunks of records are likely to not contain the records you are looking for
  • 60. Key take aways I N S U M M A R Y
  • 61. IN SUMMARY Key takeaways Profile! Serialization is expensive, and Hadoop does a lot of it Choose a storage format that fits your access patterns Use column projection Sorting is expensive -- use Raw Comparators IO may not be your bottleneck -- more IO for less CPU may be a good tradeoff
  • 62. ACKNOWLEDGEMENTS Thanks to everyone involved! Dmitriy Ryaboy @squarecog Gera Shegalov @gerashegalov Julien Le Dem @J_ Katya Gonina @katyagonina Mansur Ashraf @mansur_ashraf Oscar Boykin @posco Sriram Krishnan @krishnansriram Tianshuo Deng @tsdeng Zak Taylor @zakattacktaylor And many more!
  • 63. GET INVOLVED Contributions always welcome! github.com/twitter/scalding github.com/twitter/algebird github.com/twitter/chill github.com/apache/parquet-mr
  • 64. JOIN THE FLOCK We're Hiring! Work on data processing challenges at scale Strong commitment to open source jobs.twitter.com Data Platform: (https://about.twitter.com/careers/positions?jvi=oipMYfwb,Job)
  • 65. Q U E S T I O N S ? A L E X L E V E N S O N | I A N O ' C O N N E L L | @ T H I S W I L LW O R K @ 0 X 1 3 8