Businesses want to execute an analytical job at scale in Hadoop, but different parts of that job are potential candidates for specific execution engines so that the job performs under optimal conditions. In addition, past engines such as classic Map Reduce are potentially giving way to new ones such as Spark. This talk will demonstrate how you can leverage the Datameer application to hide the complexity of choosing the right execution engine for an analytical job at scale in Hadoop, and how Spark fits into this context.

2. © 2014 Datameer, Inc. All rights reserved.
Datameer’s Vision!
Make big data analytics
simple for everyone

3. © 2014 Datameer, Inc. All rights reserved.
What Datameer Offers!
Wizard-led Data Integration!
• No ETL!
• 59 Connectors + plug-in API!
• Smart Sampling!
Point-and-click Analytics!
• Interactive spreadsheet UI!
• 270 pre-built analytic functions!
• Macros & function plug-in API!
Drag-and-Drop Visualization!
• Blank canvas for design !
• HTML5, consumable on any device!
• Visualization plug-in API!

4. © 2014 Datameer, Inc. All rights reserved.
Smart Analytics!
Column Dependencies
Decision Tree
Recommendation Engine
Clustering

5. © 2014 Datameer, Inc. All rights reserved.
Where does Datameer sit?!

6. © 2014 Datameer, Inc. All rights reserved.
Classic Business Analytics Data Flow !

7. © 2014 Datameer, Inc. All rights reserved.
New Business Analytics Data Flow!

8. © 2014 Datameer, Inc. All rights reserved.
Datameer On Premise Installation!

9. © 2014 Datameer, Inc. All rights reserved.
Datameer Implementation - Cloud!

10. © 2014 Datameer, Inc. All rights reserved.
! Stefan Groschupf
CEO, Co-Founder!
Problem

11. © 2014 Datameer, Inc. All rights reserved.
Typical Data Analytics Funnel
Raw Data (TB-PB)
Insights (KB)
! More sophisticated
! Less change
! High value
! Power users
! Planned / scheduled
! More ad hoc
! More change
! High & low value
! Casual users
! Interactive sessions
5 - 15 steps,
iterative algorithms
Explore
Summarize
Prepare
Learn
Aggregate
Present
Slice

12. © 2014 Datameer, Inc. All rights reserved.
Raw Data (TB-PB)
Insights (KB)
Map Reduce
• Inefficient for small data!
• High latency!
Current Approaches: Either - Or
Raw Data (<TB)
Insights (KB)
In-Memory
• Only small data!
• Very expensive!
• Not Hadoop!
Not
New

13. © 2014 Datameer, Inc. All rights reserved.
Small Data, Big Machine
VS

14. © 2014 Datameer, Inc. All rights reserved.
600h Spent on Jobs < 100MB!

15. © 2014 Datameer, Inc. All rights reserved.
! Stefan Groschupf
CEO, Co-Founder!
Our Solution

16. © 2014 Datameer, Inc. All rights reserved.
Smart Execution
Raw Data (TB-PB)
Insights (KB)
New
Optimized
MapReduce
In-Memory
Single
Node

17. © 2014 Datameer, Inc. All rights reserved.
Architecture
Hadoop
MapReduce
Dataflow Graph Engine
YARN
Smart Execution Engine
In-Memory
Tez
Others
Data Integration
Visualization
Spreadsheet
Other
(SQL)
Single Node

18. © 2014 Datameer, Inc. All rights reserved.
Workflow
Data Sets
System Resources
Optimized!
MapReduce!
Single Node!
In-Memory!
Future!
Technology!
Analytics

19. © 2014 Datameer, Inc. All rights reserved.
DAG Processing
vs.!

20. © 2014 Datameer, Inc. All rights reserved.
Transparent for End Users

21. @Datameer!