Big Data: Architectures and Approaches
•Transferir como PPTX, PDF•
37 gostaram•13,171 visualizações
ThoughtWorkers David Elliman and Ashok Subramanian present how the big data world is moving quickly with predictions of amazing industry growth. For more information on how the 'Internet of Things' is playing an increasingly larger role, read David's blog post or watch the video from the London-based event. http://www.thoughtworks.com/insights/blog/big-data-and-internet-things
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (14)
Destaque
Destaque (20)
Semelhante a Big Data: Architectures and Approaches
Semelhante a Big Data: Architectures and Approaches (20)
Mais de Thoughtworks
Mais de Thoughtworks (20)
Último
Último (20)
Big Data: Architectures and Approaches
- 1. w e l c o m e BIG DATA Architectures and Approaches David Elliman & Ashok Subramanian
- 6. 1944 https://www.flickr.com/photos/timetrax/376152628/sizes/l 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 7. 1961 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 8. 1971 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 9. 1996 https://www.flickr.com/photos/epsos/8336691931 ge becomes more cost effective for storing da 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 10. 1996 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 11. 1998 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 12. 1998 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 https://www.usenix.org/conference/1999-usenix-annual-technical-conference/big-data-and-next-wave-infrastress-problems
- 13. 2004 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 14. 2006 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 15. 2008 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 16. 2010 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 17. 2013 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 "alottabytes"
- 18. 2015 https://www.flickr.com/photos/will-lion/2595830716/ 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
- 24. THE OPPORTUNITY
- 25. <- 1990 DATA INSIGHT DATA INSIGHT DATA INSIGHT 1990s - 2000 2000 ->
- 26. Key Takeaways • This isn’t a new problem • The problem isn’t going away • Remember to focus on the VALUE https://www.flickr.com/photos/djwtwo/8331524425/
- 29. Complexity Value Descriptive Analytics Diagnostic Analytics Predictive Analytics Prescriptive Analytics What happened? Why did it happen? What will happen? How can we make it happen? Analytics - Goals
- 33. THINK BIG S M A L L A C T S M A L L A C T Small is the New Big (Seth Godin)
- 37. “80% of the work in any data project is in cleaning the data” – D J Patil https://www.flickr.com/photos/desideratum/8595251348/
- 42. SQL
- 46. Key Takeaways • Start small • Start with the ? • Iteratively follow the value • Using freely available tooling • Volume vs Velocity https://www.flickr.com/photos/djwtwo/8331524425/
- 50. –attributed to Gene Amdahl 1967 “Amdahl’s law is used to find the maximum expected improvement to an overall system when only part of the system is improved.”
- 53. Batch Speed Serving Query query = function(all data) All Data Lambda Architecture
- 54. Scaled Data Store Event Processing Network QueryAll Data Lambda Architecture Batch View Realtime View Batch Write Random Write
- 55. Batch Speed Serving Query query = function(all data) All Data Lambda Architecture
- 56. Client Master Node JobTracker Name Node Metadata Operations to Get Block Info Job assignment to cluster Task Tracker Slave Node Data Node Map Reduce Task Tracker Slave Node Data Node Map Reduce Task Tracker Slave Node Data Node Map Reduce Task Tracker Slave Node Data Node Map Reduce 1 3 1 2 1 5 6 4 Data Replication on Multiple Nodes DataWrite DataRead Batch - Hadoop (MR1)
- 57. Batch - MapReduce Map Shuffle Reduce
- 60. Batch - Spark
- 61. Segment Servers Query processing and data storage Network Interconnect Master Servers Query planning & dispatch External Sources Loading, streaming, etc. SQL or MapReduceBatch - MPP database
- 63. Batch Speed Serving Query query = function(all data) All Data Lambda Architecture
- 64. Speed - Storm
- 65. CEP
- 66. Batch Speed Serving Query query = function(all data) All Data Lambda Architecture
- 67. Lambda Architecture - Serving
- 69. Pull-based Batch Loads Enterprise Data Models Complex ETL Logic Poorly Suited to Non-Relational Data Emergent design is difficult Conventional Architectures
- 70. Pivotal Business Data Lake Architecture http://www.gopivotal.com/sites/default/files/Pivotal-Business-Data-Lake-Technical_Brochure_WEB.PDF
- 71. DATA CORE RAW FACTUAL DATA HISTORIZED EVENTS RETAIN BUSINESS KEY DATA LINEAGE
- 72. DATA INGESTION EVENT DRIVEN MESSAGE QUEUE TRICKLE FEED BATCH LOAD
- 73. INFORMATION PUBLISHING TOPICAL QUEUES POST PROCESSING
- 74. INFORMATION TIER PURPOSE BUILT DATA SUBSETS TRANSFORMATION DATA GOVERNANCE MDM CONCERNS POST PROCESSING
- 75. PRESENTATION TIER BUSINESS VALUE APPLICATIONS DATA SERVICES AD HOC QUERYING WRITE BACK?
- 77. Transformation Logic Data Post Processing Near Real Time Feed Emergent Design & Agile Delivery
- 81. Drive Towards In Memory Processing
- 86. Key Takeaways • Embrace the cloud • Fit the Architecture to the problem • Remember Knuth https://www.flickr.com/photos/djwtwo/8331524425/
- 91. http://www.datameer.com/blog/uncategorized/the-hadoop-ecosystem-visualized-in-datameer.html 48 30 26 22 18 18 16 15 15 15 13 13 13 13 12 0 13 25 38 50 63 Hadoop Ecosystem
- 104. No matter how much you speed up the computers or the way you put computers together, the real issues are at the DATA LEVEL
- 107. Localised Formats
- 109. SoR is a process not a place
- 110. Database Integration (by another name)
Notas do Editor
- Dave http://www.forbes.com/sites/gilpress/2013/05/09/a-very-short-history-of-big-data/Reference
- Ashok Big data analytics are driving rapid growth for public cloud computing vendors with revenues for the top 50 public cloud providers shooting up 47% in the fourth quarter last year to $6.2 billion
- Dave http://nsa.gov1.info/utah-data-center/
- Ashok Who is that handsome man!
- Dave & Ashok Growth in retail, usage of iBeacons, Precision marketing, some sophistication with web analytics & CRM - greater penetration. Healthcare - remote monitoring, automated procedures
- Ashok
- Ashok Validation or Discovery picture of fork in the road?
- Ashok & Dave
- Dave
- Ashok Exploring alternate models
- Dave
- Ashok Lambda Architecture - section heading
- Ashok - high level description of components
- Dave Batch Hadoop 2.0/MR2 goal: allows you to share a large cluster of machines between different frameworks. Similar to Mesos, both are steps towards distributed data OS.
- Dave Data Lakes
- Dave
- Ashok
- Ashok
- Ashok Fast and Scalable Analytics depends on efficient data structures Matching the Algorithm to the data structure Morphing the Raw data into the data structure Raw data > Data Structure > Algorithm > Insight
- Conclusion
- Ashok Balance shifting from Commercial to Open-Source Innovations coming from the open source world
- Ashok Quantum computing - this is one apparently!
- closing statement before Q&A
- Dave