Scalable Machine Learning BerkeleyX: CS190.1x Scalable Machine Learning overview

1. Scalable Machine Learning
BerkeleyX: CS190.1x Scalable Machine Learning

2. outline
1. what is machine learning ?
2. linear regression
3. how to scale linear regression
4. gradient descent
5. how to scale gradient descent

3. Challenge: Scalability

4. What is Machine
Learning?

5. Machine learning explores the study and construction of algorithms
that can learn from and make predictions on data
A Definition
Face recognition
Link prediction
document classification

6. Raw data Raw data comes from many sources
apache log
email
image

7. Raw data
Feature Extraction
Initial observations can be in arbitrary format
We extract features to represent observations
We can incorporate domain knowledge
We typically want numeric features
Success of entire pipeline often depends on choosing
good descriptions of observations!!

8. Raw data
Feature Extraction
Supervised Learning
Train a supervised model using labeled data,
e.g., Classification or Regression model
weather
sunny
rainy
cloudy
vehicle count
1000
99
5

9. Raw data
Feature Extraction
Supervised Learning
Evaluation
TP FP
FN TN
Has cancer
No cancer
Predict cancer
Predict No cancer
confusion matrix

10. Raw data
Feature Extraction
Learning Model
Evaluation
Predict

11. full
dataset
training
set
validation
set
testing
set
classifier
turning model
model builder
final classifier predict

12. Linear regression

13. Example: Predicting shoe size from height, gender, and weight
For each observation we have a feature vector, x, and label, y
We assume a linear mapping between features and label:

14. Goal: find the line of best fit
x coordinate: features
y coordinate: labels
1 D Example

15. Linear Least Squares Regression
Assume we have n training points, where x(i) denotes the ith point
Recall two earlier points:
• Linear assumption : y = wTx
• squared loss : ( y - y )2

16. Linear Least Squares Regression

17. Computing Closed Form Solution
Computational bottlenecks:
• Matrix multiply of XT X : O(nd2) operations
• Matrix inverse: O(d3)operations
d
nX:
n
dXT:
O(nd2)O(dn2) ?

18. Matrix Multiplication via Inner Products
Matrix Multiplication via Outer Products
result

21. if Big n and Big d ?

23. Gradient Descent
=

24. Gradient Descent
Start at a random point

25. Gradient Descent
Start at a random point
1. Determine a descent direction
2. Choose a step size
3. Update

26. Gradient Descent
Start at a random point
1. Determine a descent direction
2. Choose a step size
3. Update
Repeat
Until stopping criterion is satisfied

27. Choosing Descent Direction (1D)
1. Determine a descent direction

28. Choosing Step Size
2. Choose a step size

29. =

30. Parallel Gradient Descent for
Least Squares

31. Gradient Descent Summary
• Easily parallelized
• Cheap at each iteration
positive
• Slow convergence
• Requires communication across nodes!
negative