Recomendados
Recomendados
Mais conteúdo relacionado
Último
Último (20)
Destaque
Destaque (20)
Improving TripAdvisor Photo Selection With Deep Learning
- 1. Improving TripAdvisor Photo Selection With Deep Learning Greg Amis May 25, 2018
- 2. About Me: Applied ML, new to MV 2 Machine vision ● Photo selection Text processing ● Inappropriate reviews Metadata processing ● Review fraud Online learning ● Adaptive radar jamming Text processing ● Topic time series Agent-based modeling ● Personnel forecasting ● Brain models ● Brain-inspired architectures ● Semi-supervised learning ● Some classes on biological and machine vision Department of Cognitive & Neural Systems
- 3. About TripAdvisor 3 (1) Includes 1.1M hotels, inns, and bed & breakfasts, as well as 800K vacation rental listings (2) TripAdvisor internal log files, average monthly unique visitors during Q2 2017 + 40M photos from professionals = 150M total ● Largest travel website ● ~400 engineers ● ~40 data scientists and ML engineers
- 5. 5 Sometimes we show great photos...
- 6. 6 Sometimes… not so much
- 7. Amenity-specific shelves don’t always show the amenity 7
- 8. Photo ordering matters 8 #6 hotel in Atlantic City 4-bubble hotel in Cancun, 6.7k reviews
- 9. 1. Good primary photos 2. Relevant amenity photos 3. Good default sort order Goal: Show attractive, useful photos 9
- 10. Approach 15 interns, a mini-fridge of GPUs, great OSS 10
- 11. 11 Gather Training Data ● Interns/MTurk labeled photos ● Pairwise ranking ○ 200,000 photo pairs ○ “Which one motivates you to click?” ● Label photos containing humans ● Label photos by scene type ○ Pool, beach, room, etc. ○ Food, drink, inside, outside, etc. ● Simple infrastructure Python Pandas HTML JavaScript Python CherryPy
- 12. ● Start with 50-layer ResNet convolutional neural network trained on 1,000-class ImageNet data ● Remove upper layers concerned with classification ● Remaining lower layers make an excellent feature extractor for other machine vision problems Feature Space 12 (He et al, 2015) ... ResNet 50 ... 2,048 “bottleneck features”
- 13. For subjective scoring: Siamese Network For classification: Multi-layer feedforward networks with dropout Model architecture 13 def create_mlp(input_size=2048,output_size=1, hidden_layer_sizes=(2048, 2048), dropout_rates=(0.5, 0.5)) -> (Model, Model): model = Sequential() model.add(Dense(hidden_layer_sizes[0], activation='relu', input_shape=(input_size,))) model.add(Dropout(dropout_rates[0])) for (h, d) in zip(hidden_layer_sizes[1:], dropout_rates[1:]): model.add(Dense(h, activation='relu')) model.add(Dropout(d)) model.add(Dense(1, activation='sigmoid')) model.compile(optimizer="adadelta", loss="binary_crossentropy", metrics=["binary_crossentropy", "accuracy"]) return model Dense 2048 σ Dense 2048 Dropout 0.5 Dense 1 Bottleneck Features For Better Image Dropout 0.5 Dense 2048 Dense 2048 Dropout 0.5 Dense 1 Bottleneck Features For Worse Image Dropout 0.5 + − Maximize Siamese Network Inspired by Microsoft’s RankNet (Burges et al, 2005) and by Michael Alcorn’s Keras implementation.
- 14. ● R&D tech stack ○ Two consumer-grade GPUs ○ Keras + Tensorflow ○ Pandas ● Random hyperparameter search ○ Hidden layer width ○ Dropout rate ○ Mini-batch size ○ Epoch count ● Evaluation ○ Cross validation ○ A/B Testing (50% of users see photos selected by machine vision) Training and Evaluation 14 Kenmore - our “mini-fridge” of GPUs
- 15. Deployment 1515 Kubernetes Cluster For Computation Spark+YARN Cluster For Storage t_photo t_photo _bottlenecks t_photo _vision_models 1. Get URLs as PySpark DataFrame 2. Split into partitions 3. Feed partitions to process pool: a. Get image bytes from CDN (thread pool) b. Calc stats c. Calc bottlenecks (if necessary) d. Calc model outputs 4. Write back partitions asynchronously SELECT … FROM t_photo LEFT ANTI JOIN t_photo_vision_models ... FROM tmp_new_partition INSERT INTO t_photo_bottlenecks PARTITION(...) SELECT ... WHERE bottlenecks NOT NULL INSERT INTO t_photo_vision_models PARTITION(...) SELECT ... NVIDIA GeForce GTX 1080 Ti
- 16. Results Pretty food, fewer bathrooms 16
- 17. Better Restaurant Hero Photos 17 Initial Hero Photos Photos Selected Using Machine Vision Restaurant A Restaurant B Restaurant C
- 18. Better Hotel Hero Photos 18 Initial Hero Photos Photos Selected Using Machine Vision Hotel A Hotel B Hotel C
- 19. Hotels With A Pool 19 Original shelf, just showing hero photos Same hotels, pool photos selected using machine vision
- 20. Beachfront Hotels 20 Original shelf, just showing hero photos Same hotels, beach photos selected using machine vision
- 21. Better Sort Order 21 Original sort order Same hotel, photos sorted using machine vision
- 22. Better Sort Order 22 Original sort order Same hotel, photos sorted using machine vision
- 23. | tripadvisor is hiring! ○ Software Engineer - Machine Learning ○ Data Scientist - Attractions and Rentals ○ Data Scientist - Search Engine Marketing ○ Data Analyst - Attractions and Rentals ○ Software Engineer - Full Stack Web
- 24. ● At TripAdvisor ○ Jeff Palmucci ○ Aaron Gonzales ○ Anyi Wang ○ Tyler O’Brien ● Outside TripAdvisor ○ Google: Keras, Tensorflow ○ NVIDIA: CUDA ○ Microsoft: ResNet-50 ○ Original research at Google, Microsoft, Facebook, University of Toronto, NYU Acknowledgements 24
- 25. 25
- 26. Primary Green #00AF87 Primary Black #000A12 Accent Red #EF6945 Accent Blue #1C99CE Accent Yellow #F8C40F Secondary Dark Gray #666666 Secondary Gray #B7B7B7 Secondary Light Gray #E5E5E5 Colors 26