Human vs AI Quality Raters for Search Engines.pdf
âą
1 gostouâą813 visualizaçÔes
Human quality raters have been the mainstay of search engine evaluation for decades but a sea-change is on its way due to the need for scale as machine learning and demand evolves.
Recomendados
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Human vs AI Quality Raters for Search Engines.pdf
Semelhante a Human vs AI Quality Raters for Search Engines.pdf (20)
Mais de Dawn Anderson MSc DigM
Mais de Dawn Anderson MSc DigM (20)
Ăltimo
Ăltimo (20)
Human vs AI Quality Raters for Search Engines.pdf
- 1. Humans vs LLMs As Quality Raters for Search Engines Are major changes coming? Dawn Anderson - March 2024
- 2. Dawn Anderson â UK based SEO consultant â 17 years in SEO â Occasional SEO conference speaker â EU, UK, US, Global Search Awards judge â Previous digital marketing lecturer & trainer â Industry publication contributor â Now predominantly consulting all of the time Stalker of information retrieval threads and IR conference hashtags since 2017
- 3. A sea-change is coming for a fundamental part of search On the other side of the âfront doorâ
- 4. The important algorithmic ranking evaluation stage If importance thresholds reached Indexing Discovery & refresh Crawling Dynamic build at runtime Serving In response to a query Ranking (& Re-ranking)
- 5. The process of search results evaluation (Ranking System) Determine how well a âsystemâ (ranking system) fares either currently (continuous evaluation), or when compared to proposed changes
- 7. Whilst some change is temporally predictable Seasonality, temporality, data-driven probability of âprecision-to-relevanceâ intent shift
- 8. Often, what ârelevanceâ means is changing unpredictably Search CANNOT be static in a changing world
- 9. The process of evaluation⊠â Is both continuous - On existing âsystemsâ â And intermittent - On proposed âsystem improvementsâ
- 10. A âsystemâ is simply a recipe or multiple recipes SYSTEM == ALGORITHMIC BLEND
- 11. âSystemâ evaluation⊠is simply taste-testing Did the recipe turn out as we hoped???
- 14. âHuman in the loop (HITL)â is the mainstay of âsystemâ evaluation
- 15. Predominantly two types of âHITLâ evaluation â Implicit â Explicit
- 16. Implicit (âHumanâ in the loop has no awareness) evaluation â Tests on real searcher segments â Anonymous scroll and click behaviour â UX testing on any site (heatmaps / recordings all fall into this category)
- 17. Explicit (human knows they are actively evaluating) evaluation â E.g. Searchers asked to provide feedback â NetïŹix users asked to thumbs up a ïŹlm â Spotify favouriting or playlist building - leads to further recommendations â User groups / user panels â Sites asking for feedback â Professional expert relevance annotators â Paid human contractor evaluators
- 18. But it all mostly comes down to labels & labelling anyways IMPORTANT⊠Labels are training data for machine learning
- 19. Labels are all around us In vast numbers they are converted into mathematical form for machine learning training data
- 20. We are ALL data labellers⊠every single day Every day
- 21. A cohort of similar data labellers help with recommender systems Birds of a feather ïŹock together⊠they like the same things
- 22. Data labels teach machines to know the diïŹerence between cats and dogs (reinforcement learning) Cat, dog, dog, cat, cat, dog, cat, dog, dog, dog, cat
- 23. Search engines have used âThe Crowdâ for HITL evaluation for more than two decades
- 24. In search⊠âThe Crowdâ âlabelsâ sample comparative search result sets âRelevantâ or ânot relevantâ
- 25. Pair-wise SERP results side-by-side comparisons are in the majority of relevance evaluation exercises PAIR-WISE COMPARISON OR
- 26. Instead of âyumâ or âbarfâ labels, itâs ârelevantâ or ânot-relevantâ labels
- 27. But itâs mostly aggregated binary data Binary labels rolled up into overall relevance scores
- 28. EïŹectively a measurement of NDCG (Normalised Discounted Cumulative Gain) and / or DCG (Discounted Cumulative Gain)
- 30. The ârecipeâ ingredients or quantities are then adjusted accordingly And the cycle begins again
- 31. Until⊠Acceptable Precision at K is achieved (P@k) E.g. The top 10 (k) or 20 (k) (whatever) in enough samples are deemed relevant
- 33. And⊠roll it out
- 34. But⊠Not all labels are created equally â Gold labels - High quality / lower availability â Silver labels - Lower quality / higher availability
- 35. Real search engine users in experiments Create âgold labelsâ
- 36. Before âThe Crowdâ⊠professional subject matter expert annotators Created gold labels too
- 37. But⊠ScaleâŠ
- 38. âThe Crowdâ came for the scale Professional expert annotators were not scalable
- 39. Universal Human Relevance System
- 40. 16,000+ Google human quality raters alone
- 42. But to the detriment of quality âSuch annotation tasks were delegated to crowd workers, with a substantial decrease in terms of quality of the annotation, compensated by a huge increase in annotated data.â (Clarke et al, 2022)
- 43. âThe Crowdâ likely produce silver labels
- 44. There are issues with the data labelling industry overall
- 45. Data labelling industry crisis⊠Demand outstrips supply â There is a bottleneck (and itâs going to get worse) â Not enough labels produced to deal with the size of machine learning modes
- 46. âThe global data collection and labeling market size was valued at $2.22 billion in 2022 and it is expected to expand at a compound annual growth rate of 28.9% from 2023 to 2030, with the market then expected to be worth $13.7 billion.â Source: Grand View Research, 2021
- 47. Data labellers work across many industries, many companies â Maps â Assistant â AI content detection â Search quality evaluation â Image detection labelling â AI content detection training â Any other ML driven application
- 48. High risk of under-trained ML models due to scaling without label volume increase
- 49. Deepmind researchers - âWe ïŹnd current large language models are signiïŹcantly undertrained, a consequence of the recent focus on scaling language models while keeping the amount of training data constant. âŠwe ïŹnd for compute-optimal training âŠfor every doubling of model size the number of training tokens should also be doubled.â â âTraining Compute-Optimal Large Language Modelsâ (Hoffman et al, 2022)
- 50. Thereâs a dark side to the data labelling industry too
- 51. Low paid âghost workersâ in emerging economies
- 53. Thereâs also a problem with humans as relevance labellers too We ainât all that⊠it seems
- 54. In addition to the very subjective 170 page âQuality Rater Guidelinesâ
- 56. âThe crowd is made of people - Observations from large scale crowd labellingâ (Thomas et al, 2022)
- 57. Bing Researchers - âThe Crowd is Made of People: Observations from large scale crowd-labellingâ (Thomas et al, 2022) Findings: â Fatigue â Time of day & day of week â Anchoring â Task-switching â Left-side bias â General disagreement on relevance
- 58. Human challenges Ethical concerns Undertrained models Bottlenecks / over-demand More scale needed A perfect storm
- 60. âLarge language models can accurately predict searcher preferencesâ (Thomas et al, 2023) Bingâs LLM & GPT4 research
- 61. â GPT4 prompt engineering (role playing prompt) â (Up to 5) LLM agents to emulate the behaviour of search relevance evaluators â Produce enough gold and silver labels to build relevance training data for much larger data sets â Train the agents initially on gold labels âLarge language models can accurately predict searcher preferencesâ (Thomas et al, 2023)
- 62. Bingâs research and implementation Has caused quite a stir in the information retrieval community
- 63. âTo measure agreement with real searchers needs high-quality âgoldâ labels, but with these we ïŹnd that models produce better labels than third-party workers, for a fraction of the cost, and these labels let us train notably better rankers.â (Thomas et al, 2023)
- 64. Bingâs LLM evaluators - âA fraction of the cost and better rankersâ (Thomas et al, 2023)
- 65. Bingâs LLM evaluators are monitored Using several methods
- 66. A spectrum of LLM & human rater collaborative approach ? âFrontiers of Information Access Experimentation for Research and Educationâ (Clarke et al, 2022)
- 68. âIt is yet to be understood what the risks associated with such technology are: it is likely that in the next few years, we will assist in a substantial increase in the usage of LLMs to replace human annotators.â (Clarke et al, 2022)
- 69. ButâŠConcerns about reduced quality in exchange for scale âIt is a concern that machine-annotated assessments might degrade the quality, while dramatically increasing the number of annotations available.â Clarke et al, 2022
- 70. Will Google follow suit?
- 71. Bing test and release updates seamlessly and quickly Potential for Google to go this way with better evaluation pipelines
- 72. Google cancels their contract with Appen
- 73. Some surmised a switch to AI evaluators was part of the reason
- 74. Algorithms - bigger, broader, multi-modal / multi-aspected Aspected algorithms quickly go into core or simultaneous â Product reviews â Helpful content classiïŹer â Panda historically â Spam updates
- 75. Machine learning classiïŹers Google is learning quickly: â what âunhelpful content looks likeâ â What AI generated content looks like â What paid links look like
- 76. AI content detection is where itâs at next For the data labelling industry
- 78. Thank you Twitter - @dawnieando Website: https://bertey.com LinkedIn - https://www.linkedin.com/in/msdawnanderson/