Smart Search, Smarter Care: Powering Enterprise Search with Generative AI

SMART SEARCH,
SMARTER CARE
POWERING ENTERPRISE SEARCH WITH GENERATIVE AI
Gianna Pfister-LaPin
Center for Digital Health
Mayo Clinic
IntraNET Reloaded USA
Hyatt Regency Long Beach
March 20 – 22, 2024

©2024 Mayo Foundation for Medical Education and Research | slide-2
• I am a UX Researcher
• Focused on end-user perspective (employees)
• Give credit where credit is due
THIS IS NOT A
TECHNICAL
PRESENTATION
Image source: Midjourney

Image source: Midjourney V. 6
WHAT IS
GENERATIVE
AI?

GEN AI IS AT ITS PEAK
Image source: Gartner

GENERATIVE AI IN CONTEXT
GENERATIVE AI
Class of artificial intelligence that generates new,
original content by learning from vast datasets.
FOUNDATION MODELS
Large models trained on diverse datasets, fine-tuned
for specific applications
LARGE-LANGUAGE MODELS
Systems trained on extensive text data to generate
human-like language for linguistic tasks
GPTs
Models that generate human-like text to perform
text completion, translation, and other content
creation
CHATGPT
OpenAI product optimized for generating
conversational responses
ChatGPT
Generative AI
Foundation Models
Large-Language Models
GPTs
The Tree of
AI

IS “SEARCH” BROKEN?
• Commercial search platforms appear to be
returning increasingly outdated/irrelevant results
with too much priority being put on advertising,
spam, & SEO-manipulated content (Paul,
2024). Current research supports this
(Bevendorff, et. al. 2024).
• The overall experience of searching the public
internet has degraded.
• Users have developed hacks to get better
search results, e,g, adding “reddit” to a query
(Brerton, 2022)
• Trust in Google search results is diminishing
(u/a_latvian_potato, 2022)

Image source: Dion Hinchcliffe

(Prompt: hyperrealistic photograph of a digital network searching an intranet for data --no robots --ar 3:4)
Cure, Connect, Transform
• AI is key to achieving strategic
goals, faster discoveries, working
smarter & not harder
• Benefits for employees include
streamlined decision-making,
reduced manual tasks, and
improved access to information
AI & MAYO CLINIC
2030 STRATEGY

1
CARE GUIDANCE
CASE STUDY

ASK MAYO EXPERT
MAYO’S CLINICAL CARE “BRAIN”
AskMayoExpert (AME) is a
decision-making resource authored
by Mayo’s specialists for internal use
& licensed to members of the Mayo
Clinic Care Network.
Primarily used by GPs to decide
when & where to refer a potential
patient to Mayo Clinic.
Contains the latest vetted insights on
disease management, care
guidelines, treatment
recommendations & reference
materials.

AME
PRODUCT
TEAM
Dr. Yunguo Yu, PhD, MD
Care Guidance Data Scientist
Matt Gardner
Sr. Digital Product Manager
Jill Meyerson
Experience Design Researcher
Daveena Tauber, PhD
Principal Experience Design
Researcher

• Develop working LLM trained with AME data.
• Assess initial reactions to working prototype
and gather high-level feedback.
PROOF-OF-CONCEPT
SEPT-NOV 2023
QUERY GENERATION
JULY-AUG 2023
• Understand current user search behavior and
clinical scenarios behind AME searches.
• Learn how clinicians would format a search
query & what their expectations would be for a
potential genIA tool.
Image source: Midjourney V. 6 ©2024 Mayo Foundation for Medical Education and Research | slide-14

RESEARCH
METHODOLOGY
Participants Recruited
• Physicians who were med/heavy current users of AME (General
Internal Medicine, Neurology, Cardiology, Family Medicine)
• Advanced Practice RNs (General Internal/Family Medicine)
• Subject matter experts (Cardiology)
Study Format
• 30-minute semi-structured interviews (virtual)
• Participants were asked about current AME use, shown sample
question/answer sets for feedback, or asked to reformat
previous AME queries for new genAI tool and react to answers
generated
• Discussions explored relevance, usefulness, confidence,
comparison of existing search vs. POC, feedback on how to
improve results

EXAMPLE LLM OUTPUT

• Trust Requires Transparency: Clinicians want accurate, complete answers that come from a
visible and vetted source.
• Contextual Relevance: Clinicians want the ability to input PHI & other data to obtain answers
that are more precise and actionable at the point of care.
• Expert-Led Product Design: Interfaces, functionality, & testing scenarios must mirror real-world
situations to avoid friction when conducting research.
• One Of Many Tools: LLMs and other models are seen as a valuable potential tool in the patient
care toolbox. They add, but do not replace, other methods.
• Trust Must Be Transferred: GenAI’s impact on current workflows is still unknown. Trust has to
be transferred from existing tools to AI-enabled tools for them to be adopted successfully. Trust is
easily eroded by seemingly minor infractions or confusing output.
FINDINGS & OUTCOMES
OF USER RESEARCH EFFORTS

ROAD MAP
NEXT STEPS
• Increase the realism of the models by soliciting same queries from more user groups
• Add in additional content from CPMs, calculators, and patient education materials; explore and
test at every step
• Enable direct interaction with the proof-of-concept LLM to learn more about how users interact
with the model over time (diary study)
NEAR FUTRE
• Try different answer output formats to enhance usefulness at point-of-care
• Faster models and pilots run in the clinic can show if this solution really solves current problems
or improves the AME experience
• Continue to incorporate user feedback as models evolve
• How do we train busy people on using this tool most effectively?

2
INTRANET
SEARCH CASE
STUDY

ENTERPRISE SEARCH
UNIFYING THE DIGITAL WORKPLACE
• Mayo Clinic’s Intranet features
organizational news,
clinical/business tools, knowledge
bases and ticketing management
systems for IT, HR, etc.
• Includes department-authored
content & Office365 ecosystem,
representing a massive amount of
institutional knowledge.
• Search has been a problem for a
long time & has resisted many
different attempts to fix it over the
years

SEARCH
PRODUCT TEAM
Lisa Semidey
Sr. Experience
Design Researcher
Caroline Little
Sr. Experience
Design Researcher
Craig Hobson
Assoc. Digital
Product Designer
Brad Herr
Principal Digital
Product Manager
Katie Mau
Principal Product
Owner

• Further explore employee expectations for
genAI-powered search
• Gather reactions to updated concepts based
on previous feedback
DISCOVERY &
UPDATED CONCEPT, TEST 2
MARCH 2024
DISCOVERY & CONCEPT, TEST 1
JAN 2024
• Understand current search experience for
employees on the intranet
• Explore prior experience/sentiments regarding AI
• Gather reactions to a preliminary genAI search
concept
Image source: Midjourney V. 6 ©2024 Mayo Foundation for Medical Education and Research | slide-22

RESEARCH
METHODOLOGY
Participants Recruited
• Mayo Clinic employees across a range of roles (clinical &
business), locations, and working arrangements
• 4 groups depending on prior AI experience/comfort (Zero Use,
Low, Med, & High Comfort)
Study Format
• 60-minute structured interviews & discussion of concepts
simulating a benefits open enrollment scenario
• Participants were asked about current search behavior,
knowledge & use of existing AI tools, and expectations of how
genAI-enabled intranet search would function
• Three different versions of the scenario experience were
prepared to help guide the discussion, with emphasis on
understanding and alignment rather than task completion

DESIGN CONCEPTS
ENHANCED SEARCH ONLY RESULTS WITH SUMMARY RESULTS AS CHAT

FINDINGS & OUTCOMES
OF USER RESEARCH EFFORTS
• AI Chat Expectations vs. Reality: Users were intrigued by the “chat" functionality but had
misaligned expectations, ranging from assuming human interaction (no/low experience) to assuming
it worked like ChatGPT (high experience).
• Guidance and Training: Employees need help understanding what genAI is capable of and how
they should interact with it. In-context tooltips, clear labels, and walkthroughs can guide users within
the tool, while educational content, clear organizational policies & guidelines, and a sandbox
environment can help.
• Personalization: Some employees expect search to know who they are and deliver relevant results
according to location, role, search history, etc.
• Content Strategy Still Important: While genAI-enabled search may improve findability and
integration of different collections of content, it will emphasize deeper issues like lack of copywriting
support & governance.

ROAD MAP
ALREADY STARTED
• Updates to the prototype design to integrate feedback received; workshopping improved
solutions to the issues identified.
• Conversations with stakeholders about preferred solution platform – desire to standardize
internal & external search technologies.
UP NEXT
• Conduct additional research with updated concepts to ensure previous user feedback was
properly integrated.
• Deploy working model trained on narrow range of topics for testing purposes; a more realistic
experience will improve the quality of user research findings.

BEST PRACTICES
FOR GEN AI
SEARCH

RECOMMENDATIONS
Trust & Reliability
• Establish & Maintain Trust: Position the tool as a trustworthy and reliable component of the
organizational knowledge framework by using vetted data sources for training. Transparent
management of tool limitations and consistently accurate responses will nurture continued
trust in the tool’s outputs.
• Manage Perceptions & Optics: Actively work to align the tool's presentation and capabilities
with organizational values. The tool's perceived value will significantly impact adoption.
Address perceived risks by clearly communicating measures taken to ensure data privacy and
the tool's security.
• Evaluate Performance: Set clear benchmarks for model performance, particularly in areas
requiring high accuracy, such as clinical information. Regularly evaluate the tool against these
benchmarks, involving SMEs and relevant authorities/regulatory bodies in the process.

RECOMMENDATIONS
User Experience
• Prioritize Usefulness & Relevance: Ensure the tool delivers responses that are concise,
accurate, and directly relevant to the queries. Responses should be formatted for quick
comprehension and directly link to the deeper content related to the query, avoiding
unnecessary navigation.
• Set Expectations Via Training: Ensure users are well-trained on how to effectively interact
with the tool. This includes understanding the type of queries it can handle, how to
formulate questions for optimal results, and what information falls outside the tool’s
capabilities.
• Design for Non-Answers: If the tool does not have an answer to a query or the confidence
in the answer is low, offer referrals to in-house experts or additional resources.

RECOMMENDATIONS
Implementation
Plan Your Transition: Implement a well-planned change management strategy to build
confidence, manage expectations, and ensure seamless integration into existing workflows.
Communicate Proactively: Start communicating early with stakeholders, present progress
regularly, and address objections promptly.
Iteratively Improve: Continuously monitor how users interact with the tool, the types of
questions asked, and the satisfaction with the answers received in order to refine the
accuracy and relevance of responses,

THANK YOU

SOURCES
IMAGE CREDITS
Slide 2: Image by Midjorney, v. 6. Prompt used: dramatic photograph of many employees
working together united by digital networks--ar 3:4
Slide 3: Image by Midjourney, v. 6. Prompt used: generative artificial intelligence --no
faces
Slide 4: Image by Gartner. Source: https://www.gartner.com/en/articles/what-s-new-in-
artificial-intelligence-from-the-2023-gartner-hype-cycle
Slide 6: Image by Midjourney, v. 6. Prompt used: hyperrealistic full-color photographic
rendering of a bonsai surrounded by digital networks on a white background
Slide 7: Image by Midjourney, v. 6. Prompt used: searching through digital files and
networks
Slide 8: Image by Dion Hinchcliffe. Source: https://www.constellationr.com/blog-news/how-
generative-ai-has-supercharged-future-work
Slide 9: Image by Midjourney, v. 6. Prompt used: hyperrealistic cinematographic image
showing artificial intelligence at Mayo Clinic --no robots --ar 9:16
Slide 10: Image by Midjourney, v. 6. Prompt used: hyperrealistic cinematographic image
showing artificial intelligence at Mayo Clinic --no robots --ar 9:16
Slide 13: Image by Midjourney, v. 6. Prompt used: hyperrealistic photograph of a digital
network searching an intranet for data --no robots --ar 3:4
Slide 19: Image by Midjourney, v. 6. Prompt used: dramatic photograph of business
employee using a laptop surrounded by digital networks --no text --ar 3:4 --sref
https://s.mj.run/7K1702GN8ss
Slide 22: Image by Midjourney, v. 6. Prompt used: generative artificial intelligence --no faces)
Slide 27: Image by Midjourney, v. 6. Prompt used: generative artificial intelligence --no faces,
brains, trees
Slide 30: Image by Midjourney, v. 6. Prompt used: dramatic photograph of a middle-aged
female presenter shown from the back, standing on a stage addressing an audience that is
facing the speaker in a conference hall, surrounded by faint digital networks, --ar 16:9 --sref
https://s.mj.run/V91k37mU3SI)
WORKS CITED
Paul, B. (2024, February 8). “Google search doesn’t give answers to anything anymore”:
Expert says Google is “broken” now. Here’s why. The Daily Dot.
https://www.dailydot.com/news/google-search-broken/
Brereton, D. (2022, February 15). Google Search Is Dying. Dkb.blog; DKB Blog.
https://dkb.blog/p/google-search-is-dying
Comment by u/a_latvian_potato on ”Google Search Is Dying”. (2022). Reddit.
https://www.reddit.com/r/technology/comments/st9ri1/comment/hx3zubc/
Amanda Chicago Lewis. (2023, November). Did SEO experts ruin the internet or did
Google? The Verge; The Verge. https://www.theverge.com/features/23931789/seo-search-
engine-optimization-experts-google-results
Bevendorff, J., Wiegmann, M., Potthast, M., & Stein, B. (2024). Is Google Getting Worse? A
Longitudinal Investigation of SEO Spam in Search Engines.
https://downloads.webis.de/publications/papers/bevendorff_2024a.pdf

Smart Search, Smarter Care: Powering Enterprise Search with Generative AI

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Smart Search, Smarter Care: Powering Enterprise Search with Generative AI

Semelhante a Smart Search, Smarter Care: Powering Enterprise Search with Generative AI (20)

Último

Último (20)

Smart Search, Smarter Care: Powering Enterprise Search with Generative AI

Notas do Editor