The more you know about the mental state of a learner, player, or trainee, the better you can tailor an experience to their current state and progression over time. There are multiple non-invasive techniques for assessing trust, engagement, fatigue, emotions, and learning progress using biophysical sensors. After you collect and understand this data, many possibilities exist for augmenting their experience. Tracking learning progress with fNIR brain imaging and fatigue by measuring pupil dilation then subsequently scaling difficulty is one example. Automating certain user interface interactions through gaze detection is another. Virtual characters that react to eye contact (or its absence) are also a possibility. In the talk, the latest research on the efficacy of these techniques will be provided, along with potential use cases in games, training, and learning.

1. Games & Training + Human
Performance Data
Spencer Frazier
Lockheed Martin

2. •Spencer Frazier
•Lockheed Martin - Lead Senior Software Engineer
•Boston College, USC M.S., Georgia Tech Ph.D (hiatus)
•Startups: Moglo (Aniphon), Drizly Inc
•Research: Serious Games (Team-It, Mars Game), Multi-Agent
Systems, NNs (RNN,LSTM,etc), Compositional Frameworks for
Zero-Shot Learning
•Hobbies: Mask carving, game community (almost 15 years), jamming
sensors into everyday life

3. • Moglo – location-based capture the flag using mobile
• ZombieSC – Location-based fitness game, closed loop incentives based
on learning
• Aniphon – location-based creature battling (think Pokemon GO)
• Team-It - @USC, MAS research, evaluating teaming and collaboration
with agents
• Mars Game – Instructional game web game for STEM concepts
• Agent Trust/Real-Time Augmentation – Eyetracking, alerting, HR, GSR,
CNAP (BP), Wizard of Oz performer
• 1O -> ML -> 2O – Take 1st
Order Human Performance data and learn
second order assessment-relevant data to lighten workload on
instructors (ML, RNNs, LTSMs)

5. “Why shouldn’t I take a nap or text or answer
emails?” aka “The Agenda”
1. Hardware and assessment-based approaches to collecting human
performance data
2. What interesting things you can infer from those data
3. How you can modify your games and training based on that data to
save/make time/money
4. Examples (real work, ongoing work and some new ideas)

6. Plenary+ (More reasons to stay [awake])
•Build trust w/ avatars
• Should we use an avatar?
•Improve scalability in virtual environments
• Save resources
• Improve scalability in VR, specifically
•More immersive environments
•Capitalize on eSports growth
•More

7. • Human Factors: interactions
among humans and other elements
of a system
• Human Performance: first order
(heart rate) and second order
(stress) data from a human
• Sensors/Hardware: how we get
the data
• Biophysical: a dirty(?) word that I
still like to use
• Agent: person or automated
process
• Others: speak up!
Researchgate.net

8. Hardware – advancing and already “on”
Pupil Labs
Gregory Kovacs (Stanford)
Tobii

9. Hardware Improvements (Last 5 Years)
• Beyond Medical
• Though this is still the best for
reliable data
• Consumer Awareness/Ownership
• Durable(ish)
• Affordable ($200-$2000 for
multi-modal monitoring)
• Eyetracking is still pricy
• Less bulk/onboarding
• Gender affordances(ish)

10. Types of First-Order data
• Eyetracking
• Gaze
• Pupillometry
• Heartrate/Pulse/BP
• Perspiration
• GSR
• Respiration
• EEG
• Brain Signals
• fNIR
• Functional Near-IR (O2, Hem in
prefrontal cortex)
• Force/Haptics
• Force sensitive mouse
• ForceTouch
• Movement (aXYZ)
• Camera Images
• Video
• IR Depth
• ...and more!
USC ICT
Krysten Newby (CreativePool)

11. Types of Second-Order Data
• Gestures/Interaction/Intent
• Expressions (EKMAN)
• Stress (TALEMAN 2009)
• Engagement (SHAGASS 1976)
• Workload (SASSAROLI 2008)
• Sentiment (RAUDONIS 2009)
• Learning Rate (MAKEIG, AYAZ,
GRAMANN)
• Trust/Deceit (GRATCH 2012)
• Fatigue (SEOANE 2014)
• Posture [Engagement] (D’MELLO)
• Fitness [Avg HR]

12. Emotions/Trust/Deceit
•Face and eyes
•Pupil size and reactivity
•Observations
• Illness/recovery potential (medical)
• Attraction
• Cognitive load (larger diameter through task completion)
• Memory encoding
•Avatars can react or animate appropriately

13. Engagement
•Gaze, pupils
• Memory encoding based on pupil size
•Posture (slouching)
•Don’t lose early chunks of the “train, practice, assess” cycle in game
onboarding or in the classroom (learners left behind) (Jacobson)

14. Stress/Cognitive Load
•Heartrate
•Bloodpressure
•GSR
•fNIR
•Game or training difficulty can respond dynamically (or just report)
•Your own assessment/derived measure

15. Data Best Practices
•Timing/time is huge
• You’ll hear different things from your HFR, your architect, your engineer, your
data scientist, your customer
•Data format for analysis after the fact is huge
• xAPI?
• NoSQL/Relational?
• Binary?
• How much to save?
•We need meta-data standards for data collection to ease adoption
and interaction with this equipment
• Nobody wins long term with closed-source, closed-API software*

16. Game Use Case – Gaze, Pupillometery, PostureCharacter authoring, believability, immersion

17. Advantages (Eyetracking/Posture via Depth)
•Less time spent authoring believable characters
• Reactionary behavior can be shared across all agents in the game
• Saves $
•Immersion increased as every player action feels like they are
influencing the environment
• Flow, presence (Weibel 2011)
•Mirror the character’s posture to establish rapport or just assess
engagement

18. Training Use Case– Gaze, PupillometeryTrust, engagement
Insert image about alerting/avatars/dash
20 inch monitor
Approx
accuracy

19. Research/Methods/Results
•Avatar “Oz” controlled by a SME in the domain in another room,
semi-scripted
•Other actual domain experts as subjects
• 3.5 days of 8+ hour days performing a task
•Control: non-emotional avatar, non-augmented team
•Assessing trust, speed, accuracy, engagement…
•Augmenting by dismissing alerts generated either by Oz or by the
system when the subject looked at the alert
•Survey (Trust)
•Positive feedback from subjects, system improvements

20. Advantages (Eyetracking)
•Eyetracking for alert dismissal or acknowledgement
• More time on task
•Learn what visual features are important to a trainee during critical
decision making processes
• Improve fidelity of those features/entities
• Decrease fidelity of less important features
•Assess usefulness of an avatar – how often are they engaging it?

21. Game Use Case – Adaptive UI/RenderingInterface design, responsive UI, dynamic rendering, VR

22. Advantages (Eyetracking)
• More realistic engagement with environment
• I don’t always turn my head to look at things in real life
• Adaptive sight-based LOD (also called foveated rendering) saves time
optimizing game for specific hardware, squeezes more out of an engine
• Saves $
• Increase scalability of virtual environments/VR
• VR Nausea (Xiao, SparseLightVR)
• Player retention, $
• Assessment of your end product
• Used in supermarkets to assess product placement – assess UI speed/accuracy with
eyetracking

23. Training Use Case – Alerting/Augmentation
(Vitals, Fatigue, HR, BP, etc.)

24. Advantages (Multi-Modal Monitoring)
•Stress/fatigue system response saves lives
•Build trust with a system performing life-critical tasks during training
•Perform report generation faster
• Increase instructor’s response to trainee’s physical state
• Enable the next generation of learning (ML) approaches to assessment,
classification and decision support by collecting this data now.

25. Game Use Case – fNIR, Learning Rate, Stress
Adaptive difficulty, player assessment

26. SecondSpectrum

27. Mars Game
• Web-based
• Blockly
• Pre-calculus, Trig, Programming
• 3-4 Hours of content
• Open Source
• 9th and 10th grade math and programming concepts, and aligns to the
Common Core State Standards for Mathematics.
• Control: Typical written examinations
• Result: More engaged, learned at or slightly above their peers rate
• Did not use fNIR but would be very interested to test in this domain

28. Advantages
•Player washout/adeptness at task
• eSports roster selection, faster identification of talent, $
•Richer live analysis of players – more for announcers to comment on
• Did you see that champion pop out of the bush or not?
• More engaged spectators, $
•Is this task hard enough?
• Adaptive difficulty engages and retains players, $

29. Brief Plug for Machine Learning (“Fast
Statistics”)
•Second, Third-Order insights we don’t even know about or are hard
to generate (Boyd)
• Washout
•Collect relevant contextual information and train the trainer (Lamb)

30. Not just questions…
(Discussion,Concerns, Ideas, Requests, Sentiment Analysis)