We examined predictors of Calculus II final grades within a sample of 84 college students enrolled in a hybrid course through WEPS. Predictors included “typical” psychological correlates, including math confidence, math anxiety, spatial skills and numerosity ability, as well as clickstream data from the students’ activity in the online course. Results showed the clickstream data were the best predictors of course performance, in that students who spent more time grading other students’ assignments, and students who took fewer quiz attempts, did better in the course. Math confidence and then math anxiety were the next best predictors, in that students with higher confidence and lower math anxiety performed better in the course. We will discuss how results might be dependent on the particular content of this course, and how we might use easy to collect psychological variables along with clickstream data to better understand, and potentially predict, course performance in online courses.

1. Colleen M. Ganley & Sara A. Hart
Psychology
FCRR, FCR-STEM
Florida State University
hart@psy.fsu.edu
@saraannhart
ganley@psy.fsu.edu
@colleenganley

2. Background

3. • Student attitudes are related to higher mathematics
achievement
• Expectations of success, comparisons of ability,
academic-self concept, confidence of own ability, etc
(Reyes & Stanic, 1988; Randhawa et al, 1993, House, 1993, House, 1995)
• Cognitive factors are also related to higher
mathematics achievement
• Numerosity, spatial abilities, memory, etc (Halberda et al.,
2008; Siegler & Opfer, 2003; Casey et al., 1995)
• But these aren’t surprising, even for predicting
success in Calculus (and Calculus II)
Understanding which
students are successful

4. • Online learning is becoming more available and
popular
• These courses provide more data related to the “user”
• Every action of the student within the course is tracked
• Can these data be used to understand success in the
course?
• Future goal of intervening with students at risk for
failure early in the course
Understanding which students are
successful in a hybrid Calc course

5. • What are the most important individual differences
predictors of success in a hybrid user-driven
Calculus II course?
• We will examine both clickstream data and
information about students’ attitudes and cognitive
performance
Research Question

6. • Spring 2014 Calculus II course at FSU
• Hybrid course with a flipped classroom
• Students used the online course platform (WEPS
https://myweps.com/moodle/) to watch videos of
the course content and solved problems in class with
professor
• All teaching content was available to students at all
times (graded items time available only)
Methods

7. • Participants
• 84 participants (43% female, 84% White)
• Took ~45min battery of demographics, student
attitudes and cognitive measures (mostly online in
qualtrics)
• Outcome variable
• Final grade (0-100) in Calculus II course
Methods

8. • Math Confidence (adapted from confidence subscale of
Fennema & Sherman, 1976)
• Generally I have felt secure about attempting
mathematics
• I am sure I could do advanced work in mathematics
• I can get good grades in mathematics
• Math has been my worst subject
Attitudinal Measures

9. • Math Anxiety (MARS-R; Plake & Parker, 1982)
• Please indicate the amount of anxiety you feel in each
of the following situations.
• Buying a math textbook.
• Looking through the pages on a math text.
• Having to use tables of formulae.
Attitudinal Measures

10. • Panamath “Dots Task” (Halberda et al., 2008)
• Approximate Number System
• Are there more yellow or blue dots?
Cognitive Measures

11. • Mental Rotation Test (Vandenberg & Kuse, 1978)
Cognitive Measures

12. • So much available information
• How to get it into something useable in more
“traditional” statistical models?
• We just want a number!!!
• Tried to use variables that we thought we had
reasonable interpretations of (but honestly still
unsure)
Online Course Measures

13. • Online workshops (graded homeworks)
• Mean time to submission across 13 workshops
• From 0-100, with 100 being submitted exactly
at time due (from when workshop was
available)
• Mean time to submission of graded workshop
assignments of other students
• From 0-100, with 100 being submitted exactly
at time due (from deadline of workshop)
Online Course Measures

14. • Online quizzes
• Unlimited attempts at quizzes (7 total)
• Sum of total number of attempts
Online Course Measures

15. Results
-
-

16. • Research question: of our key variables of interest,
what are the most useful for predicting final grade?
• Dominance analysis allows for this specific test
(Budescu, 1993; Azen & Budescu, 2003)
• All key variables were added to the model, and pitted
against each other for relative importance
• https://pantherfile.uwm.edu/azen/www/damacro.h
tml
• 1000 bootstrapped samples
Dominance Analysis (DA)

17. • Complete dominance
• (math confidence = quiz attempts = assessment time)
> (math anxiety = mental rotation = ANS = workshop
time)
• Reproducibility quite low (<10%)
• General dominance
• (assessment time > quiz attempts > math confidence >
math anxiety) > (ANS > workshop time > mental
rotation)
• Reproducibility is high across parentheses
DA results

18. • (assessment time > quiz attempts > math confidence >
math anxiety) > (ANS > workshop time > mental rotation)
DA results
0.1
0.09
0.06
0.02
0.02
0.010.005
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Final Grade
Mental Rotation
Workshop Time
ANS
Math Anxiety
Math Confidence
Quiz Attempts
Assessment Time

19. Latent Profile Analysis
• Please keep in mind the following are very
underpowered
• Intention was to have more data for full model
• Simulation studies suggest we need at least n=200 at
first, and to feel comfortable making reliable
predictions with our model likely closer to n = 500
(Nylund, Asparouhov & Muthen, 2007)

20. Mental
Rotation
ADHD_In
att
Math
Anxiety
Math
Confidenc
e
Math
Interest
Math
Motivation
Math
Importanc
e
ANS
Class 1 n = 77 0.064 0.065 -0.15 0.379 0.425 0.435 0.413 -0.213
Class 2 n = 17 -0.265 -0.32 0.542 -0.975 -1.225 -1.216 -0.945 -0.292
Class 3 n = 4 0.124 0.351 0.031 -1.754 -1.411 -1.603 -2.412 3.828
-3
-2
-1
0
1
2
3
4
5
Score

21. Final Grade Exam 1 Exam 2 Exam 3 Diagnostic Test
class 1 0.09 0.15 0.07 0.09 0.12
class 2 0.02 -0.32 0.1 -0.05 -0.13
class 3 -1.59 -1.05 -1.56 -1.29 -1.06
-1.8
-1.6
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
Score

22. • Student attitudes relatively important
• Replication of previous literature showing math
confidence important positive predictor of
math/Calculus success (e.g., House, 1995)
• Possibly role for measuring math anxiety too
• May be due to this being Calc II
• What happened to the cognitive predictors?
Discussion

23. • Online data also important relative predictors
• Assessment total negative predictor
• “procrastination” variable
• OR, students who struggle in Calculus found this
very hard
• Number of times retake quiz positive predictor
• “perfection” variable
Discussion

24. • We learn more when we look at BOTH:
• student’s interactions with online platform to
prediction of student success AND
• known “psychological” student characteristics
• But SO MUCH data, and most of it requires
huge assumptions
• Hard to know what we are measuring with the
online variables!
Conclusion

25. • What other information can we get from
clickstream data that might be useful?
• How to get it into a useable form?
• Can we predict how students will use the
online system from their characteristics?
• Can we then use this information to develop a
recommendation system?
Future Directions

26. • NSF grants 1450501 & E2030291
• Dr. Olga Caprotti & Yahya Almalki
hart@psy.fsu.edu
@saraannhart
Acknowledgements
ganley@psy.fsu.edu
@colleenganley