Testing & evaluation (TEVAL) SIG workshop at JALT 2014 (Tsukuba), November 22.

1. +
Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University

2. +
Get ready to be Rasched…
(This is supposed to be a joke. Laugh.)
Structure of
this
workshop
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University

3. +
Rasch concepts
Terminology and explanations
Structure of
this
workshop
Ritsumeik
an
University

4. +
Structure of
this
workshop J. W.
Fukuoka
Women’s
University
Steps to create a
questionnaire
Lake
Constructs, concepts, items, piloting, evaluation,
revision…

5. +
e8
e9
e10
e11
Demotivation: An example
The flaws of traditional FA and why Rasch can
help questionnaire creation
e1
e2
MET
e3
ONE
PRON
e5
Structure of
this
workshop
Keita
Kikuchi
Kanagaw
a
University
Teacher
Behavior
GETA
.69
.85
.75
e4
.69
EXPL
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.67 .74 .76
.70
VID
e15
e6
GRAM
Experience
of difficulties
e7
VOCI
VOC
TEST
SELF
.72 .68 .66
.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83
.78 .77 .81
NOTU
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
.64
FRN
e16
.61
MST
e17
.50

6. +
Matthew Apple
Ritsumeikan University
Department of Communication
International Communication Program
Rasch
concepts

7. +
Rasch terms
 Rasch log-odds (logits)
 Rasch measures (logit scores)
 Infit/Outfit (means sq. and z-score)
 Item difficulty / endorsability
 Person/item reliability / separation
 Construct validity and
unidimensionality (not strictly
speaking Rasch, but…)
 Rasch Principal components
analysis (Rasch PCA)
 Loadings
 Contrasts and Residuals
A probabilistic model

8. +
Logits (log-odds)
The probability of a person correctly
answering an item 50% of the time

9. +
Fit
0.75 to 1.3 logits
0.60 to 1.4 logits
“Within 2 standard deviations of the mean”
“Only Outfit z-scores of 3.0”

10. +
Separation
The ratio of error-free variance and
observed variance
(Fisher, 1992)
The number of groups distinguishable by the
measurement instrument
(Wilson, 2005;
Wright, 1996)

11. +
Item map
Persons and items on the same
linear logit scale

12. +
The “Line”
Items and persons on the same scale
Item-person map (or)
Wright map
Ben Wright

13. +
The “Line”
Items and persons on the same scale
Item-person map (or)
Wright map

14. +
How Rasch works
for questionnaires
 Likert-scale data
 Likert-type category data
 Questionnaires do not produce
true interval but ordinal data
 The steps in the “scale” can be
conceived as thresholds (τ)
 named after Thurston, originator
Refer to Andrich (1977, 1978) of factor analysis)

15. +
How Rasch works
for questionnaires
Refer to Andrich (1977, 1978)

16. +
How Rasch works
for questionnaires
Refer to Andrich (1977, 1978)
N
1
2
3
4
5

17. +
How Rasch works
for questionnaires
Refer to Andrich (1977, 1978)
N
1
2
3
4
5
1 + 3 = 4

18. +
How Rasch works
for questionnaires
Refer to Andrich (1977, 1978)
N
1
2
3
4
5
SD + N = A?
1 + 3 = 4?

20. +
J W Lake
Fukuoka Women’s University
Steps to
create a
questionnaire

21. Steps in scale development: Issues to
consider (Netemeyer, Bearden, Sharma,
2003)
Step 1: Construct definition and content domain
The importance of clear construct definition,
content domain, and the role of theory.
Construct dimensionality: unidimensional,
multidimensional, or a higher-order construct?
Determine the purpose of the scale:
measurement or correlational analysis or
model building

22. Step 2: Generating and judging measurement items
Theoretical assumptions about items (e.g., domain
sampling)
Generating potential items and determining the
response format
How many items as an initial pool
Dichotomous vs. polytomous response formats
Item wording issues
 The focus on “content” validity in relation to
theoretical dimensionality
Item judging (expert and layperson) --- the focus on

23. Step 3: designing and conducting studies to develop
and refine the scale
Pilot testing as an item-trimming procedure
The use of several samples from relevant populations
for scale development
Designing the studies to test psychometric properties
Initial item analyses via exploratory factor analyses
(EFAs)
Initial item analyses and internal consistency estimates
Retaining items for the next studies
EFA may be useful for correlational analysis or model
building

24. Step 4: Finalizing the scales
The importance of several samples from relevant
populations
Designing the studies to test the various types of
validity
Item analysis via EFA
The importance of EFA consistency from Step 3 to
Step 4
Deriving an initial factor structure—dimensionality
and theory
Item analyses and confirmatory factor analyses
(CFAs)
Testing the theoretical factor structure and model

25. +
e8
e9
e10
e11
Keita Kikuchi
Kanagawa University
e1
e2
MET
e3
ONE
PRON
e5
e6
Demotivatio
Teacher
Behavior
n: An
example
GETA
.69
.85
.75
e4
.69
EXPL
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.67 .74 .76
.70
VID
e15
GRAM
Experience
of difficulties
e7
VOCI
VOC
TEST
SELF
.72 .68 .66
.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83
.78 .77 .81
NOTU
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
.64
FRN
e16
.61
MST
e17
.50

26. Example Study
focused on specific external /internal forces that
Japanese high school students may experience
which might cause their motivation to be reduced
or diminished
 administered the questionnaire asking high school
students to report what diminished their motivation
to study in their high school days, which contained
40 Likert-scale questions (4-points)
analyzed the quantitative data using a
confirmatory factor analysis using Amos and
Rasch PCA of the residuals using Winsteps.
 If you’d like to read this process thoroughly, please
locate Kikuchi (forthcoming).

27. Demotivation
Dörnyei (2001)
Definition of demotivation
 “specific external forces that reduce or
diminish the motivational basis of a
behavioral intention or an ongoing action” (p.
143).
I expand this definition and explore
demotivators including both internal and
external forces.

28. Previous Studies (Dörnyei, 1998)
 Based on interviews with 50 secondary school students,
he identified following as demotivators, the reason to get
demotivated.
1. Teachers’ personalities, commitments, competence,
teaching methods.
2. Inadequate school facilities (very big group, not the right
level or frequent change of teachers).
3. Reduced self-confidence due to their experience of failure or
success.
4. Negative attitude toward the foreign language studied.
5. Compulsory nature of the foreign language study.
6. Interference of another foreign language that pupils are
studying.
7. Negative attitude toward the community of the foreign

29. Previous Studies (Kojima, 2004, p.42)
Languag
e Level
English
Learning
Demotivation
Learning
situation
Amoun
t of
study
.58
.86
.71
.46
.41
Gramm
ar
Readin
g
Self-confidenc
e
.89 personality
Writing Learning
method
Teacher
Change
of teaching
style
Teaching
approach
Memori
zing
Vocab.
Learner
Level
Listening
problem
Class
atmospher
e
.77
.89
.89
.85 .90 .92 .80 .83 .85
GFI = 0.906
AGFI = 0.890
RMSEA = 0.052

30. Previous Studies
 Kikuchi (2009)
 47 university students
 open-ended questionnaires
 reflection on high school days
 Kikuchi and Sakai (2009)
 112 university students
 a 35-item questionnaire with a 5-point scale
 Sakai and Kikuchi (2009)
 676 high school students
 a 35-item questionnaire with a 5-point scale

31. Common demotivation factors
 Sakai and Kikuchi(2009)
 F1: Learning Contents and Materials,
 F2: Teachers’ Competence and Teaching Styles
 F3: Inadequate School Facilities [Classroom Environment]
 F4: Lack of Intrinsic Motivation
 F5: Test Scores [Experience of Inferiority]
 Kikuchi and Sakai (2009)
 F1: Course Books
 F2: Inadequate School Facilities
 F3: Test Scores
 F4: Non-Communicative Methods
 F5: Teachers’ Competence and Teaching Styles
 Both studies used a principal axis factor analysis using the direct
oblimin rotation

32. Six Original Constructs
 Teachers: Teachers’ attitude, teaching competence,
language proficiency, personality, and teaching style
 Characteristics of classes: Course contents and pace,
focus on difficult grammar or vocabulary, monotonous and
boring lessons, a focus on university entrance exams and
the memorization of the language
 Experiences of failure: Disappointment due to test scores,
lack of acceptance by teachers and others, and feeling
unable to memorize vocabulary and idioms.
 Class environment: Attitude of classmates, compulsory
nature of English study, friends’ attitudes, inactive classes,
inappropriate level of the lessons, and inadequate use of
school facilities such as not using audio-visual materials
 Class materials: Not suitable or uninteresting materials (e.g.,
too many reference books and/or handouts)
 Lack of interest: Sense of English used at schools is not

33. Participants (N=1,266)

34. Method
Materials
background questions
a 40-item questionnaire
 “We would like to study the situations of English study in high schools.
The following statements are possible demotivating factors for English
learning. To what extent are these statements true for you? Answer
based on your experience.”
 Questions are revised from what I used in my
previous studies (Kikuchi and Sakai, in-press;
Sakai and Kikuchi, 2009).
Example of Items (1=Strongly disagree, 2=disagree,
3=agree, and 4=Strongly agree)
 Teachers made one-way explanations too often.
 The number of students in classes was large.
 A great number of textbooks and supplementary readers were
assigned.
 I lost my understanding of the purpose of studying English.

35. Results of EFA
An Exploratory Factor Analysis
40 items
principal axis factor analysis with a promax
rotation procedure
a four-factor solution
teachers behaviors
class environment
experiences of difficulties
Lack of interest
Only 22 items left to be included in Confirmatory
Factor Analysis.

36. Table 2:Factor Analysis of Demotivation
No. Item descriptions F 1 F 2 F 3 F 4
Factor 1: Experience of difficulties(α = .87)
i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11
i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08
i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09
i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05
i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08
i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09
i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09
Factor 2: Teacher behavior(α = .84)
i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00
i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02
i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02
i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07
i34 I could not get along with teachers. -0.10 0.68 0.12 0.05
i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03
Factor 3: Class environment(α = .85)
i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06
i23 The Internet was not used. -0.12 -0.16 0.81 0.04
i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11
i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04
i10 My friends did not like English. 0.02 -0.05 0.67 -0.07
i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13
Factor 4: Lack of Interest(α = .90)
i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91
i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85
i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81
i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79
i24 I lost my interest in English. 0.12 -0.02 0.01 0.71

37. Method
Analysis
Rasch PCA of the residuals/Confirmatory
factor analysis of these six factors
Conventional factor analysis confirmed
only four factors!
Rasch PCA factor analysis /Confirmatory
factor analysis of these four factors were
conducted once again…

38. +
Enter the Rasch
With Matt

39. +
Category utility
Measures the distance between
thresholds among the Likert-type
categories (“steps” of the scale)

40. +
Rasch PCA output
Loading
Measure
Infit means squared
Outfit means squared
Principal components analysis

41. +
Item fit analysis
Measure
Standard error
Infit Outfit means squared & z

42. +
e8
e9
e10
e11
Keita Kikuchi
Kanagawa University
e1
e2
MET
e3
ONE
PRON
e5
e6
Demotivatio
Teacher
Behavior
n, Part
Deux: The
Rasched
GETA
.69
.85
.75
e4
.69
EXPL
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.67 .74 .76
.70
VID
e15
GRAM
Experience
of difficulties
e7
VOCI
VOC
TEST
SELF
.72 .68 .66
.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83
.78 .77 .81
NOTU
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
.64
FRN
e16
.61
MST
e17
.50

43. Results
 Rating Scale Instrument Quality Criteria (based on Fisher, 2007)
Criterion Poor Fair Good Very Good Excellent
Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3
Person and item measurement
<.67 .67-.80 .81-.90 .91-.94 >.94
reliability
Variance in data explained by
measures
<50% 50-60% 60-70% 70-80% >80%
Unexplained variance in 1st
contrast of PCA residuals
>15% 10-15% 5-10% 3-5% <3%

44. Table 1: Variance in measure explained by each demotivator
construct.
Six Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance
explained by 1st
contrast
Teachers 57.0% 43.0% 12.3%
Characteristics of classes 45.4% 54.6% 9.7%
Experiences failure 57.1% 42.9% 12.7%
Class environment 40.4% 59.6% 13.3%
Class materials 55.1% 44.9% 10.8%
Lack of interest 62.1% 37.9% 12.8%

45. Results –Rasch PCA- Loadings of
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Contrasts
Factor
loadings
1. teachers(k=6, Rp=0.99, Gp=11.02)
15 Teachers shout or got angry 0.97 1.42 1.38 0.68
5 Teacher asked us to use accurate grammar 0.54 1.26 1.25 0.68
14 Teachers explanation not easy -0.85 0.79 0.85 -0.63
11 Teachers bad pronunciation 0.16 0.98 0.91 -0.45
40 Teachers’ bad teach method -0.31 0.75 0.72 -0.44
13 Teachers one-way explanation -0.51 0.83 0.84 -0.03
2. Characteristics of Classes (k=9, Rp=0.98, Gp=6.53)
10 Inappropriate pace of lesson -0.10 1.00 1.04 0.58
41 Monotonous class -0.22 0.98 1.04 0.58
1 Rare chance of communication -0.14 1.19 1.24 0.45
2 Focused on translation 0.06 0.81 0.83 -0.57
3 Focused on grammar -0.35 0.96 1.01 -0.56
42 Amount to study for mid-term/final tests -0.32 1.19 1.29 -0.20
6 Required memorizing passages in textbooks 0.26 0.94 0.95 -0.15
43 Amount of handout distributed 0.11 0.94 0.95 -0.13
4 Focused on college entrance exam. Prep. 0.69 0.90 0.89 -0.12

46. No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
3. Experiences of Failure (k=6, Rp=0.99, Gp=8.89)
27 Did not do well on tests compared w friends 0.64 1.04 1.06 0.76
8 low scores on school test 0.25 0.84 0.84 0.68
37 Did not understand grammar -0.15 1.07 1.07 -0.64
36 Did not understand class 0.23 1.01 1.01 -0.56
9 Get lost in self-study -0.26 1.05 1.07 -0.12
7 Could not memorize vocabulary and idiom -0.71 0.95 0.96 -0.05
4. Class Environment (k=6, Rp=0.98, Gp=7.40)
31 English being compulsory subject -0.73 1.08 1.20 0.82
26 Too many students in class 0.00 1.05 1.00 0.23
22 Video and DVDs not used -0.41 0.91 0.90 0.05
25 Audio not used 0.04 0.87 0.85 -0.59
29 Friends did not like English 0.67 0.97 1.03 -0.42
23 Internet not used 0.43 1.08 0.99 -0.31

47. No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
5. Class Materials (k=6, Rp=1.00, Gp=7.30)
16 Topics of Passages uninteresting 0.14 1.06 1.09 0.57
20 Topics of Passages old 1.42 0.96 0.95 0.56
35 Unclear answers to Questions 0.22 1.01 1.03 0.41
19 too much reading 0.44 0.97 1.00 0.19
44 Many difficult Vocabulary -1.21 0.98 1.00 -0.56
18 Sentences were difficult to read -0.97 0.98 1.03 -0.55
17 Passages too long -0.04 1.00 1.01 -0.47
6. Lack of Interest (k=5, Rp=0.98, Gp=6.87)
34 No goal for being a person who can use Eng 0.31 0.92 0.89 0.75
39 No need studying English 0.63 0.85 0.84 0.73
33 Lost interest in English -0.30 1.15 1.14 -0.56
32 Lost purpose of study English -0.59 1.06 1.06 -0.42
46 No use of English in the future -0.05 0.97 0.95 -0.31

48. Figure 1: CFA of 6 factor models of demotivators
Notes.
GFI=800
CFI=808
RMSEA=
.072
e6 i6
e5 i17 .76
teacher
.82
e4 i18
.60
e3 i30.73
e1 i36
.56
e2 i34
.68
e7 i5
.74
.47
.60
.55
lesson
e14 i4
e13 i19
e12 i20
e11 i21
.60
e10 i22
.58
e9 i25
e8 i29
.61
.59
e15 i31
.59
e16 i32
.46
environment
i10 e17
i23 e18
i28 e19
i35 e20
i37 e21
i40 e22
.63
.67
.71
.70
.65
.62
material
i1 e23
i9 e24
i12 e25
i14 e26
i16 e27
i27 e28
.53
.57
.50
.56
.59
.57
interest
i2 e29
i3 e30
i11 e31
i24 e32
i26 e33
.82
.83
.77
.79
.81
failure
i39
e35
i33
e36
i15
e37
i13
e38
i8
e39
i7
e40
.68 .74 .71 .67 .60 .67
i38 e41
.68
.79
.37
.27
.59
.54
.70
.56
.93
.79
.54
.88
.52
.73
.63
.57

49. Figure 2.
CFA of
4 factor
models
Notes.
GFI=916
CFI=926
RMSEA=
.062
e3
ONEW
PRON
Teacher
Behavior
e1
GETA
e2
MET
.69
.85
.75
e4
.69
e5
EXPL
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.67 .74 .76
.70
VID
e15
e6
GRAM
e9
e10
Experience
of difficulties
e7
VOCI
e8
VOC
TEST
SELFS
.71 .67 .66
.68
Lack of
Interest
e11
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOU
e18
.82
.83
.79 .77 .81
NOTU
.75
.67
.64
FRN
e16
.61
MST
e17
.46
.51
.44
.53
.23
.35

50. e3
ONE
PRON
Teacher
Behavior
e1
GETA
e2
MET
.69
.85
.75
e4
.69
e5
EXPL
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.67 .74 .76
.70
VID
e15
e6
GRAM
e9
e10
Experience
of difficulties
e7
VOCI
e8
VOC
TEST
SELF
.72 .68 .66
.68
Lack of
Interest
e11
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83
.78 .77 .81
NOTU
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
.64
FRN
e16
.61
MST
e17
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
Figure 3.
My tentative
model of
demotivation

51. Results –Rasch PCA of four factor
models-
Four Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance explained
by 1st contrast
Teachers 58.5% 41.5% 10.7%
Experience of difficulties 58.0% 42.0% 12.2%
Class environment 39.9% 60.1% 13.3%
Lack of interest 54.8% 45.2% 15.1%

52. This is how the poor factor was
working…

53. An activity for this workshop
 Let’s try to make item bank of questionnaire
items together for your practice.
Topic is Demotivating factors in English
education for communication in Japan.
With your partner, think of constructs first and
write items for each construct. How many
constructs? How many items for each of them?
Remember “the Line”!
Please use the questionnaire items that you have
in your handout about demotivating factors in
English education in high school English
classroom to generate your discussion.

54. References
 Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in
the human sciences. Mahwah, NJ: Lawrence Erlbaum.
 Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman.
 Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007,
from http://www.rasch.org/rmt/rmt211m.htm
 Kikuchi, K. (2009). Student demotivation in Japanese high school English classrooms:
Exploring with qualitative research methods. Language Teaching Research, 13(4), pp.453-
471.
 Kikuchi, K. (forthcoming). What are possible demotivators in SLA? –An insight from English
teaching contexts in Japan. Multilingual Matters
 Kojima, S. (2004). English learning demotivation in Japanese EFL students: Research in
demotivational patterns from the qualitative research results of three different types of high
schools. Unpublished master thesis, Kwansei Gakuin University, Hyogo, Japan.
 Linacre, J. M. (1997). Guidelines for rating scales. Retrieved November 25, 2007, from
http://www.rasch.org/rn2.htm.
 Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and
applications. Thousand Oaks, CA: Sage Publications.
 Sakai, H., & Kikuchi, K. (2009). Japanese learners' demotivation to study English: A survey
study? JALT Journal, 31 (2), pp.183-204.
 Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah,
NJ: Lawrence Erlbaum.

58. Q & A and discussion

59. The Rasch model
(Rasch, 1960)

60. The Rating Scale Model
(Andrich, 1978)

61. +
Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
Send future inquires to Keita:
keita@kanagawa-u.ac.jp