Different aspects of research quality are covered and concepts surrounding research triangulation are discussed.

1. Managing Quality in
Qualitative Research

2. How to Address Research Quality
Standards, Criteria, Checklists and Guidelines
Concepts of Triangulation
Triangulation of Qualitative and Quantitative Research
Focusing on Process and Transparency

3. How to Address Research Quality

4. The issue of quality in
qualitative research is located
at the crossroads of internal
needs and external challenges

5. • Researchers interest
• Can I trust the results that I have created?
• Are the methods that I have used in this situation
correct?
• If I work with others will our results be similar?
• Funding Institution approach
• How is one qualitative approach more better than
another?

6. • Journal Editor / Conference Editor decisions
• Can I trust the rigour of this piece of work?
• How do I know that this is the best approach?
• Readers questions
• Can I trust what I’m reading?
• How much detail did the researcher go into when
they were creating this?

7. It has become a crucial issue
with the establishment of
qualitative research as a valid
method and in competition with
other forms of work

8. The Independent Samples t-test compares the means of two
independent groups in order to determine whether there is
statistical evidence that the associated population means are
significantly different.

9. What is the qualitative
equivalent?

11. • Evaluation built on Standardisation
• One-Size-Fits-All Approach
• Criteria Appropriate to Research or
Research Appropriate to Criteria
• How to Assess Research Quality in a Sensitive Way
Different potential ways to approach this

12. The formulation of research
‘criteria’ is only one solution to
the problem

13. Standards, Criteria, Checklists and Guidelines

14. Reformulating Traditional Criteria
Aim here is to make the whole research process
more transparent
• Exact and Coherent Guidelines are used
• Interviews are transcribed in a certain
way using specific methods
• Distinction between verbatim
statements in field notes and summaries
are given
• Reliability of whole process can be seen
in reflexive documentation

15. Reformulating Traditional Criteria
Validating an Interview Situation
• Are interviewees given any cause to construct a biased version
of their experiences that doesn’t fit with their views?
• Script is analysed to look for these specific points and to
see whether the data is reliable
Communication Validation
• Participants can be involved in a secondary research study to
assist in validation of data
• Participants can reflect on previous answers and then
better articulate their thoughts

16. Reformulating Traditional Criteria
Procedural Validation
• Validation is looked at for the entire research process and the
different relationships that take place:
• Relationship between what is being observed and the larger
cultural, historical, and organisational contexts within which the
observations are made
• The relationship among the observer, the observed, and the setting
• The perspectives that are used to render an interpretation of any
ethnographic data
• The role of the reader in the final product
• The issue of author style to render the description or interpretation

17. Alternative, method-appropriate criteria
Instead of using traditional metrics, we use those
that are more credible for qualitative work
• Are findings grounded in the data?
• Is sampling appropriate
• Are data weighed correctly?
• Are inferences logical?
• Analytic strategies applied correctly?
• Alternative explanations accounted for?
• Is the category structure appropriate?
• Can decisions be justified?
• What is the degree of researcher bias?
• What strategies were used for increasing credibility?

18. Traditional criteria tend to miss
the specific qualities of
qualitative research.
Alternative criteria mostly come
without benchmarks for
distinguishing good from bad.

19. Evaluating Grounded Theory
Specific suggestions on evaluating Grounded
Theory studies come from Charmaz (2006) who
suggests breaking this down into four criteria
• Credibility
• Originality
• Resonance
• Usefulness

20. Evaluating Grounded Theory
• Has the research achieved familiarity with the setting or topic?
• Is the data sufficient to merit your claims?
• Think about range, number and depth of observations
contained in the data
• Have you made systematic comparisons between observations
and between categories
• Do the categories cover a wide range of empirical observations
• Are there strong links between the gathered data and your
argument and analysis?
• Has the research provided enough evidence for your claims to
allow the reader to form an independent assessment, and
agree with your claims?
Credibility

21. Evaluating Grounded Theory
• Are your categories fresh, do they offer new insights?
• Does your analysis provide a new conceptual rendering of the
data?
• What is the social and theoretical significance of this work?
• How does your theory challenge, extend, or refine current
ideas, concepts and practices?
Originality

22. Evaluating Grounded Theory
• Do the categories portray the fullness of the studied
experience?
• Have you revealed both liminal and unstable taken-for-granted
meanings
• Have you drawn links between larger collectivities or
institutions and individual lives?
• Does your grounded theory make sense to your participants or
people who share their circumstances? Does your analysis offer
them deeper insights about their lives and world?
Resonance

23. Evaluating Grounded Theory
• Does your analysis offer interpretations that people can use in
their everyday worlds?
• Do your analytic categories suggest any generic processes?
• If so, have you analysed these processes for
implications?
• Can the analysis spark further research in other substantive
areas?
• How does your work contribute to knowledge, how does it
contribute to making a better world!?
Usefulness

24. A Framework for Analysing Qualitative
Research Data
Research should be:
• Contributory in advancing wider knowledge
or understanding
• Defensible in design by providing a research
strategy
• Rigorous in conduct through the systemic
collection, analysis, and interpretation of
qualitative data
• Credible in Claim through offering well-
founded and plausible arguments about the
significance of the data generated
Quality in Qualitative Evaluation:
A framework for assessing
research evidence
A Quality Framework
Liz Spencer, Jane Ritchie, Jane Lewis and Lucy DillonNational Centre for Social Research
Government Chief Social Researcher’s Office
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c3

25. Standards will only be helpful if
they apply to qualitative
research in general and not to
specific approaches

26. Concepts of Triangulation

27. Triangulation is the method of location of a point
from two others of known distance apart, given the
angles of the triangle formed by the three points
x
y°
z°

28. Triangulation includes researchers taking different
perspectives on an issue under study or more
generally in answering research questions.
These perspectives can be substantiated by using
several methods or theoretical approaches,
Triangulation should produce knowledge at
different levels, which means they go beyond the
knowledge made possible by one approach and
thus contribute to promoting quality in research.

29. Types of Triangulation
• Data Triangulation
• The use of different sources of data
• Allows the researcher to reach maximum profit when using the
same methods
• Investigator Triangulation
• The use of different observers or interviewers
• This is not the sharing of work, it is a systemic comparison of
researchers influences on the issue
• Theory Triangulation
• Approaching the data with multiple perspectives and
hypothesis in mind
• Points are put side by side to assess their utility and power.
• Methodological Triangulation
• Different methods are used to look at the same issue

30. Within-Methods Triangulation
Method
Issue of
ResearchApproach 1
Approach 2
Example of this technique is the episodic interview which
combines questions and narratives.

31. Within-Methods Triangulation
Interviews can be used to understand the everyday
knowledge that people have
Semi-Structured interviews can include narratives where
participants talk at length about specific aspects that they
have encountered
BUT there is the chance for participants to switch from a
narrative discussion to a descriptive, argumentative, or
other non-native form of presentation
Episodic Interviews

32. Within-Methods Triangulation
Episodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Episodic Interview
Narrative
Presentation
Argumentative-
Theoretical
Presentation

33. Within-Methods Triangulation
Episodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Concept 1 Concept 2
Sub-Concept 1 Sub-Concept 3Sub-Concept 2
Episodic Interview
Semantic Knowledge
Narrative
Presentation
Argumentative-
Theoretical
Presentation

34. Within-Methods Triangulation
Episodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Concept 1 Concept 2
Sub-Concept 1 Sub-Concept 3Sub-Concept 2
Episodic Interview
Semantic Knowledge
Narrative
Presentation
Argumentative-
Theoretical
Presentation

35. Argumentations
Subjective Definitions
Examples
Repisodes
Narratives of Situations
Stereotypes
Situational Narratives based on
different levels of concreteness
Repisodes regularly occurring
situations, no longer based on a clear
local and temporal reference
Examples abstracted from concrete
situations, and metaphor also ranging
to clichés and stereotypes
Subjective Definitions which are asked
for within sessions
Linked to Argumentative Definitions of
terms, explanations of concepts
Within-Methods Triangulation

36. Between-Methods Triangulation
Method 1
Issue of
Research
Choose at least one method which is specifically suited to exploring the
structural aspects of the problem and at least one which can capture the
essential elements of its meaning to those involved
Method 2

37. Triangulation of Qualitative and
Quantitative Research

38. In this module we’ve dealt exclusively with
qualitative work. However, it is possible to combine
qualitative and quantitative methods in order to
increase research quality through triangulation

39. Quantitative
Qualitative
continuous collection of both sorts of data

40. Qualitative
Quantitative
continuous field research
wave 1 wave 2 wave 3

41. Qualitative
exploration
QualitativeQuantitative
questionnaire deepening and
assessing results

42. Quantitative
survey
QuantitativeQualitative
field study experiment

43. Linking Qualitative and Quantitative
Results
Results may converge, that is, are consistent completely, in general, by
tendency or partially. For example - answers in a representative survey
may match with statements from semi-structured interviews
Results may be complementary. Interviews can provide deeper, more
detailed explanations to complement results from a questionnaire
Results may diverge. For example, interviews may produce views that are
different when compared to questionnaires. This would cause the need
for further research.

44. Triangulation of qualitative and
quantitative research is not per
se a quality indicator for
qualitative research, but it can
contribute to overall quality

45. Qualitative Research Quantitative Research
Data Sets
Single Cases
Triangulation
Focus
Groups
Interviews
Questionnaires
Lab Studies

46. Positioning Triangulation in the Research
Process
Exploration Data Collection Data Interpretation
Generalisation

47. Focusing on Process and
Transparency

48. Quality in research is produced
through the whole process.

49. Questions to ask for selecting a qualitative
research method
I. What do I know about the issue of my study or how detailed is my
knowledge already?
II. How developed is the theoretical or empirical knowledge in the
literature about this issue?
III. Is my interest in more generally exploring the field and the issue of my
study?
IV. What is the background of my study and which methods fit with this?
V. What do I want to get close to in my study?
I. Personal Experiences of a group of people / social process
II. Reconstruction of underlying research structures

50. Questions to ask for selecting a qualitative
research method
VI. Do I start with a focused research question right away or do I start with
an unfocused approach in order to develop the question?
VII. Which aggregate do I want to study: personal experience, interactions or
situations, or bigger entities like organisations or discourse?
VIII.Is it more the single case I am interested in or the comparison of various
cases?
IX. Which resources (time, money, manpower, skills etc.) are available for
my study?

51. Questions to ask for selecting a qualitative
research method
X. What are the characteristics of the field I want to study and of the people
in it? What can you request of them and what not?
XI. What is the claim of generalisation of my study?
XII. What are the ethical issues to take into account that are affected by
selecting a specific method?

52. Decide and reflect carefully whether
you should use qualitative or
quantitative research
• Why qualitative research?
• Which reasons do you have for the
one or the other?
• What are your expectations for the
qualitative research that you are
planning?

53. Reflect on the theoretical background
of your knowledge interest
• What is the impact of your setting on
the research?
• How open and closed is your access
to what you want to study?

54. Plan your study carefully, but allow
for reconsidering the steps and
modifying according to the state of
play
• What are the resources available for
the study?
• How realistic are the aims of your
research in relation to the available
resources
• What are necessary and appropriate
shortcuts
x
y
z
a
b

55. Plan your sampling carefully!
• What are your cases?
• What do they stand for?

56. Think about whom in the field you
should contact and inform about your
research. Reflect about the relation
to establish to field subjects
• What can you learn about your
research field and issues from the
way you get into the field or are
rejected?

57. Think about why you chose your
methods for collecting data
• Is it a decision for a favourite method
or for habitual reasons?
• What could or would alternative
methods provide?
• What are the impacts of the methods
you use on your data and your
knowledge?

58. Plan carefully how to document your
data and research experiences
• How exactly should you write your
notes?
• What are the influences of the
documentation on your research and
on your field subjects?
• What are the impacts of the
documentation on your methods of
collection and analysis?

59. Think about the way that you want to
present what you have experienced
in the field and found in your
research
• What are the target audiences of your
writing?
• What is it mainly you want to
convince them about your research?
• What is the impact of the format of
your writing on your research and its
findings?

60. Plan how to establish the quality of
your research
• What are the quality criteria your
research should meet?
• How should these criteria be
realised?
• What is their impact on your research
and your field subjects or
relationships?

61. How to Address Research Quality
Standards, Criteria, Checklists and Guidelines
Concepts of Triangulation
Triangulation of Qualitative and Quantitative Research
Focusing on Process and Transparency

62. Information in this
presentation was
based on…