1. 1© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
See the PDE booklet,
Collecting evaluation data: Direct observations
http://learningstore.uwex.edu/pdf/G3658-05.pdf
Using
observation
to collect
evaluation
data

2. 2© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Checking in…
– Who has used observation as a data
collection method?
– What are your experiences with it?
– Who is hoping to use it – How?

3. 3© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Participant observation: What it is

4. 4© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Quantitative vs. qualitative methods
Quantitative Qualitative
Survey Observation
Tests Interview
Questionnaire Focus group

5. 5© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Observation…
Involves all 5 senses:
sight, hearing, smell, touch, and taste

6. 6© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Use in program evaluation
• When you want direct information
• When you are trying to understand an
ongoing behavior, process, unfolding
situation, or event
• When there is physical evidence, products,
or outcomes that can be readily seen
• When written or other data collection
methods seem inappropriate

7. 7© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Observations
Advantages
– Provides direct
information; not
dependent upon
someone’s
response
– Unobtrusive
– See things in
natural context
– Flexible; discovery
oriented
Disadvantages
– Observer’s presence
may create artificial
situation
– Potential for bias
– Time consuming
– Requires diligence,
preparation
– Challenging to collect
data while participating

8. 8© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Observation – Ethical issues
• Unobtrusiveness is its greatest strength; also
potential for abuse in invasion of privacy
• You can venture into places and gather data
almost anywhere so questions re. what is
ethical
– Overt vs. covert
• Remember our Human Subjects Protection
guidelines
– Consent form for participating in an
observational study
Consider cultural appropriateness of using observation

9. 9© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
What are the implications for using observation
as an evaluation data collection method?

10. 10© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Participant observation: How to do it?

11. 11© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
What to observe
• People (individuals, groups,
communities)
– Characteristics
– Interactions
– Behaviors
– Reactions
• Physical settings
• Environmental features
• Products/physical artifacts
Observing what does not happen may be as important as
observing what does happen

12. 12© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Types of observation
Structured Unstructured
Looking for vs. Looking at
Sometimes we have
something specific we want
to observe – leadership
skills; level of participation;
etc. We use a structured,
preset guide of what to
observe or a checklist.
Sometimes we want to
see what is naturally
occurring or exists without
predetermined ideas. We
use have an open-ended
approach to observation
and record all that we
observe

13. 13© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Practice:
Structured/unstructured observations
Imagine you are sitting in a room where ten
youth are sitting at computers learning about
Web 2.0 applications.
1) If you want to assess to what extent
students are interested and learning, what
specifically would you look (listen) for?
2) If you aren’t sure what specifically indicates
student interest or learning and you want to
see what is going on during the
demonstration, how would you proceed?

14. 14© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Example – Observing participation in an
after school program
• Who you will observe:
youth attending the program
• What you will observe:
– Age, gender
– Length of time student stays in the program
– Involvement in activities: which activities
• Level of involvement
– Interactions with other youth; with staff
• When you will observe: all hours the program
is open for one week each month during 2007

15. 15© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Recording your observations
It is not good enough to just observe,
you need to systematically record your
observations. You might use:
– Observation guide
– Recording sheet
– Checklist
– Field note
– Picture
– Combination of the above

16. 16© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Sample Observation Guides
Guide for structured observations Guide for unstructured observations

17. 17© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Structured observation guide used for pre and post program evaluation

18. 18© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Notebooks to record and collect observations
as they occur

19. 19© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Who does the observations?
• You – program staff
• Participants - Youth
• Parents
• Teachers
• Volunteers
• Other stakeholders
• Colleagues

20. 20© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Training –
preparation/orientation may be necessary
– To learn what to look for
– To learn how to record observations
– To practice
– To ensure that observations across sites are
consistent: observers use the same
methods, rate an observation in same way

21. 21© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
How well do you observe?

22. 22© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

23. 23© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Becoming a skilled observer includes…
• Learning to pay attention, see what there is to see,
and hear what there is to hear;
• Practice in writing descriptively;
• Acquiring discipline in recording notes;
• Knowing how to separate detail from trivia;
• Using rigorous methods to validate and triangulate
observations;
• Reporting the strengths and limitations of one’s own
perspective
» M.Q. Patton, 2002. Qualitative Research and
Evaluation Methods. Sage, pg 260

24. 24© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Data analysis and interpretation
Qualitative data = qualitative data analysis
– Standard content analysis
• Get to know your data
• Focus the analysis
• Categorize information
• Identify patterns and connections
• Interpret – bring it all together
PDE booklet: Analyzing Qualitative Data
http://learningstore.uwex.edu/Assets/pdfs/G3658-12.pdf

25. 25© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Practice
your observation skills
everyday in everyway!
And,
add observation
to your data collection toolbox.

26. 26© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
Steps in planning for observation
• Determine who/what will be observed.
• Determine aspects that will be observed
(characteristics, attributes, behaviors, etc.).
• Determine where and when observations
will be made.
• Develop the observation guide
• Pilot test the observation guide
• Train the observers and have them practice.
• Conduct the observations
• Analyze and interpret the collected information.
• Write up and use your findings.