When you collect data has important implications for the conclusions that you can draw from that data. In education research, we often try to measure a difference, such as what students learn or how their experiences or perceptions change. Because we often try to make conclusions about differences, it can be equally important to take measurements at the beginning and end of a study.
Pre-Post Design
To measure a difference, we need to measure the level at which students start, such as their prior knowledge, and the level at which students finish, such as after a course or intervention, to make claims about how they’ve changed. The type of design that measures before (pre-test) and after (post-test) an intervention is called a pre-post design. This design is good at measuring any change from before the research started to after, such as how students’ perceptions of computer science differ from the beginning to the end of a course.
Research question – How do programming students’ learning outcomes differ when they complete homework problems in a group instead of by themselves?
Research design – Give students a test at the beginning of the semester, ask whether they complete problems in a group or by themselves, give students the same test at the end of the semester, and compare test performance between those that completed problems individually and those that worked in groups.
Researchers sometimes include multiple post-tests in this type of design. For example, if you wanted to measure prior knowledge before a course, learning at the end of a course, and retention 6-months after the course, then you could administer the same test at those time intervals. Technically, that would be considered a pre-post-post design.
Note:
It is tempting to argue that students should not have prior knowledge, perceptions, etc., and skip the pre-test. However, the conclusions that can be drawn about changes with a pre-test are much stronger than without one. Some researchers use a truncated pre-test to reduce measurement requirements when no prior knowledge is expected.
Post-Only Design
Post-only designs collect data only after the intervention. This design is good at measuring opinions or behaviors not present at the beginning. For example, the course evaluation surveys students complete at the end of the semester are a post-only design. This design is appropriate because it measures opinions about the course rather than something about the student, which could have existed before the course.
Like the pre-post design, a post-only design can include multiple post-tests to create post-post designs that take measurements throughout the intervention or after it finishes. Like a pre-post design, post-post designs can measure a change over time. The difference is whether the first test is at the start of the study or after the intervention has already begun.
Research question – How do students’ discussions differ when they use online forums instead of in-class discussions?
Research design – Assign students to use online forums or in-class discussions. Measure the discussions (e.g., number of contributions per person) and compare the quality between those who used online forums and those who discussed in class.
A caution about gain/difference scores:
A gain or difference score takes two points of data for a participant, such as a pre- and post-test, and calculates one score to represent the change over time. For example, a student who got a 50% on a midterm and 90% on the final exam would have a gain score of 40%, suggesting large improvements. Gain scores used to be popular in education research, but they have fallen out of favor as statistical techniques have improved because they are unreliable. Each measurement, such as the pre-test and the post-test, includes measurement error. When these scores are combined into one score, statistical analysis can no longer account for this error, leading to less reliable results than analyzing the two scores simultaneously. How to handle multiple data points per participant will be discussed in a later post on statistical analysis.
| Definition | Example | Explanation | |
| Pre-post design | Takes measurements before and during/after intervention to capture change | Give the same test at the beginning (pre-test) and end (post-test) of a course | By comparing the pre- and post-tests, the learning gain can be determined |
| Post-test | Takes measurements during/after intervention to capture outcomes | Give a survey (post-test) to measure students’ opinions at the end of a course | Survey responses show students’ opinions at the end of the course |
| Multiple post-tests | Takes measurements at multiple points during/after intervention | Give the same test at the end of a course (post-test) and 6 months later (post-test) to measure retention | Multiple post-tests allow researchers to track progress or retention |
To view more posts about research design, see a list of topics on theResearch Design: Series Introduction.
