|
Our quasi-experimental study compared experimental groups
(ACEPT reformed teaching) and comparison groups (non-reformed teaching) on
the basis of pretests - posttests gains in student achievement. We
discovered that the classroom observation measure we used co-varied with
student gains in a remarkable way (see the accompanying graphs).
Figure 1 (Full Size)
Figure 2 (Full Size)
The Physical Sciences 110 study involved 3 experimental groups and 3 control
groups. A 30 item physics concept test was administered as a pretest and
again as a posttest. The mean gain in learning (actually a "normalized
gain" - more on that later) for each class is plotted side-by-side with the
class's mean score on a classroom observation measure. As can be seen, when
one bar goes up the corresponding bar goes up. When one goes down, the other
goes down. The two means go up and down as if they were linked together! The
co-variation is remarkable. We were stunned. How could one variable so
closely track the other? Imagine our surprise when the same pattern
occurred in another study (Physics 121 - introductory physics) involving 4
classes. Again, the classroom observation mean follows the gain in
achievement almost perfectly.
At this point, it may be appropriate to explain what a
"normalized" gain score is just in case ECEPT visitors are thinking there is
something fishy about "normalized gains". Normalized gains were defined by
Physicist Richard Hake to account for the fact that different classes start
with different pretest scores. It may, for example, be easier to improve a
low pretest score than to improve a high one. He therefore "corrected" for
the initial pretest level by doing this:
"corrected gain" = (Posttest - Pretest)/(Total - Pretest) = "normalized gain".
What he did was compare the
actual gain to the possible gain as a ratio. If this is still unconvincing,
the graphs change very little if simple gain scores are used instead of
normalized gains.
The classroom observation measure we used is called the
Reformed Teaching Observation Protocol (RTOP), an instrument developed by
ACEPT evaluators. It was designed to measure the degree to which classroom
teaching of math or science is reformed. It was not designed to predict
student achievement. That this instrument would track achievement gains so
closely was a very pleasant outcome, both for the ACEPT project and for the
evaluators. Such "tracking" ability says something about the instrument. But
what?
Furthermore, as remarkable as the data are, would we be
stretching matters if we said, "the data show that reformed instruction
CAUSES students to learn more?" Is this going too far?
If you'd really like to get involved (and prove us
wrong!), leave your email and we will send you a copy of the "Reformed
Teaching Observation Protocol" and a User Manual to go along with it. We'd
like to know if we are having a strange aberration occurring here.
If you give it a shot in your studies, we'd be awfully
glad to hear from you.
Comments? Questions? Reactions?
Daiyo Sawada
ACEPT Project Evaluator
daiyo.sawada@ualberta.ca
| 
