Chapter 17

Experimental and Quasi-Experimental Research

Experimental Research attempts to establish cause and effect relationships. That is, an independent variable is manipulated to judge its effects on a dependent variable.

Three criteria are used to establish cause and effect:

•      The cause must precede the effect in time.

•      The cause and effect must be correlated with each other

•      The correlation between cause and effect cannot be explained by another variable.

Cause and Effect are not established by Statistics

Statistical techniques can only reject the Null Hypothesis, and identify the percentage of the variance accounted for by the independent variable or the effect size.

Both of these procedures are necessary but not sufficient to establish C&E.

Cause and Effect can only be established by the application of logical thinking to well-designed experiments.

The Logical Process

This logical process establishes that no other reasonable explanation exists for the changes in the dependent variable except the manipulation of the independent variable.

To facilitate the logical process:

•    Select a good theoretical framework

•    Use appropriate participants

•    Apply appropriate experimental design

•    Properly select and control the independent variable.

 

5) Appropriately select and measure the dependent variable

6) Use the correct statistical model and analysis

7) Interpret the results correctly.

Review the following terms

Independent variable – the cause

Dependent variable – the effect, the variable measured.

Categorical variable –A kind of independent variable that cannot be manipulated; age, sex, race, etc.

Control variable – a factor that could possibly influence the results and that is kept out of the study

Extraneous variable - a factor that could possibly influence the relationship between the independent and dependent variables, but is not controlled or included.

Sources of Invalidity

Internal validity is the basic minimum without which any experiment is uninterpretable.

Did in fact the experimental treatments make a difference in this specific experimental instance?

External validity asks the question of generalizability.

To what populations, settings, or treatment variables can this effect be generalized?

Internal Validity

Gaining internal validity involves controlling all variables so that the researcher can eliminate all rival hypotheses as explanations for the outcomes observed.

By controlling and constraining the research setting to gain internal validity, generalizability is in jeopardy.

 

The researcher must decide:  Is it more important to be certain that the manipulation of the independent variable caused the observed changes in the dependent variable, or is it more important to be able to generalize the results to other populations, settings, etc.

No single experiment can meet all design considerations.

Is internal or external validity more important?

Threats to Internal Validity

History – events occurring during the experiment that are not part of the treatment

Maturation – processes within the participants that operate as a result of time passing (e.g., aging, fatigue, hunger)

Testing – the effects of one test on subsequent administrations of the same test

Instrumentation – changes in instrument calibration, including lack of agreement within and between observers.

 

Statistical regression – the fact that groups selected on the basis of extreme scores are not as extreme on subsequent testing

Selection bias – choosing comparison groups in a nonrandom manner

Experimental mortality – loss of participants from comparison groups for nonrandom reasons

Selection-maturation interaction – the passage of time affecting one group but not the other in nonequivalent group designs.

Expectancy – experimenters’ anticipating that certain participants will perform better.

Threats to External Validity

Reactive of interactive effects of testing – The pretest may make the participant more aware of or sensitive to the upcoming treatment.  The treatment is not as effective without the pretest.

Interaction of Selection Bias and Experimental treatment – When a group is selected on some characteristic, the treatment may work only on groups possessing that characteristic

 

Reactive Effects of Experimental Arrangements – Treatments that are effective in very constrained situations may not be effective in less constrained settings.  Hawthorne effect is when people’s performances change when attention is paid to them.

Multiple-treatment Interference – When participants receive more than one treatment, the effects of previous treatments may influence subsequent ones.   Two groups may be better

Controlling Threats to Internal Validity

To make the threats to internal validity less the experimental and control groups must be as much alike as possible.

Randomization – allows the assumption that the groups do not differ at the beginning of the experiment. Controls for history before the experiment, for maturation, for statistical regression, selection biases, and selection-maturation interaction (the latter 3 occur when randomization does not occur).

 

A matched-group technique may also control for equality of groups, by matching subjects on some characteristic, but this may also mean that they are not similar on another characteristic.

In within subjects design the group serves as both the control and the experimental group.

Placebos, Blind Setups, and Double-blind Setups

Placebo is used to evaluate whether the observed effect is produced by the treatment or is a psychological effect. 

Blind setup is when the participant does not know whether they are receiving the experimental treatment or the control.

Double-blind is when neither the experimenter of the subject know whether the subject is receiving the experimental treatment or the control.

 

All these techniques are useful  in controlling the Hawthorne effect, the halo effect, and the Avis effect (where the subjects in the control group try harder just because they are not suppose to get the treatment).

Uncontrolled threats to Internal Validity

Not controlled by randomization:

•      Reactive or Interactive Effects can only be controlled by eliminating the pretest, or use another design; either the pretest-posttest randomized-groups, or Solomon Four-group

•      Instrumentation cannot be controlled or evaluated by any design

•      Experimental mortality cannot be controlled by any design

Controlling Threats to External Validity

•     External validity is generally controlled by selecting the participants, treatments, experimental situation, and tests to represent some larger population.

•     Randomization is the key to controlling most threats to external validity.

•     Randomize the selection of subjects, treatment levels, experimental situations, and dependent variables.

 

•     How do the participants perceive the study? 

Types of Designs

Preexperimental Designs

True Experimental Designs

Quasi-experimental designs

Notations:

•    Each line is a group of participants

•    R signifies random assignment of participants to groups

•    O signifies an observation or a test

•    T signifies that a treatment is applied; blank space in a line indicates a Control

Preexperimental Designs

•     Control very few sources of invalidity

•     None has random selection of subjects

One-shot study:  all subjects receive a treatment followed by a test to evaluate the treatment. Cannot attribute the level of performance (O) to the treatment.

                   T  O

One-Group Pretest-Posttest Design:

This is weak but better than the one-shot design:

 

O1  T  O2

 

This design does not tell why the subjects improved. 

Static Group Comparison

Compares two groups, one of which receives the treatment and one of which does not.

 

T     O1

----------

       O2

The groups were not equivalent before the study.

 

True Experimental Designs

Groups are randomly formed, allowing the assumption that the groups are equivalent.

Randomized Group Design:

 

R   T   O1    or    R    T1    O1

R        O2                R    T2   O2

                                       R           O3

 

Randomized Factorial Design

     R     A1     O1

B1  R     A2     O2

     R     A3     O3

---------------------

     R     A1     O4

B2  R     A2     O5

     R     A3     O6

Independent variable A has 3 levels and B has 2 levels. A 3X2 ANOVA, but not a true experimental design since B levels are not randomized.

 

Pretest-Posttest Randomized-Groups Design

•     Both groups are randomly formed and both get a pretest and a posttest

 

R   O1   T   O2

 

R   O3         O4

 

Purpose is to determine the amount of change  produced by the treatment

 

There are three ways to do a statistical analysis:

•     A 2-factor ANOVA where the treatment vs. notreatment is one factor and pretest vs posttest is the other.

•     ANCOVA – Analysis of Covariance uses the pretest scores of each group to adjust the posttest scores.

•     Difference scores – the pretest score is subtracted from the posttest and a simple ANOVA is performed.

Difference Scores

•      Tend to be unreliable

•     The level of initial values applies: participants who start lower in performance can improve more easily than those who begin with high scores. 

•     Initial scores are negatively correlated with the difference scores

Solomon Four-Group Design

A True Experimental Design which specifically evaluates the threat to external validity: reactive or interactive effects of testing.

R  O1  T   O2

R  O3       O4

R       T    O5

R             O6

 

This combines the randomized-groups and the pretest-posttest randomized-groups designs.

This determines whether the pretest increased the sensitivity of the participants to the treatment.

Is O2 > O4; is O5  > O6 ;  replication of the treatment effect.

Is O2  - O1 > O4  - O3 ; an assessment of the amount of change due to the treatment

 

Is O4  >  O6 ; an evaluation of the testing effect.

Is O2  >  O5 ;  an assessment of whether the pretest interacts with the treatment

This design is powerful but inefficient, since it requires twice as many subjects

The best alternative is the 2X2 ANOVA

                 No T      T

Pretest          O4        O2

Nopretest      O6        O5

 

Quasi-Experimental Designs

The Purpose is to fit the design to settings more like the real world while still controlling as many of the threats to internal validity as possible.

Time-Series Design attempts to show that the changes from the treatment differ from the times when the treatment was not administered.

O1  O2  O3  O4  T  O5   O6  O7  O8

 

 

Reversal Design is used increasingly in school settings.

O1  O2  T1  O3  O4  T2  O5  O6

 

 

Nonequivalent-Control-Group Design is frequently used in real-world settings where groups cannot be randomly formed

O1   T   O2

------------

O3        O4

This is a pretest-posttest design without randomization

O1 and O3 are compared and if there is no significant difference they are deemed equivalent, even though they may differ on other variables.  If they differ ANCOVA is used to adjust O2 and  O4

 

Ex Post Facto Design is a static group comparison where the treatment is not under the control of the experimenter.

Example:  athletes vs. nonathletes; fit vs. unfit; male vs. female

Purpose is to search for variables that differ between the groups

It is asking the question, “Did these variables influence the way these groups became different?”  This question cannot be answered by this design, but it may increase the insight into the characteristics that could be manipulated in a different experimental design.

Switched-Replication Design

Can be either true or quasi-experimental depending on whether levels are random or intact groups.

If they are randomly assigned to the groups it is a true experimental design.

If the levels are intact groups then the design is quasi-experimental

The number of trials must be one greater than the number of levels