User Tools

Site Tools


Experimental Designs I: Types of Designs


Experimental design A planned interference in the natural order of events.
Subject(s) Subject of interest - that to be studied, whether people, technology, natural phenomena, or other
Sample Typically you can't study all the individuals of a particular subject pool (set), so in your experiment you use a sample (subset) and hope that the results gathered using this subset generalize to the rest of the set (subject pool).
Between subjects vs. within subjects design Between subjects: Two separate groups of subject/phenomena measured
Within subjects: Same subjects/phenomena measured twice, on different occasions
Quasi-Experimental When conditions do not permit an ideal design to be used (a properly controlled experiement is not possible), there may still be some way to control some of the variables. This is called quasi-experimental design.
Dependent variable The measured variable(s) of the phenomenon which you are studying
Independent variable The variable(s) that you manipulate in order to systematically affect (or avoid affecting) the dependent variable(s)
Internal validity How likely is it that the manipulation of the independent variables caused the effect in dependent variables?
External validity How likely is it that the results generalize to other instances of the phenomenon under study?

Correlational Studies & Quasi-Experimental Design

Correlation Some factors/variables co-vary when changes in one variable are related with changes in the other, negative or positive
Correlation: Powerful tool Any variables in the world can be measured for correlation. Only two variables are needed (independent and dependent) for doing correlation studies
Main operating principle behind correlation There is no causation without correlation
Correlation: Pitfall Correlation does not imply causation between the variables measured!
Quasi-experimental designs
How it works 1. One-shot case study (no control group)
2. Single group pre- and post-test (minimal control)
3. ABAB: Single-group repeated measures (slightly less minimal control)
Limitations Much greater uncertainty as to the internal and external validity of the quasi-experiments than true experimental designs


What is it? A more loose, pre-study using the intended experimental design to tune it
A pre-study intended to gauge the nature, scales or other factors of the variables to be measured, or the subject to be measured
Why and when Pilots are much more useful than you might think. Yes, it will increase the duration and effort of your experiment BUT: It can significantly improve the quality of the subsequent experiment in many cases. It will certainly clarify and sharpen the experimenter's understanding of one or more of: the experiment, experimental procedure, variables and subjects.
Bottom line Do not try to “save time” by skipping a pilot if a pilot study seems to makes sense.

Field Studies

What is it? Quasi-experimental design. To study a phenomeon “in the wild”.
When When an experimental setup is out of the question.
Example H1: “The popularity of Nokia phones has to do with the quality of their user interface.”
H0: “The user interface has nothing to do with it.”
How Try to approximate a true experimental design as possible, by randomizing where possible, and by controlling the independent variables, if possible. Make the best attempt possible at analyzing potential alternative variables related to the dependent variable to be measured.
Bottom line Unavoidable in all fields of study. Very useful as a supportive method to true experiments.

ABAB (aka Repeated Measures)

What is it? Repeated measurements of the same sample, varying the independent variables between sessions
When When control group is not possible; When the group of subjects is small or single-case (e.g. medical studies)
Often done with only ABA Adding the last “B” increases tremendously internal validity
Bottom line Much more powerful than most books on experimental designs will tell you

Controled Experiment

What is it? A fairly recent research method, historically speaking, for testing hypotheses / theories
When When it is possible to control and select everything of importance to the subject of study
How Select subjects freely, randomize samples, remove experimenter effect through double-blind procedure, use control groups, select independent and dependent variables as necessary to answer the questions raised.
Why randomize? Given a complex phenomenon, it is impossible to know all potential causal chains that may exist between the various elements under study. Randomization lessens the probability that there is systematic bias in any factors that are not under study but could affect the results and thus imply different conclusions.
What is randomized? The sample should be randomized; subjects should be randomly assigned to control group versus experimental group; Any independent variable identified which could affect the results but is not considered of interest to the research at hand.
Bottom line The most powerful mechanism for generating reliable knowledge known to mankind.

Usability Studies

What is it? The study of human use of technology.
Not an experimental design paradigm in and of itself, yet important enough to warrant special discussion
When When technology and/or its users are of interest
How Experimental setup - easy to use true experimental design, but field studies also common
Not as common: Models of users - simulations, e.g. Model Human Processor (Card, Moran, Newell) - typically used in addition to basic experiments or as a pilot
Origin As people interact more with technology, questions regarding the outcome necessitate studying users and technology in context with each other
Bottom line Increasingly important in a world where more and more technology is interacting with humans


/var/www/ailab/WWW/wiki/data/pages/public/rem4/rem4-15/experimental_designs_i.txt · Last modified: 2015/09/15 11:19 by thorisson