Factorial Experiment Design
SweetPea is a domain-specific programming language for creating factorial experimental designs. Our goal is to make it easy to specify and set up all of the factors and levels needed for your experiment.
Hint
There’s a glossary at the bottom of this page for a brief overview of the most important terms.
Wikipedia gives a decent introductory definition of a factorial experiment:
In statistics, a full factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or “levels”, and whose experimental units take on all possible combinations of these levels across all such factors. A full factorial design may also be called a fully crossed design. Such an experiment allows the investigator to study the effect of each factor on the response variable, as well as the effects of interactions between factors on the response variable.
Factors
A factor is an independent variable in your experiment. In simple experiments, there may be only one factor — in which case SweetPea is definitely overkill for you! But experiments can be successfully run with multiple factors without issue, and SweetPea will help you plan this out.
Fun Fact
The term factorial likely first appeared in the 1935 text The Design of Experiments by Ronald Fisher. The book is also credited with introducing the concept of the null hypothesis.
Levels
In a factorial experimental design, each factor can take on one of a finite number of discrete possible values, called levels. To make this more concrete, consider a simple example.
Example
Imagine we are conducting an experiment to observe the Stroop effect. In a simple Stroop experiment, we show a participant the name of a color, and we render the text of that word in a color other than that which was named. For this experiment, we identify our factors and levels:
Factor |
Levels |
|---|---|
Color Name |
red, blue, green, yellow |
Text Color |
red, blue, green, yellow |
If this is a full factorial experiment (also said that the experiment is
fully crossed), then we must conduct one trial for each possible
combination of factors and levels. In other words, we must have
Derivations
A derivation (or derived level) is a new level produced by
combining other levels. This combination of levels can be
directly constructed with sweetpea.DerivedLevel().
Derivations can also be constrained by derivation windows, which allow for specifying specific manners in which different levels from multiple factors can interact across trial boundaries:
Within-trial windows look only within a single given trial.
Transition windows, look at a trial and the previous one, so it can describe transitions between two consecutive trials.
General windows are defined in terms of a trial and any number of preceding trials.
ContinuousFactor windows offer similar functionality to general derivation windows, but operate on values of
ContinuousFactoracross a trial and any number of preceding trials, rather than on discrete levels.
These are explained more below.
Within-Trial Windows
The within-trial windows
(sweetpea.WithinTrial) describe a level that
is selected depending on levels from other factors, all within the same trial.
For example, when one factor is a color and another factor is the text of a color name,
each trial can be categozied as “congurent” or “incongrent” individually.
Transition Windows
A transition window
(sweetpea.Transition) describes a level that
is selected depending on a combination of levels from other factors in the
current trial and the immediately preceding trial. For example, a trial
might be categorized as “same” if it is categorized as “congurent”
and the previous trial was also categorized as “congurent”.
General Windows
A general derivation window
(sweetpea.Window) creates a level that is selected
depending on a combination of levels from other factors in the current trial and
zero or more preceding trials.
ContinuousFactor Windows
A continuous factor window
(sweetpea.ContinuousFactorWindow) is special kind of general window
that operates on ContinuousFactor instead of discrete levels.
It allows access to values of one or more ContinuousFactor
across a number of previous trials defined by window width.
This is useful for defining new continuous factors that depend on recent trends,
changes, or history — for example, computing the difference between the current
and previous trial values of reward.
Glossary
- constraint
An element of an experiment design that affects the generation of trials for the experiment. For example, a constraint may exclude a particular combination of levels, it may prevent a certain number of levels from appearing in consecutive sequences, or it may increase the number of trials in an experiment by establishing a minimum trial count.
- crossing
Short for experiment crossing.
- derivation
An artificial level that results from the combination of other levels. Also called a derived level.
- derivation window
A window constraining a derivation that depends on a combination of levels from other factors in the current trial and zero or more preceding trials.
- derivation window, transition
A derivation window that depends on a combination of levels from other factors in the current trial and the immediately preceding trial.
- derivation window, within-trial
A derivation window that depends on levels from other factors, all within the same trial.
- derivation window, continuous factor
A continuous factor window that operates over continuous-valued continuous factors rather than discrete levels. It provides runtime access to a sliding window of numeric values across multiple trials.
- derived level
See derivation.
- design
Short for experiment design.
- experiment
Usually, a particular instantiation of a sequence of trials for an experiment design. When clear from context, “experiment” may be used instead as a shorthand for experiment design.
- experiment crossing
A subset of the factors that define an experiment. Except as modified by exclusions and minimum-trials constraints, a sequence of trials for an experiment combines every possible level of each factor in the crossing with every level of every other factor in the crossing. For a factor that is not in the crossing, a level is assigned independently or based on explicit constraints or derived levels.
- experiment design
The set of factors and coonstraints that define an experiment and that determine the trials of the experiment.
- factor
An independent variable in a factorial experiment, composed of finitely many levels.
- continuous factor
A factor whose values are sampled from a distribution function at runtime, rather than chosen from a predefined list of discrete levels.
- factorial experiment
An experimental design measuring multiple independent variables (called factors) consisting of finitely many discrete possible values (called levels). A factorial experiment can either be classified as a full factorial experiment or a fractional factorial experiment.
- fractional factorial experiment
A factorial experiment consisting of a specific subset of possible trials that together expose meaningful information about important features of the problem without the resources or redundancy of a full factorial experiment. An experiment of this nature is said to only be partially crossed.
- full factorial experiment
A factorial experiment consisting of enough distinct trials to independently observe each possible combination of factors and levels. An experiment of this nature is said to be fully crossed. Contrasts with a fractional factorial experiment.
- fully crossed
A factorial experiment run with enough distinct trials to be able to distinguish the effects of each factor and level from one another.
- level
A discrete possible value that a factor can have.
- partially crossed
- transition window
- trial
An individual event in an experiment that is defined by a combination of levels. The numebr of trials contained in an experiment is determined in part by the number of factors and levels in the experiment design and the way that they are crossed.
- window
See derivation window.
- within-trial derivation
- continuous factor window