How a psychometric test is actually built

A psychometric instrument that reaches publication has usually been through five or six years of work and thrown away most of what was written for it. The sequence is worth knowing, because every stage of it corresponds to a way a test can be inadequate.

1. Define the construct

Before any item is written, the construct has to be specified precisely enough to say what falls outside it. This means a definition, a statement of dimensional structure — one dimension or several, and if several, how they relate — and an explicit account of which neighbouring constructs it is meant to be distinct from.

Skipping this stage is the origin of most redundant instruments. A scale built from an intuitive sense of a concept usually ends up measuring an established trait under a new name.

2. Generate an item pool

Items are drafted well in excess of what will survive — typically three to five times the final count. They come from theory, from existing instruments, from interviews with people who have the experience being described, and from subject-matter experts.

At this stage the pool is reviewed for content coverage (does it span the whole construct or cluster on one facet?), for reading level, for double-barrelled wording, and for items that are really asking about something else.

3. Pilot and cut

The pool goes to a first sample. The analysis is mostly destructive.

Items with almost no variance go — a question everyone answers the same way distinguishes nobody. Items that correlate poorly with the total score go. Items that cross-load onto more than one factor go. Items that behave differently across demographic groups for reasons unrelated to the trait — differential item functioning — go, and this check is one of the most commonly skipped.

What survives is typically a third of what was written.

4. Confirm the structure in a fresh sample

This is the step that separates serious instruments from the rest. Factor analysis on the development sample will recover the structure that was designed in — it has to, since the items were selected to produce it. The real test is confirmatory factor analysis in an independent sample.

A great many published scales recover their intended structure only in the data used to build them. When someone else collects new data, the factors do not appear. That failure is common enough that "was the structure confirmed in an independent sample?" is one of the most efficient questions to ask about any instrument.

5. Accumulate validity evidence

Correlations with established measures of the same construct. Correlations with constructs it should be distinct from, which had better be lower. Relationships with outcomes that matter. Incremental validity over what already exists. Test-retest stability over an interval appropriate to the construct.

This stage does not end. It continues for as long as the instrument is in use, and it is where instruments are most often found wanting years after publication.

6. Build norms

A reference sample large enough and representative enough for the intended population, stratified where the trait varies by age or sex, documented as to year and country. Then repeated periodically, because norms drift.

7. Translate, if it will be used elsewhere

Translation is not a linguistic exercise. Proper adaptation means forward translation, independent back translation, expert review, cognitive interviews with target-language speakers, and then a fresh psychometric study in the new language to test whether the structure holds and whether items function equivalently. An instrument translated without that study is an untested instrument in that language, whatever its credentials in the original.

What this implies about the quick quiz

A test written in an afternoon has completed step two. It may well produce a plausible-looking result, a percentile and a paragraph of description. The steps it skipped are the ones that would have established whether any of that means anything.