Can AI spot weak personality questions before we test them on people?

Key facts

Testing a personality item normally needs a sample of real people, which is slow and costly. We tested whether an AI model, acting as a stand-in respondent, can flag weak items, translations and new measures before anyone is recruited.

• The data: 300 real respondents to our Core Drivers assessment, drawn from 8,565 complete profiles. For each person, 12 of their 90 answers were hidden and predicted, resulting in 86,400 predictions.
• The main takeaway: given a person's six trait scores, the model matched 74.0% of their hidden forced-choice answers. Given 78 of their own earlier answers instead, it matched 66.4%.
• What it is for: deciding which items to send for human review first. It does not replace testing with people.
• The main limit: the trait scores came from the full assessment, which includes the answers being predicted. So this tests how well the model reconstructs a person, not how well it predicts one.

How we did it

We used Core Drivers, our 90-item personality assessment across six dimensions (Agreeableness, Conscientiousness, Drive, Extraversion, Openness and Stability). Its items are forced-choice (pick the statement that fits you better) or four-step ratings. We ran the study twice, on real English-language respondents from the same pool of 8,565 complete profiles. The first used 100 people. The larger replication used 300, chosen to cover low, middle and high scores; the correlations between their traits closely matched the full pool (r = .987). The model predicted each person's 12 hidden answers from one of two descriptions: their six trait scores (profile), or 78 of their own earlier answers (response history). In the larger study we repeated every run with different settings, such as whether the model could see which trait each question measured, and whether it was told to answer honestly, in a rush or in a slightly idealised way. Accuracy is the share of hidden answers the model matched.

What we found

A short, structured profile beat a long list of past answers on every measure, in both the first study (100 people) and the larger replication (300 people).

• The profile result held up at scale. It fell a little, from 78.2% to 74.0%, with three times as many people, and stayed ahead of response history.
• More information was not better information. The 78 past answers held the signal, but the model could not pick out the part that mattered.
• Settings mattered less than the description of the person. With profiles, temperature moved accuracy by 0.3 points. Letting the model see which trait each question measured moved it by 4.5 points (71.8% to 76.3%).

In short, the model can mostly mimic an individual, but it is a reconstruction, not a digital twin. Simulated groups looked human only when built from real people's profiles. Changing the model's settings added variety, but not human-like variety.

Where we have used it so far

The same approach has been tried on three practical problems, using the same 300 profiles and GPT-4.1-mini. In each, the simulated answers were used to spot items that look problematic, so that people can review those first. They were not used to approve anything.

• Translations: machine-translated Spanish and Hindi test versions of all 90 items. A screen flagged 25 of 90 Spanish and 34 of 90 Hindi items for review. In one flagged item, 41.0% chose "strongly agree" in simulated English but only 2.3% in Hindi. A second translation round moved 21 of 25 Spanish items closer to English, but only 12 of 30 Hindi items.
• New items: 30 draft Agreeableness items gave 9,000 simulated answers. The pool looked strong overall (alpha .925), yet 8 of 30 items needed a second look: one barely separated people, and another behaved more like Stability than Agreeableness.
• New types of measure: on situational judgement tests, answers varied very little and did not resemble human choices. This is an open result.