Deeper Signals logo
Contact Us

How far can AI go in defining job fit?

Comparing Large Language Models with Subject Matter Experts on Soft Skills

Key facts

Hiring managers know the technical requirements of a role but struggle to define the soft skills behind success, and more are turning to large language models (LLMs) for help. This study tested how closely AI agrees with expert judgment on which soft skills matter most.

  • The data: more than 1,000 occupations from O*NET, with expert (SME) importance ratings for 16 soft skills. Gemini, GPT-4 and GPT-3.5 ranked the same skills for each role.
  • The main takeaway: convergence was moderate. On average, the models identified about half of the top four soft skills that SMEs rated most important, with the strongest agreement on the top one or two.
  • What it is for: soft skill job templates for about 1,000 roles, anchored in SME ratings, with AI as a secondary signal.
  • The main limit: links between the profiles and actual performance are still being validated, and the profiles don’t replace employer-specific validation.

How we did it

We gave each LLM a job title, description and the 16 soft skill definitions, and asked it to rank and score the skills. We measured agreement with SMEs on each role’s top four skills, using top-four overlap and Jaccard similarity. Two prompt designs gave similar results. We also ran an adverse impact analysis using data from more than 30,000 working adults.

What we found

Gemini, GPT-4 and GPT-3.5 matched about half of the experts’ top four soft skills for 1,000+ occupations, averaging 2.22, 2.14 and 1.97 of four.
  • It depends on the role. Agreement was highest for office and administrative support (55% Jaccard) and lowest for arts and design (28%), where creative judgment and tacit knowledge matter most.
  • Clarity beats visibility. Agreement tracks how clearly a role’s behaviors are defined more than how often the role appears in the data.
  • Fair, checked job by job. No roles showed meaningful group differences by gender, age or ethnicity, and none violated the Four-Fifths Rule.

What this means for hiring teams

  • Use AI to validate, not to arbitrate. Agreement modestly raises a skill’s weight. Where AI and SMEs diverge, SMEs decide.
  • Expect less agreement in creative, tacit or context-specific roles.
  • Measure fairness job by job, and keep humans responsible for how results are used.

Key takeaways

Field Detail
Benchmark O*NET SME ratings of 16 soft skills across 1,000+ occupations
Models tested Gemini, GPT-4, GPT-3.5
Main result LLMs identified about half of SMEs’ top four soft skills (average overlap 1.97 to 2.22)
Strongest agreement Top one or two skills; administrative, repair and business operations roles
Weakest agreement Arts and design, cleaning and maintenance, and management roles
Fairness No meaningful group differences or Four-Fifths Rule violations
Use case Soft skill job templates for about 1,000 roles, with AI as a secondary signal

Authors

Director of AI and Assessment R&D at Deeper Signals
Dr. Luke Treglown

Dr. Luke Treglown is a personality and social psychologist, specializing in personality, cognition, resilience, burnout, organizational culture, and artificial intelligence, and he holds a PhD in the psychology of employee disenchantment. Luke has published more than 35 papers and books, with work featured in leading journals and outlets including the BBC and the European Business Review. He has advised governments, royal families, and global organizations on leadership, talent, and culture.

Get your copy