Why More Questions Make Assessments Worse

Written by Erika Bill Peter

In today’s market, most personality assessments follow a simple assumption: The more questions you ask, the more accurate the result.

So assessments get longer. 100 questions. 200.Sometimes 300.

They’re time-consuming. Cognitively fatiguing. And that fatigue is often accepted as the price of precision. At first glance, this makes sense.In psychometrics, more data should increase reliability and validity.

But here’s the problem: That assumption breaks down faster than most people think.

When More Becomes Noise

Many long assessments ask the same question in slightly different forms:

• “I enjoy leading others.”
• “I feel comfortable taking charge.”
• “I like being in a leadership role.”

In theory, this increases reliability. In practice, it often introduces response bias:

• People anchor on earlier answers
• They aim for consistency rather than accuracy
• Subtle distinctions get lost

The result? Clean-looking data with less real signal.

In psychological measurement, what actually matters is signal quality, not volume. After a certain point, adding more items doesn’t improve accuracy. It does the opposite. 

Why? Because humans aren’t static measuring instruments. They get tired. They adapt. They optimize.

Around question 120, something shifts. The respondent is no longer carefully reflecting. They’re scanning. Clicking. Repeating patterns.

• “Strongly agree… strongly agree… strongly agree…”
• Minimal differentiation
• Reduced attention

What you’re measuring now isn’t personality. It’s fatigue behavior.

The Science Behind It

From a psychometric perspective, three effects start to dominate in long assessments:

• Redundancy → diminishing informational return per item
• Response bias → consistency over truth
• Decision fatigue → declining cognitive engagement

This degrades construct validity, the degree to which you’re actually measuring what you think you’re measuring. In other words: More questions can make you feel rigorous…
while quietly making your data worse.

A Different Approach: Precision Over Volume

At Tilt, we took a different route. Instead of maximizing quantity, we focused on construct clarity.

• What are the core psychological dimensions that actually drive behavior?
• Which questions best discriminate between patterns?
• How do we reduce overlap and increase signal per item?

This leads to a fewer questions, sharper distinctions, cleaner data.

Why does this feels counterintuitive? Especially for data-driven thinkers, fewer questions can feel like less rigor. But rigor isn’t about length.