Modelling the Model

What happens when being able to model the model becomes a stronger predictor than raw intelligence?

I keep coming back to this tweet.

We're treating language models like people, not language models.

Animal intelligence is different because it was shaped by evolution - embodiment, survival and death. We needed broad and robust competence, or (very literally), game over. So we form beliefs and values and follow social norms.

LLMs aren't belief-holders or norm-followers, at least in the way we are. They're next-token predictors shaped by reinforcement learning and feedback to produce helpful-sounding outputs. Coherence can be mistaken for understanding.

But ... it's all muddied af - because LLMs are trained on human language, so they sound human and trigger our social/psychological instincts, even though their competence is distribution-shaped and jagged. It's really hard to escape this.

Regardless of your position on whether models are 'truly understanding' or not, they just don't have the same evolutionary pressures that shaped animal intelligence.

What happens if (when?) human problem-solving/intelligence starts decoupling from steering AI? i.e. being able to 'model the model' becomes a stronger predictor of outcomes than one's own intelligence.

At this point, developing better 'model intuition' becomes critical.

Put more bluntly, if you believe that (1) AI becomes increasingly generally intelligent, and (2) that intelligence is fundamentally different than human intelligence, then it's likely that 'understanding how a model thinks and works' will become a far greater predictor than raw human intelligence.

A recent study quantifying human-AI synergy found that Theory of Mind capabilities predict collaborative performance better than traditional problem-solving skills.

My interpretation of this, is - understanding how the model thinks and works is a stronger predictor of success than raw human intelligence.