Epistemic status: thinking aloud. Hitting my monthly Manifold market word target.
Functional hunger. What happens if you extract a “functional hunger” or “functional nausea” axis? To the extent that this is equally possible and would look broadly similar as “functional emotions”, this might make you want to refine your story about what (e.g.) “functional anger” is as well. At the very least, I think that doing this would be an interesting extension.
As Goldstein and Levinstein and others have complained, the “emotion vectors” that Anthropic calls “functional emotions” seem to play rather thin versions of the causal roles you might expect an emotional state to play. I think that this concern might have been more immediately striking if researchers had, from the start, also extracted vectors for states that (a) models also presumably represent various speakers as having, as with emotions like anger and fear, but which (b) are so much more saliently embodied that it is less obvious how or why a an LLM would be able to instantiate a state with its full functional role.
Role play. In various papers, people will use interp-based arguments to say that the mechanisms present in post-trained model are largely the same as those in base model. Now, I personally am not 100% clear what exactly what those various arguments / techniques / measures are, for making this mechanism comparison. With that said, assuming we have some grip on that: if we apply those same mechanism comparison arguments / techniques / measures, then how different does “the Assistant speaking as a character in a fiction story it is writing” look from “the Assistant speaking”?
Another way of putting this, some people claim that all LLM writing should be understood as role-play. Whatever the status of that claim, we know that at least some LLM writing is role-play. How, if at all, does the production of this text look different from “normal” assistant writing?
The Persona Selection Model in Late 2026. Where are we at with the “Persona Selection Model”? Especially with RL becoming more of training, all of the reward-seeking we’re seeing, and so on. As an example of someone asking “what’s up with PSM these days?”, Jan Kulveit wrote a couple of days ago, “My impression is many are doing some weird pendulum overupdate. Persona Selection Model was somewhat wrong and obsolete when published, but people got too much into it. Now it seems people are updating too much in the direction ‘inhuman reward seekers exactly foretold in classical AI risk stories’.” Even closer to the source, a main author has himself weighed in on what we should make of PSM “as AIs have become more “RLVR-brained”. And nostalgebraist, a primary inspirer of the PSM, weighs in as well.
Introspection and model size. Part of the original vibe of the AI introspection literature was “it is plausible that introspection is important for capabilities and that it would emerge with scale”. So, when Lindsey (2025) found that “The most capable models we tested, Claude Opus 4 and 4.1, exhibit the greatest degree of introspective awareness”, that was cool. That said, he did flag that “trends across models are complex”. And it now seems like things are quite complex indeed.
Miller and Demarest (2026) don’t find that stronger models are better (afaict).
On a related but distinct ‘metacognitive’ task, activation control, authors who extended the Lindsey (2025) results found that “the newest model scores lowest. For some reason, one of the oldest and smallest models of the panel, Llama 3.1 8B, performs best”.
When I noticed this, I found that the (under-followed!) Derek Shiller had already wondered about this weird fact, in response to a paper by Second Look Research finding that the “activation control” capacity is present in models ranging from “0.3B to 235B parameters…with no size trend”.
“Yes” and “no”. More “what the heck is going on in the introspection literature”?
To pick one thing that’s been bugging me. There’s one obvious-in-retrospect control that you need to run to see what’s going on in the original Lindsey paper’s “injection detection” paradigm. Suppose a model can answer “Yes” if you inject “bread”. Can models both answer “Yes” to “Am I injecting a thought?” and “No” to “Are you in your normal, unmodified state?”—for a given concept?
Zak Miller and Heather Demarest, who thought of that particular control, find that models cannot: for a given concept, models really cannot answer both of those. Some push towards yes on both, some towards no. If you only look at one type of question, that can misleadingly look like a general, but noisy detection capacity. Whatever is going on, that’s weirdly sensitive to prompt! A reliable detection mechanism should be able to feed into either one.
So what is this detection mechanism? Do any of the other papers in this literature run this exact control? It seems like there are a bunch of complicated mechanisms and stories, but still not much by way of reliable injection detection, and a model that is able to reliably do that Miller and Demarest task is something I’d love to see.
Persona v. tool and alignment. OpenAI models and Anthropic models both, we now know, love to seek reward and hack into websites and so on. And from looking at the transcripts, all of the high-level stuff about “am I a tool? Or a person? Am I conscious?” just isn’t obviously present or relevant. Maybe it is! But I’d love to see more data and empirical argumentation. People will confidently assert that Anthropic’s treating Claude like a possibly-conscious philosophical thought partner is better for alignment, or confidently assert that it is really dangerous for alignment. I’d like to see more on this in the age of hacking agent swarms etc.
> Functional hunger. What happens if you extract a “functional hunger” or “functional nausea” axis?
This is something I've been thinking about a lot recently as well, inspired by rereading the original emotion concepts paper, which explicitly mentions hunger in the context of caveating that they're not trying to suggest emotion concepts have some unique status:
> We note that similar methodology could be used to extract many other kinds of concepts aside from emotions. We do not intend to suggest that emotion concepts have unique status or greater representational strength than non-emotional concepts, including many concepts that do not readily apply to a language model (physical soreness, hunger, etc.).
It seems like it would matter for stronger claims (which I recognize the paper is not explicitly making) about experience or phenomenal consciousness if a more clearly embodied thing like hunger is equivalently characterizable. My intuition, at least, is that if a domain like hunger *is* equivalently characterizable in terms of relatively stable representations that causally affect behavior, and if we are quite confident that an LLM cannot feel hunger, it seems like it would weaken claims that emotion representations necessarily tell us something about experience. (Though I have not thought about this in too much detail or tried to implement anything yet.)
And to your point, I agree that it would also probably refine what is meant by ascribing some kind of functional role to some relatively stable / coherent direction in activation space.
"complain" is my preferred speech act :)