LLMs and Agency
In which we take mind blowing remarks around LLMs and try to expand on them in reference to the System1 vs. System2 thinking dichotomy
A few days ago I stumbled upon this post named “18 Remarks on GPT” from an anonymous author that goes by the pen name of “Cleo Nardo”. I’m going to try to expand on them here as a way to organize my own thoughts that were triggered by some of them.
In my previous post, I mentioned how I resonate with the notion that “System2” thinking is the frontier of the possible in terms of AGI capabilities. In my own experience dealing with GTP4 for coding assistance, my perception is that it is highly capable in terms of breath but it’s somewhat lacking in terms of depth. Could this be because GPT4 is lacking in System2 thinking?
These 18 remarks offer some very interesting thoughts to stimulate our mind around this topic.
Static vs. Dynamic GPT
Remark 2 states that GPT does two separate things:
generate the next token from the current context and append it to it
deletes the first token
This is necessary because of the “finite” length of the context that the transformer is able to work with, but the author makes a very interesting argument that while #1 consists of a single forward pass thru a neural network, it’s #2 that creates a way for it to be agentic. It’s deleting the tokens that makes it different. Dare I say, intelligent? The author claims that it’s the coupling of these two actions that creates the feedback loop that generates agency.
I’m not sure I buy the argument that a GPT composed of these two steps is Turing complete: a Turing machine has an infinite tape while GPTs have a very specific limit to the number of states they can encode. Remark 15 which give us the evocative comparison of GPT4 and a Commodore 64 in terms of capacity of states that it’s able to encode.
At the same time, the notion that it’s not the neural network alone that imbues agency but it’s the combination of the neural network and additional “feedback loops” around it very much resonates with me and it’s extremely thought provoking that something as simple as a “FIFO” treatment of context can grant agency abilities to a system. It makes me wonder how many other things that we associate with AGI are hiding in plain sight.
There is also another important aspect here: GPT2 and GPT3 had been available for a while but what really took off was ChatGPT. My intuition at the time was that ChatGPT was better because of fine tuning but there is another aspect of agentic dynamism that is also hiding in plain sight: the context is augmented both by the LLM and by the user’s brain.
Most people approach a ChatGPT prompt the same way they approach a search engine query, with a default “oracle” posture. I have a problem, give me the solution. In my experience, this is a very inefficient and unproductive way of using ChatGPT which seems to shine as a “muse” or “rubber duck”. A way to write down and communicate your thinking so that it’s clearer to yourself as well.
ChatGPT is a fantastic interactive rubber duck, way better than humans for that because it’s indefatigable, it has RTFM and won’t make us feel bad if leak our “noviceness” by asking very basic questions.
But now we have 4 different processes in a modern chatbot like ChatGPT Pro:
the forward pass next token generator
the instruction interpreter that can augment context with information retrieval
the FIFO context pruner
the human user appending their own tokens to the context
This is a FAR more complex system than just an auto-regressive GPT. In fact, we can’t really predict ahead of time what happens in #2 and we don’t even fully understand what goes on in step #4. This makes AI alignment of these chatbots such a difficult problem.
LLMs as Bayesian Networks
In a previous life, I worked on symbolic knowledge representation (first at MIT, then at Metaweb, then at Google) the result of that work was Google’s Knowledge Graph, a very large collection of uniquely identifiable entities stitched together in a very large graph of Subject -(predicate)→ Object triples.
One of the things that became clear to me while working on knowledge graphs was that triples needed a “degree of belief” value (these are known as bayesian networks). It wasn’t possible to encode “common sense” without it. The Google’s KG ultimately is the manifestation of all the triples that can be modeled with “high confidence”. Aside from the giant problem of defining and measuring this confidence without introducing bias, the result is a dramatically smaller dataset, one that ends up encoding only but the most superficial “facts” about the world.
This was not a problem for Google Search because it was a huge improvement over not having it at all and provided an epistemic skeleton against which to map all sorts of things (from web pages, to user queries, to content in videos and images, to ads targets). But it became an issue for trying to model reasoning and artificial intelligence of a world “common sense” the same way Cyc attempted to do.
Remark 4.B suggests that a GPT can be interpreted as a very large sparse bayesian network. For GPT3, it’s a graph of 109628 nodes. To give you a sense of how large this is, the number of atoms in the universe is estimated to be 1082
Each “node” in this graph is a context. The relationships encode the probabilities that each given context will transition to another context. It is fascinating to me to think of an LLM this way because it makes me wonder a whole lot of things:
could we emerge a “common sense” knowledge graph from the weights of pretrained LLMs?
could this be a way to gain interpretability of LLMs?
could this be a way to compress them down by finding and removing all the “facts” that they have collected during training that occupy the LLM capacity but could be retrieved when needed instead?
Granted, these graphs are truly enormous in size and from experience with trying to interpret visual artificial neurons, it is to be expected that these entities emerged organically by the training process of a transformer by backpropagation and loss function minimization could be extremely difficult to comprehend or to map to logical concepts (interesting are also Chris Olah’s thoughts on the interpretability of LLMs).
Could we fine tune the model using alignment with existing knowledge graphs as the grounding function? I find this line of inquiry extremely fascinating and Remark 7 seems to be orbiting the same attractor.
Yes, I’m fully aware that this might be a rehashing of good old “connectionism vs. computationalism” debate (also read this for a take with the right amount of spice added to it) although I am not interested in theory or academic dogma here but rather in utility of the tools for human assistance and augmentation.