15 - Natural Abstractions with John Wentworth
AXRP - the AI X-risk Research Podcast
Release Date: 05/23/2022
38.1 - Alan Chan on Agent Infrastructure
AXRP - the AI X-risk Research Podcast
Road lines, street lights, and licence plates are examples of infrastructure used to ensure that roads operate smoothly. In this episode, Alan Chan talks about using similar interventions to help avoid bad outcomes from the deployment of AI agents. Patreon: Ko-fi: The transcript: FAR.AI: FAR.AI on X (aka Twitter): FAR.AI on YouTube: The Alignment Workshop: Topics we discuss, and timestamps: 01:02 - How the Alignment Workshop is 01:32 - Agent infrastructure 04:57 - Why agent infrastructure 07:54 - A trichotomy of agent infrastructure 13:59 - Agent IDs 18:17 - Agent channels...
info_outline
38.0 - Zhijing Jin on LLMs, Causality, and Multi-Agent Systems
AXRP - the AI X-risk Research Podcast
Do language models understand the causal structure of the world, or do they merely note correlations? And what happens when you build a big AI society out of them? In this brief episode, recorded at the Bay Area Alignment Workshop, I chat with Zhijing Jin about her research on these questions. Patreon: Ko-fi: The transcript: FAR.AI: FAR.AI on X (aka Twitter): FAR.AI on YouTube: The Alignment Workshop: Topics we discuss, and timestamps: 00:35 - How the Alignment Workshop is 00:47 - How Zhijing got interested in causality and natural language processing 03:14 - Causality and...
info_outline
37 - Jaime Sevilla on AI Forecasting
AXRP - the AI X-risk Research Podcast
Epoch AI is the premier organization that tracks the trajectory of AI - how much compute is used, the role of algorithmic improvements, the growth in data used, and when the above trends might hit an end. In this episode, I speak with the director of Epoch AI, Jaime Sevilla, about how compute, data, and algorithmic improvements are impacting AI, and whether continuing to scale can get us AGI. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:38 - The pace of AI progress 0:07:49 - How Epoch AI tracks AI compute 0:11:44 - Why does AI compute grow so smoothly?...
info_outline
36 - Adam Shai and Paul Riechers on Computational Mechanics
AXRP - the AI X-risk Research Podcast
Sometimes, people talk about transformers as having "world models" as a result of being trained to predict text data on the internet. But what does this even mean? In this episode, I talk with Adam Shai and Paul Riechers about their work applying computational mechanics, a sub-field of physics studying how to predict random processes, to neural networks. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:42 - What computational mechanics is 0:29:49 - Computational mechanics vs other approaches 0:36:16 - What world models are 0:48:41 - Fractals 0:57:43 - How the...
info_outline
New Patreon tiers + MATS applications
AXRP - the AI X-risk Research Podcast
Patreon: MATS: Note: I'm employed by MATS, but they're not paying me to make this video.
info_outline
35 - Peter Hase on LLM Beliefs and Easy-to-Hard Generalization
AXRP - the AI X-risk Research Podcast
How do we figure out what large language models believe? In fact, do they even have beliefs? Do those beliefs have locations, and if so, can we edit those locations to change the beliefs? Also, how are we going to get AI to perform tasks so hard that we can't figure out if they succeeded at them? In this episode, I chat with Peter Hase about his research into these questions. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:36 - NLP and interpretability 0:10:20 - Interpretability lessons 0:32:22 - Belief interpretability 1:00:12 - Localizing and editing models'...
info_outline
34 - AI Evaluations with Beth Barnes
AXRP - the AI X-risk Research Podcast
How can we figure out if AIs are capable enough to pose a threat to humans? When should we make a big effort to mitigate risks of catastrophic AI misbehaviour? In this episode, I chat with Beth Barnes, founder of and head of research at METR, about these questions and more. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:37 - What is METR? 0:02:44 - What is an "eval"? 0:14:42 - How good are evals? 0:37:25 - Are models showing their full capabilities? 0:53:25 - Evaluating alignment 1:01:38 - Existential safety methodology 1:12:13 - Threat models and capability...
info_outline
33 - RLHF Problems with Scott Emmons
AXRP - the AI X-risk Research Podcast
Reinforcement Learning from Human Feedback, or RLHF, is one of the main ways that makers of large language models make them 'aligned'. But people have long noted that there are difficulties with this approach when the models are smarter than the humans providing feedback. In this episode, I talk with Scott Emmons about his work categorizing the problems that can show up in this setting. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:33 - Deceptive inflation 0:17:56 - Overjustification 0:32:48 - Bounded human rationality 0:50:46 - Avoiding these problems 1:14:13 -...
info_outline
32 - Understanding Agency with Jan Kulveit
AXRP - the AI X-risk Research Podcast
What's the difference between a large language model and the human brain? And what's wrong with our theories of agency? In this episode, I chat about these questions with Jan Kulveit, who leads the Alignment of Complex Systems research group. Patreon: Ko-fi: The transcript: Topics we discuss, and timestamps: 0:00:47 - What is active inference? 0:15:14 - Preferences in active inference 0:31:33 - Action vs perception in active inference 0:46:07 - Feedback loops 1:01:32 - Active inference vs LLMs 1:12:04 - Hierarchical agency 1:58:28 - The Alignment of Complex Systems group Website of...
info_outline
31 - Singular Learning Theory with Daniel Murfet
AXRP - the AI X-risk Research Podcast
What's going on with deep learning? What sorts of models get learned, and what are the learning dynamics? Singular learning theory is a theory of Bayesian statistics broad enough in scope to encompass deep neural networks that may help answer these questions. In this episode, I speak with Daniel Murfet about this research program and what it tells us. Patreon: Ko-fi: Topics we discuss, and timestamps: 0:00:26 - What is singular learning theory? 0:16:00 - Phase transitions 0:35:12 - Estimating the local learning coefficient 0:44:37 - Singular learning theory and generalization 1:00:39 -...
info_outlineWhy does anybody care about natural abstractions? Do they somehow relate to math, or value learning? How do E. coli bacteria find sources of sugar? All these questions and more will be answered in this interview with John Wentworth, where we talk about his research plan of understanding agency via natural abstractions. Topics we discuss, and timestamps:
- 00:00:31 - Agency in E. Coli
- 00:04:59 - Agency in financial markets
- 00:08:44 - Inferring agency in real-world systems
- 00:16:11 - Selection theorems
- 00:20:22 - Abstraction and natural abstractions
- 00:32:42 - Information at a distance
- 00:39:20 - Why the natural abstraction hypothesis matters
- 00:44:48 - Unnatural abstractions used by humans?
- 00:49:11 - Probability, determinism, and abstraction
- 00:52:58 - Whence probabilities in deterministic universes?
- 01:02:37 - Abstraction and maximum entropy distributions
- 01:07:39 - Natural abstractions and impact
- 01:08:50 - Learning human values
- 01:20:47 - The shape of the research landscape
- 01:34:59 - Following John's work
The transcript: axrp.net/episode/2022/05/23/episode-15-natural-abstractions-john-wentworth.html
John on LessWrong: lesswrong.com/users/johnswentworth
Research that we discuss:
- Alignment by default - contains the natural abstraction hypothesis: alignmentforum.org/posts/Nwgdq6kHke5LY692J/alignment-by-default#Unsupervised__Natural_Abstractions
- The telephone theorem: alignmentforum.org/posts/jJf4FrfiQdDGg7uco/information-at-a-distance-is-mediated-by-deterministic
- Generalizing Koopman-Pitman-Darmois: alignmentforum.org/posts/tGCyRQigGoqA4oSRo/generalizing-koopman-pitman-darmois
- The plan: alignmentforum.org/posts/3L46WGauGpr7nYubu/the-plan
- Understanding deep learning requires rethinking generalization - deep learning can fit random data: arxiv.org/abs/1611.03530
- A closer look at memorization in deep networks - deep learning learns before memorizing: arxiv.org/abs/1706.05394
- Zero-shot coordination: arxiv.org/abs/2003.02979
- A new formalism, method, and open issues for zero-shot coordination: arxiv.org/abs/2106.06613
- Conservative agency via attainable utility preservation: arxiv.org/abs/1902.09725
- Corrigibility: intelligence.org/files/Corrigibility.pdf
Errata:
- E. coli has ~4,400 genes, not 30,000.
- A typical adult human body has thousands of moles of water in it, and therefore must consist of well more than 10 moles total.