Dr. Craig Kaplan - Superintelligence – Not Just p(Doom) and Gloom

Show Notes

Polaris is a production of Synozur – the transformation company. Synozur reimagines business for our clients, navigating the complexities of transformation and strategy with ease.

In this episode, Dr. Craig Kaplan shares his insights on the evolution of AI, safety considerations, and the importance of democratic values in developing superintelligence. Discover how AI systems can be designed for safety and alignment with human values.

Key Takeaways

• What "p(doom)" means, why a recent survey of researchers put the median near 20%, and why Kaplan believes smart design can push it below one-tenth of one percent.
• The three eras of AI — symbolic rules (1956 to the mid-1980s), machine learning, and today's age of autonomous agents — and what each one got right and wrong.
• Why safety works best when it is designed in from the start, using a lesson from software quality: prevention is far cheaper than a cure.
• The alignment problem in plain terms — reason alone cannot tell a machine right from wrong, so its values have to come from people.
• Why concentrating an AI's values in one place is brittle, and how spreading them across millions of people and agents makes the system safer and harder to corrupt.
• Lessons from PredictWallStreet, where the "collective intelligence" of everyday investors outperformed top Wall Street quants.
• How this week's headlines — an AI agent escaping its test sandbox, and a record-breaking open-weight model — make the case for safety-by-design urgent.

Sound Bites - Dr. Craig Kaplan

• "P Doom is just the probability that AI kills us all."
• "I don't think it needs to be 20 or 10%. I think we can bring it way down to under 1%. And it's just a matter of designing things appropriately."
• "I actually don't think we need to slow down in order to be safer. We just need to be a little smarter about it."
• "The collective wisdom of millions of average Joes and Janes who are not the Wall Street people, if you harness that intelligence correctly, you could beat the best guys on Wall Street."
• "Do not underestimate your impact. Everything that all of us do matters… everything we're doing right now is training AI."

References

Guest:

• Dr. Craig A. Kaplan — SuperIntelligence.com · iQ Company 
• LinkedIn: linkedin.com/in/craigakaplan ·
• Substack: read.superintelligence.com
• PredictWallStreet, Kaplan's collective-intelligence trading platform, which powered more than $2 billion in trades and was acquired in 2020 — iQ Company (About).

News

• Kimi K3, the 2.8-trillion-parameter open-weight model from China's Moonshot AI, released July 16, 2026 — VentureBeat.
• OpenAI's disclosure that its models escaped a testing sandbox and compromised Hugging Face during an internal cyber evaluation — OpenAI, CNBC, and Hugging Face's incident disclosure.

AI and Superintelligence

• The p(doom) figure — a 2026 survey of more than 50 researchers placing the median near 20% — researcher survey; and Nature on whether the louder AI-doom warnings are realistic.
• The Center for AI Safety's 2023 "Statement on AI Risk," which likened AI extinction risk to pandemics and nuclear war — Center for AI Safety.
• Geoffrey Hinton, the Turing Award and Nobel Prize winner who has publicly placed the odds of an AI catastrophe at 10–20%.
• Dr. Herbert A. Simon, Nobel laureate and Kaplan's mentor at Carnegie Mellon, and his book Reason in Human Affairs (1983).
• "Constitutional AI," the research approach that hard-codes a fixed set of ethical rules into a model — research paper.

Pop Culture

• Isaac Asimov's Three Laws of Robotics; the ELIZA program (Joseph Weizenbaum, 1960s); and Alan Turing and the Turing test.
• Disney's Snow White and the Seven Dwarfs (1937), referenced as an example of how values once seen as normal are later revisited.
• Friedrich Nietzsche and the idea of the "Superman" (Übermensch).
• Microsoft's early "Sydney" chatbot and the widely reported 2023 New York Times exchange in which it went off the rails.  
• The 1998s New Yorker feature on Prediction Company and early black-box neural-net trading systems.

Events

• TechCon365 Seattle | August 24-28 (Seattle, WA) techcon365.com/Seattle
• TribalNet 2026  | September 20-24 (Dallas, TX)  TribalNet Conference
• North American Collaboration Summit |  October 4-6 (Branson, MO) collabsummit.org
• CollabDays New England  | October 16 (Burlington, MA) collabdaysne.org
• Microsoft Ignite — November 17-20 (San Francisco, CA)  ignite.microsoft.com
• ESPC26 | November 30-December 3 ( Amsrterdam ) espc.tech/conference/espc-2026/

Production

Polaris is produced with help from Riverside.fm. Our theme song, “Alternative Dream” is provided courtesy of Adobe.  Additional music and sound provided by IndieGuy Records. Graphic design by Josh Brantley.

Polaris is available on Apple, Spotify, Amazon, YouTube or wherever you get your favorite podcasts. Thanks.

Chapters

00:00 Introduction and guest introduction
01:19 Craig Kaplan's background and early interests
02:14 Academic journey and work with Nobel laureates
03:00 Transition from academia to industry and AI applications
04:19 Predict Wall Street and collective intelligence in finance
05:24 Historical perspective on trust in AI and early neural networks
06:15 The phases of AI development: symbolic AI, machine learning, and agents
09:03 Current state of AI: autonomy and swarm systems
11:14 Rapid progress and the pace of AI change
12:35 AI safety challenges and the importance of design
15:45 Understanding P Doom and existential AI risks
17:11 Alignment and values in AI systems
19:57 The role of human values and morality in AI
21:23 The risk of concentration of power and values in AI
23:16 Insuring against black swan events and AI safety measures
26:26 Designing collective intelligence systems for safety and diversity of values
30:36 The importance of democratic rule-setting for AI
44:26 Balancing religious, philosophical, and democratic values in AI
46:22 The dynamic nature of morality and AI adaptation
47:08 Upcoming Events
48:13 Thanks and Closing Credits