loader from loading.io

How to Move from Prompt Engineering to Harness Engineering in Testing with Matt Wynne

TestGuild Automation Testing Podcast

Release Date: 07/14/2026

Testing AI: Engineering Confidence in Non-Deterministic Systems with Jason Arbon show art Testing AI: Engineering Confidence in Non-Deterministic Systems with Jason Arbon

TestGuild Automation Testing Podcast

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now:  Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on...

info_outline
Playwright With AI: How to Automate Tests Without Shipping AI Slop with Andrew Knight show art Playwright With AI: How to Automate Tests Without Shipping AI Slop with Andrew Knight

TestGuild Automation Testing Podcast

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow? In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright. You will discover: How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents,...

info_outline
Test Automation Won't Save Your QA Career, but These Skills Will with Keith Klain show art Test Automation Won't Save Your QA Career, but These Skills Will with Keith Klain

TestGuild Automation Testing Podcast

Keith Klain has spent 25+ years leading enterprise quality programs in financial services and is one of the testing industry's most respected voices. In this episode, he joins Joe to discuss his BrowserStack talk, "The Death of Test Engineering," why testing isn't dying—but outdated testing roles are—and how AI is reshaping the future of software quality. In this episode: Why "The Death of Test Engineering" isn't what you think The difference between checking and testing Will AI replace software testers? Why confidence engineering matters How to communicate testing in business terms Where...

info_outline
AI Testing Strategy: Stop Being a Cost Center, Start Protecting Revenue with Nandini Srinivasan show art AI Testing Strategy: Stop Being a Cost Center, Start Protecting Revenue with Nandini Srinivasan

TestGuild Automation Testing Podcast

Nandini Srinivasan has spent 25 years in the quality industry and leads a global QA organization of over 150 engineers across the US, Canada, India, and Pakistan. In this episode, she breaks down exactly how she built a QA AI acceleration charter, ran a train-the-trainer model, and used a phased proof-of-concept approach to separate real AI from what she calls "powerful automation dressed up as AI.  We get into: How she frames quality metrics for executives Using language around revenue protection Risk mitigation Feature velocity instead of test coverage percentages. She talks about the...

info_outline
How to Move from Prompt Engineering to Harness Engineering in Testing with Matt Wynne show art How to Move from Prompt Engineering to Harness Engineering in Testing with Matt Wynne

TestGuild Automation Testing Podcast

Matt Wynne, co-creator of Cucumber and BDD practitioner, joins Joe for the first time in over a decade to talk about what two years inside a Silicon Valley AI startup taught him about the future of software testing. Matt spent time at Mechanical Orchard working alongside experienced XP practitioners to modernize legacy COBOL mainframes using LLMs, and then spent a week with the team that coined the term "software factory," where the rule was simple: humans never write the code, never read the code. In this episode, Matt breaks down what harness engineering actually means, why shared...

info_outline
Agentic Engineering for Testers: How to Automate Your Way to the Top with Amit Rawat show art Agentic Engineering for Testers: How to Automate Your Way to the Top with Amit Rawat

TestGuild Automation Testing Podcast

Amit Rawat is an agentic engineer who spent two decades in QA before shifting fully into building AI agents. He's the creator of PromptWright, a desktop tool that turns natural language prompts into automated Playwright browser tests, complete with screen recording, Gherkin scenario generation, and self-healing locators. In this episode, Amit and Joe get into what it actually takes to work with AI agents at a high level, starting with why the planning phase matters more than the prompt itself. Amit breaks down his own workflow, brainstorming with AI, building a detailed plan in HTML before...

info_outline
How to Test Any API Without Documentation with Liudas Jankauskas show art How to Test Any API Without Documentation with Liudas Jankauskas

TestGuild Automation Testing Podcast

Most API testing stops at the happy path. The problem is that the bugs that actually hurt you in production are sitting in everything many testers skip, like the boundary values, the oversized payloads, the missing tokens, the security headers, the inputs that make no sense at all. In this episode, Joe sits down with Liudas Jankauskas, who has spent almost twenty years breaking software and testing APIs since 2008. Liudas demonstrates Rentgen, his free and open-source API testing tool, live on screen. You'll watch him take a single request from a real app, map it in seconds, and generate...

info_outline
Your AI Code Review Is Lying to You (Here's the Fix) with Evan Marshall show art Your AI Code Review Is Lying to You (Here's the Fix) with Evan Marshall

TestGuild Automation Testing Podcast

Your AI code review tools read the diff. They stare at your code. But they never actually run it. So the bugs that only show up at runtime, the broken user flows, the bad query plan, the duplicate submission, sail right past review and land in front of your customers. In this episode, Joe Colantonio sits down with Evan Marshall, founder of Ito and a fifteen year engineer who spent five years in applied cryptography securing hundreds of millions of dollars for millions of people. Evan is taking that ship fast without breaking things discipline and pointing it straight at testing. Ito is an...

info_outline
AI QA Agents Explained: How Amikoo Helps Testers with Ivan Barajas Vargas show art AI QA Agents Explained: How Amikoo Helps Testers with Ivan Barajas Vargas

TestGuild Automation Testing Podcast

What happens to QA when AI is writing ten times more code than your team can test? That is the exact problem Ivan Barajas Vargas set out to solve with Amikoo, a purpose-built AI QA agent designed to help testers, SDETs, and even developers move faster without sacrificing coverage or quality. Ivan is no stranger to AI in testing. Before generative AI became mainstream, he co-founded MuukTest, a test automation platform built on symbolic reasoning and expert systems. After six years and thousands of customer conversations, he went back to first principles to build Amikoo from scratch, this time...

info_outline
The AI Illusion: Why Testing Still Needs Humans with Joe Colantonio show art The AI Illusion: Why Testing Still Needs Humans with Joe Colantonio

TestGuild Automation Testing Podcast

Everyone is talking about AI replacing testers, writing tests, and transforming software quality. But what if we're asking the wrong question? In this solo episode, Joe Colantonio shares a growing concern he's seen while traveling across the country for TestGuild IRL events: a decline in testing fundamentals at the exact moment AI hype is reaching a fever pitch. Drawing insights from Carissa Véliz's book Prophecy: Prediction, Power, and the Fight for the Future, Wayne Roseberry's work on AI and meaning, and Tariq King's concept of Human Experience Testing, Joe explores why AI systems may be...

info_outline
 
More Episodes

Matt Wynne, co-creator of Cucumber and BDD practitioner, joins Joe for the first time in over a decade to talk about what two years inside a Silicon Valley AI startup taught him about the future of software testing.

Matt spent time at Mechanical Orchard working alongside experienced XP practitioners to modernize legacy COBOL mainframes using LLMs, and then spent a week with the team that coined the term "software factory," where the rule was simple: humans never write the code, never read the code.

In this episode, Matt breaks down what harness engineering actually means, why shared understanding is still the real bottleneck even in an agentic world, and how testers can use multiple LLMs to review AI-generated pull requests without reading every line.

He also gets honest about the grief that comes with realizing you can encode years of hard-won expertise into a Markdown file, and why that does not mean your skills are worthless.

If you are working in a brownfield codebase, wondering how to handle the flood of agentic PRs, or trying to figure out where testers fit in a world where agents write the code, this conversation is worth your time.

Find Matt at:

  • mattwynne.net
  • leansoftware.ai

Also check out his course:
Build a Software Factory: Hands-off agentic coding for experienced engineers
https://testgld.link/mattcourse