AI in Higher Education: Bias, Procurement, and Human Oversight

At sixteen, with straight A's in math and science, Dr. Karen Panetta's school career assessment told her to sell makeup or be a cook. A male friend with lower scores got engineer or politician. No AI was involved. Just a rules-based system applying gender and biographical filters to two teenagers. That same logic now sits inside AI tools landing in admissions offices and HR systems across higher ed, with one critical difference: AI does not eliminate human bias, it removes the human accountability that used to make bias correctable.

In this episode of the Changing Higher Ed® podcast, Dr. Drumm McNaughton speaks with Dr. Karen Panetta, Dean of Graduate Education for the School of Engineering at Tufts University and an IEEE Fellow.

Panetta lays out a procurement framework presidents and boards can use to evaluate AI tools before signing a contract. She and McNaughton work through the four questions most vendors cannot answer, why IRB principles already give higher ed a working framework for AI, and what happens to graduate research when students ask AI for a unique contribution and accept whatever comes back.

This conversation is especially relevant for institutional leaders making decisions about AI procurement, classroom adoption, and data governance who want a clear set of questions to ask before they buy and a clear standard for keeping humans accountable for the decisions AI tools are increasingly being asked to make.

Topics Covered:

• The four procurement questions every higher ed leader should ask before signing an AI contract
• Why expert disagreement on ground truth limits what any AI tool trained on that judgment can do
• How IRB principles apply to AI deployments, and why every kind use of technology has a misuse case sitting next to it
• The risk of AI's interpretation of truth aging with the consensus
• Why faculty in English, history, and the arts are essential to AI policy
• What IEEE's 500,000 technical professionals are doing on AI standards that no single corporate vendor will do

Real-World Examples Discussed:

• The career assessments that pointed a top math student toward cooking and a Navy veteran toward forest ranger work
• A cancer detection project where six doctors agreed on whether something was cancer but disagreed on every grade beyond that
• A conservation project where the same tracking data that helps park rangers could help poachers if security is weak
• Graduate admissions committees where different faculty weight credentials, projects, and volunteer work differently, and what gets lost when only one set of weights is encoded into an AI screen

Three Key Takeaways for Leadership:

1. AI does not create bias. It scales whatever bias is already in the institution's decision systems, at the speed and volume the institution chose to deploy.
2. Every consequential decision needs a human in the loop who can explain the call out loud. Without that, the institution cannot defend the decisions it is making.
3. The sticker price on the AI tool is not the story. The data behind it is, and most vendors cannot tell you what it is.

This episode gives presidents, provosts, and boards a practical framework for AI procurement and governance, along with a clear answer to the trustee asking why the institution has not bought what everyone else is buying.

Read the transcript: https://changinghighered.com/ai-bias-procurement-framework-higher-education/

#HigherEducation #AIinHigherEd #HigherEducationPodcast #AIGovernance #AIBias #HigherEducationLeadership