Define the Problem: Path for Improvement (Part 5)

Before you start solving a problem, you need to know what, precisely, you're trying to solve. In this episode, John Dues and Andrew Stotz talk about how to figure out the problem on which you will focus your team's efforts.

TRANSCRIPT

0:00:02.2 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is, Define The Problem. John, take it away.

 

0:00:22.9 John Dues: Hey, Andrew. It's good to be back. Yeah. So, we've been going through this improvement model just as a refresher. Two episodes ago we looked at the three phases of performance measurement, research, accountability, and improvement. That was an important frame at the start of this process. And then in the last episode we discussed how we made sure in that first step of the model, we step back and see our system, see the full system in which we work. And I showed you some improvement tools that we use to visualize the improvement team or the group's thinking for each of a set of guiding questions. So, we looked at a system flow chart and that sort of said to us, within our system, kind of saw how things flow into the system, the things that we do, and then the outputs, and the questions there were, what's the largest system to improve on and what's the aim of that system? And then we took a look at an affinity diagram, and we used that just to answer the question, what are the opportunities for improvement within the target system? And then we used this tool called an interrelationship digraph, where we prioritize basically the various opportunities for improvement. And then it's been through this model that everybody can see these four steps that we're talking...

 

0:01:57.7 AS: Well, for the listeners, they may not be able to see, but for the viewers, yeah.

 

0:02:01.7 JD: For the listeners, they can't see it. But the model that we've been talking through, for sure, we've been working through this four-step improvement model, and we've spent most of our time on step one, which is set the challenge and direction. And we'll remain here at this point in the process today as well. And then later on in the series, we'll go on to the subsequent steps. So step two, grasp the current condition. Step three, establish your next target condition. Step four, experiment to overcome obstacles. And some of the first episodes in the series, we talked through just a high level overview of each of those things. And then we've also said that we wanna do all of these steps with a team that's made up of somebody that has Profound Knowledge, some number of people that have the authority to work or change or redesign the system, and in some number of people that are working in the system.

 

0:02:57.3 JD: So that's just kind of a refresher for those that have been following along. And again, in step one of this model, this is where we're at right now, we ask, where do we wanna be in the long run? And so we're really thinking about a longer range goal that will differentiate us from other organizations. So in our case, schools. And it seems nearly impossible at the outset. We've said that. And we've also talked about what's the right time period for this challenge or direction to be set for, in terms of out in the horizon? Then what I said was, somewhere in the neighborhood of six months to three years, sort of anything less than six months, it's just too fast to put the team together and really dive in and do the work that you need to do. Anything beyond three years, it just seems so far that, you now, things can kind of get away from you if you set the vision out that far.

 

0:03:48.8 JD: Not that it's impossible, but six months to three years seems to be a sweet spot in my perspective. And then I gave this example, we're working on this chronic absenteeism problem. Chronic absenteeism is when a student in a K-12 school is absent for more than 10% of the school year, and coming out of the pandemic, we've talked about a very high number of kids across the country in the United States are chronically absent. And in our particular system, like a lot of high poverty systems, those numbers are particularly stark. So over the last few years since the pandemic, the chronic absenteeism rate in our school system has been hovering right around 50%.

 

0:04:34.3 AS: It's just so incredible, every time I hear you say that, I just can't believe that.

 

0:04:38.3 JD: It is incredible, incredible. And we're trying to get that down, that number down to 5%. So it'd be a huge...

 

0:04:44.0 AS: Which is also an incredible stretch goal.

 

0:04:47.3 JD: Incredible stretch goal. Seems almost impossible. That's sort of how we've framed things in our school system.

 

0:04:54.4 AS: So let's stop there just for a second, because I think for the listeners and the viewers, what's your long range? Let's take three years. What is your three-year goal that is nearly impossible? Where do you wanna be? Yeah, I liked it the way, you know, the diagram that you're showing is kind of a mountain, and so why not think, what mountain do you wanna plant your flag on three years from now? And that really is what you're describing, what you guys are focused on is a very challenging goal, but for the listener and the viewer, what's yours?

 

0:05:31.4 JD: Yeah. Yeah, that's a great point. Yeah, I mean, I think encouraging people to think through how they would step through this process, how they would frame goals within the system that they're working in, I think that's a really important sort of thought process to be going through as you're listening to, at least to our approach.

 

0:05:49.8 AS: Yep.

 

0:05:51.1 JD: So at this stage in the process, like I said, we've stepped back. We have this sort of long term goal. We've mapped out our system, we've talked about some opportunities for improvement and prioritize those. So at this stage in the process, the next thing we're gonna do is we're gonna define the problem. We know we have an issue with chronic absenteeism, but we don't necessarily know what the specific problem is that we're going to work on. So they're, like, with See the System, and we had a set of guiding questions, similarly, we have a set of guiding questions that we're asking at this step in the process to find the problem.

 

0:06:32.1 JD: So the one that we're gonna focus on today is how is the project being funneled from a general to a more specific problem? So again, we've got this general problem area, chronic absenteeism. Some other guiding questions that we'll tackle later, what's the precise problem statement? What are key operational definitions, concepts that we're gonna measure? Who will benefit the most from the improvement effort? What are their needs? And then what's the vision for excellence? So those are all the things that we're tackling in this stage that we're calling Define the Problem. But we're gonna focus just on this one question today, how is the project being funneled from a general to a more specific problem?

 

0:07:19.1 JD: So, in our case, and we talked about this last time, by this point we've formed an improvement team. So we have an actual group of about 10 people cross-disciplinary, meaning different roles that are, and they're coming from different parts of our system, different departments, all four of our campuses are represented. So this is the improvement team that we've formed to work on this particular problem. And really what we're trying to do in this step is show the importance of this particular attempted improvement. And we're trying to paint a picture for everybody else that's not on the team why energy should be spent here instead of elsewhere. Because there's gonna have to be resources both obviously in people, money, whatever, deployed as a part of this effort. And we have any number of problems like most school systems...

 

[laughter]

 

0:08:19.7 JD: That we could focus on. And so we have to really paint the case for why we're gonna focus on chronic absenteeism.

 

0:08:27.3 AS: I have to tell you a story, John. When I was, I don't know, 10 years ago I went to visit my first boss who was so successful that he bought a piece of land in New Zealand, and it was a farm. And it was 250 square kilometers. So I went to visit him and he had a little house on this big farm, and it had all these mountains. And he looked at me and said, what mountain do you want to climb today? And I said, "That one." And so we climbed up that one, and I was there for five days and we had to take a break in between, 'cause it's pretty exhausting climbing up one of these mountains, but there was endless mountains, and you can't climb 'em all.

 

0:09:03.2 JD: Yeah.

 

0:09:03.7 AS: And you only have limited time, and you only have limited energy. So what mountain are you gonna climb? And I think part of what you're talking about, showing the importance of spending the energy here is that, you know, hey, this is a very hard challenge that we want to get to, and we have to explain that we cannot climb two of these mountains at the same time, and we cannot climb all of these mountains over a period of time. We have to really be able to focus and make the argument of why we're planting our flag on that particular mountain.

 

0:09:35.5 JD: Yeah, absolutely. Absolutely. And in terms of the team, one of the things we talked about last week was that, or last episode is that, one of the ways that we use this set of guiding questions for each step in the process is, we're tagging that guiding question to a tool that we use to visualize the group's thinking. So again, we're funneling from this sort of general problem, this mountain to climb to a more specific mountain to climb, so to speak. So we'll keep doing that today. And again, what we're trying to get to is what's the precise problem statement? But we have sort of a general area of focus now, and we're thinking about, are we ready to write that more precise problem statement? Or perhaps is there more study that's needed? And one of the things that we do is we use this tool called a Problem Statement Readiness Check.

 

0:10:41.0 JD: And, so basically, so the group's been working for about a month and a half, and so we have quite a bit of information that we've gathered about this process thus far. And part of the power of this Problem Statement Readiness Check is it puts all of that information in one place, or at least the most important information. So you can kind of see at the top for folks that are viewing, there's this general problem focus area. And basically this captures the problem as we understand it early in the process. And basically what it says is that our average daily attendance across our four campuses is between 85% and 88%. And currently chronic absenteeism rates are between 43% and 53% depending on the campus. High number of students are excessively absent or habitually truant.

 

0:11:39.1 JD: So those are sort of officially defined attendance terms in Ohio's attendance laws, basically. And that we... We know that, you know, fairly obvious that missing a high amount of school, missing a lot of school is detrimental to students and their ability to grow and achieve academically. So it's, also when you have, you know, so students that are experiencing these problems require a lot of intervention. And so when you have high numbers, obviously it becomes harder to intervene because it's an intensive process. And when you're interviewing with chronically absent students, school leaders are being pulled away from other things. So that's sort of part of the case that we're making for this being a worthy problem. Although it's not very hard actually to paint this picture.

 

0:12:35.4 AS: Right.

 

0:12:35.5 JD: And so we have this focus area. Then what we say is, what insights into the problem focus area are helpful? What learning has the potential to lead to effective solutions? So this is stuff we've done so far. Again, we're kind of summarizing where we are to date. So the leader of this group is saying, well, we created this affinity diagram. We have these six categories of problems that we could focus our attention on: Data, academic systems, communication, transportation, culture and engagement, our intervention systems. And then we use that in a relationship digraph to try to come up with what's the dominant cause of the attendance problem? And if you remember from last time, what we settled on when we looked at that digraph was transportation. But then we said, well, we're not gonna focus on transportation because we already have an improvement team in our system that's working on the transportation issues that we're seeing here in Columbus.

 

0:13:40.8 JD: The other thing is we're looking at a bucket of questions that remain to be answered. So after we looked at of all the data that we've gathered so far, the conversations that we've had, what questions remain to be answered, what do we still need to know as a team? Maybe what new questions has the problem analysis revealed? So some of those questions are things like, why aren't students coming to school consistently? What does the intervention process look like at each campus? There's variation, we've already found out there's variation in the campus... How the campuses intervene. What does the typical attendance intervention plan look like at the individual student level? Is that intervention process at each campus effective? How do we know? Are we tracking data? So you can start to see how there are a lot of questions to sort of consider even after you've done some study digging into your data.

 

0:14:37.7 JD: So that's sort of the first part of the problem focus... Sorry, the Problem Statement Readiness Check. So this is a worksheet that we use. And the second part is we have a series of six questions that we ask after the group has talked through that first part of the worksheet. And basically they're just yes/no questions. They're six questions. So the first question is, has our team investigated multiple perspectives on the problem focus area? And basically what we said there is the team basically understands a higher percentage of students are not coming to school consistently, but the team doesn't have a strong understanding of what those causes are. So in that case we checked no, we haven't investigated multiple perspectives. The second question is, have you challenged assumptions our team held about why the problem occurs?

 

0:15:37.3 JD: So you can imagine anytime you're working in team with or without data, people are gonna have these preconceived notions, "Oh, I know why kids don't come to school." It's this reason or that reason, right? These are assumptions that we have. So in terms of challenging those assumptions with data, with study, we said, no, we haven't done that. The third question is, have you gained useful insight into why previous efforts haven't been as successful as desired? Basically there we said not systematically. We haven't studied this issue for a while, so we checked no there. The fifth question is, has your team gained sufficient insight into students' needs to give you confidence that you know which kinds of improvement will lead to improve student experiences and outcomes? To that we said no. We haven't done a lot of talking to students and families about specific challenges individual students are facing.

 

0:16:31.5 JD: And then the last question is, have you identified existing school-based practices or processes connected to the problem that might be improved? So that was the one question that we checked yes to. So basically there's this little guidance done at the bottom, then of the worksheet. It says, if the team checks three or more boxes, we'll move on to draft the problem statement. Well, we only draft... Sorry, we only checked one of the six boxes as a yes. The other five were no. So we're basically saying there, we don't have enough information to write the specific problem statement that we're gonna work on. So we have to do something else that's going to give us additional information about our process. So you can see how this thing has, you know, it's starting to be funneled from sort of a general understanding that we have a problem with chronic absenteeism, a problem with students attending school consistently.

 

0:17:29.8 JD: We think we know generally some of the reasons that's happening, but we haven't done enough study to really understand the specific obstacles that individual students are facing. So, then we, you know, since we're not ready to write the problem statement, we have to think about, well, what do we do to dig in and get sort of additional information? And so in our case, what we decided to do next was a Plan-Do-Study-Act cycle. So this is a way, if properly designed, where we can get some of that additional information from students and families. So that's what we're in the process of doing right now. And it's been really interesting. So what we did was, so we started the PDSA a couple weeks ago. This sort of coincided with the end of our first grading period.

 

0:18:22.9 JD: We're on trimester, so this was at the end of November. And we knew that parent conferences were coming up during that time. So that would give us a unique chance to talk to some of the students and families that are having attendance issues. So basically what we did was we said, well, we're gonna talk to one student and one family member from each campus that's been chronically absent. And we created a structure for the PDSA. So the plan was, so we have a Dean of Family and Community Engagement at each campus, and they're the point person for this attendance work. And so the guy that's leading this project, he's like the System of Profound Knowledge coach, the improvement coach on this, he created a little interview protocol, and basically then the DFCEs, the family engagement person at each campus took the protocol and sat down with the family member to interview them.

 

0:19:24.0 JD: And they're basically asking, you know, if the person is missing days of school, they're asking why, what's going on, what are the obstacles? Sometimes, because Ohio counts attendance by hours, so sometimes families are consistently bringing their student late, and those one and two hour misses in the morning add up to days pretty quickly, or you get the vice versa where someone is coming to pick the student up early repeatedly for some reason. And then of course some folks have a combination of these things. But what we said was, let's get some information from specific families. And we made a prediction, we said, you know, if we talk to people about the attendance issue, start to gather some information from them and let them know that, hey, your child, your student has this attendance issue, we all then made predictions for, is this gonna have an impact?

 

0:20:23.5 JD: Just this little intervention, is this gonna have an impact on the attendance rates? So we're in the middle of this, now we're gathering this data. It's been really fascinating to see the information that was gathered, to see, like, on a day by day level, the specific reasons, not the attendance codes that get put into the system, like drilled down specific data, specific anecdotes about what's happening on any given day that leads to a kid missing all or part of a school day. So this is sort of the first part in our process, PDSA cycle one is just to gather this information to see, you know, get a better sense, a detailed sense of some of the obstacles that our families are facing when it comes to attendance.

 

0:21:08.9 AS: Like, I got up and I went out and started the car and it didn't start.

 

0:21:15.4 JD: That's exactly right. That's exactly right. That's a...

 

0:21:19.6 AS: So I gave up...

 

0:21:21.8 JD: Oh, sorry, go ahead.

 

0:21:22.1 AS: So I gave up and I thought, I can't be bothered with this. I've gotta take care of other stuff and whatever.

 

0:21:28.7 JD: Yeah, it's stuff like that. And we've talked about this bus issue that we're having in Columbus. So it's not uncommon for a bus to come late, an hour late in the morning to a student bus stop.

 

[overlapping conversation]

 

0:21:39.4 AS: Do you sit there for an hour, wait for it, or, you know?

 

0:21:42.9 JD: Tomorrow it's gonna be something like 19 degrees in Ohio. Are you gonna sit out there for an hour when it's 19 degrees and wait for your bus? Right? And maybe you go home and you're right, and the car doesn't start. Or we had a family tell us that the bus stop that the kid is assigned to is dangerous, you know? And so you start to dig into these things and they're very reasonable explanations for attendance issues.

 

0:22:12.6 AS: Yeah.

 

0:22:13.2 JD: And so this is a way for us to sort of dig into some of those questions to help us better define the problem. And we're spending a lot of time on these early stages. And what I typically tell people is these early stages, before we ever develop a solution, account for at least 50% of any of these projects, because we wanna be very sure that when we start solutions that the problem is well understood, otherwise if there's misalignment between that problem identification and then the solution, then we're just wasting time and spinning our wheels.

 

0:22:50.7 AS: What do you say to people, like, "Come on, this is so much work. I mean, let's just solve the problem. Come on, John. You know, it's a problem. We know it's out there. Let's go, let's solve it"?

 

0:23:00.8 JD: Yeah. I mean, that's actually something that, I mean, there's that. And then when you're in an improvement team, there's, I have many other responsibilities that I need to attend to. And for a team like this that's cross campus, we have to have the meeting somewhere. So we pick a campus, and that means the people at the other three campuses have to travel to the meeting. So there are sort of hardships or at least extra work that's associated with this. But we just sort of keep reminding people like, this deep study is worth it. And the good thing with us, I think, is that we're a very mission-driven organization. That mission orientation is assessed during the interview process. So the vast majority of people that work at United are very bought in and very driven by that mission. And something like this is a very mission-aligned effort. So it's not like we're trying to improve some web traffic or something like that that may be harder to find, maybe important, but maybe harder to find meaning in. This is kids' futures, like improving this...

 

0:24:16.6 AS: People are generally aligned.

 

0:24:17.7 JD: Have a significant... What's that?

 

0:24:19.7 AS: People are generally aligned.

 

0:24:22.6 JD: People are generally aligned. That's right. Yeah. Yeah. So I'm really interested, like I said, we're in the middle of this PDSA, so I'm really interested in what the outcome is gonna be. We're gonna run this for about 20 days, basically until our winter break here in about a week and a half, gather the data, we're gonna use that to go back and then see if we're ready to write this problem statement, basically.

 

0:24:44.8 AS: So how would you summarize today's discussion?

 

0:24:49.0 JD: Yeah. If I was gonna kind of recap where we are, just sort of overall, I'd say the first thing is we, as an organization, senior leadership team, we set this chronic absenteeism as a key priority, right? So that was set at the organizational level, our senior organizational level, this overall challenge. And then in talking with the team, because we're telling them for the first time, hey, this goal is to go from 50-ish percent down to 5%, so it's really important, again, with that team that we frame this as an improvement orientation. This is not an accountability orientation. This isn't an individual person's fault if we don't achieve this metric. This is a team effort. And people have to understand that because people, especially in education, really have this accountability sort of mindset ground in. So we have to sort of...

 

[overlapping conversation]

 

0:25:46.8 AS: Which can also bring in defensiveness or...

 

0:25:50.1 JD: Defensiveness or what's gonna happen if we don't meet these goals, those types of things. So we kind of constantly have to talk about that frame. I think it's important for that challenge to be quantitative. So it's very clear if we have met it, if we're on track, if we didn't meet it, if we're not on track, if we're not moving in the right direction. I think that's really important. I also talked about how long we're spending defining the problem, understanding the problem, understanding the system before we ever got to solutions, before we ever attempted any solutions. This PDSA is not about testing an intervention, it's about gathering information, gathering additional data, which is a fine use of the PDSA. And then the final thing I would say is, the model is then paired with this improvement process. These guiding questions that are tagged to these tools are very helpful, because we have this now written record and people can respond to it.

 

0:26:53.0 JD: They see it in black and white, and then we can refine it based on the input of the group. And I know that sounds simple, but most of the time we just talk and we don't write stuff down, and we don't have a record, and just makes it harder to see sort of what we said we were gonna do, did we do those things? One thing I didn't mention, in our sort of project and meeting tracker, there's a tab called Journey. And every milestone, every meeting, there's a one-sentence description of what happened, and there's a link to any tool that was created at that meeting. So on one single tab of a spreadsheet, anybody could see the entire process that a particular improvement team went through on their particular project.

 

0:27:40.3 AS: Excellent. Well, on behalf of everyone at the Deming Institute, I want to thank you again for your discussion. And for listeners, remember to go to deming.org to continue your journey. You can find John's book Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools on amazon.com. This is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."