Remove Barriers To Joy In Learning: Deming in Schools Case Study (part 18)

How do grading systems, teacher ratings, school rankings, and other programs like those create barriers to learning? Should we eliminate them entirely, or do they have their place? John Dues and host Andrew Stotz talk about how to preserve joy in learning.

TRANSCRIPT

0:00:02.4 Andrew Stotz: My name is Andrew Stotz. I'll be your host as we continue our journey into the teachings of Dr. W Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode 18, and we're continuing our discussion about the shift from management myths to principles for the transformation of school systems. John, take it away.

 

0:00:31.3 John Dues: Good to be back, Andrew. In this episode, we're doing the 12th principle. So we're on 12 of 14, remove barriers to joy in work and learning. So that's a certainly a concept that we've talked about, but I'll start by just reading the principle. "Principle 12, remove barriers that rob educators and students of their right to joy in work and learning. This means working to abolish the system of grading student performance, the annual rating of staff and accountability rating systems for schools and school systems. The responsibility of all educational leaders must change from sheer numbers to quality." There's two really great quotes I like from Deming. One on joy in learning where he says, "Our schools must preserve and nurture the yearning for learning that everyone is born with. Joy in learning comes not so much from what is learned, but from learning."

 

0:01:24.4 JD: And then for joy in work, he says, "Joy in the job comes not so much from the result, the product, but from contribution to optimization of the system in which everybody wins." So he is saying basically the same thing in those two quotes, but he is talking about the contributions to the process is where the joy comes from, not necessarily the outcome. And so much of the time, we're focused on the outcome, be it the work product in a work setting or the test scores perhaps in a, in a school setting. But he's really talking about what is that process that you're contributing to? And, and you know, how do you feel because of that, those contributions you're making? I think whether you're talking about joy in work or joy in learning, sort of unifying theme in principle 12 as it's, this concern with the pride of workmanship, whether that's the workmanship of making a product or in the learning that you're doing or something you're doing as a result of that learning, like a report or a poem that you've written or whatever.

 

0:02:31.6 JD: And so I think as a result, it's barriers that get in the way of joy in work and learning. And you know, maybe one of the most important obstacles to improvement of the quality of our education systems in the United States. And you know, just like, sort of, it says in the outline of the principle, there's really sort of three levels that these barriers exist at. You got the students and the grading of students. And then you have oftentimes some type of rating system, evaluation system for teachers, for principals, perhaps sometimes those rating systems use test scores or other similar metrics. And then that third level is, you have the actual schools or school districts themselves that are being rated within these state accountability systems. So you sort of have, you know, these three levels. And then there's this common problem at all three levels, regardless of which one. And that's basically this thing that we've talked about repeatedly, where you under-appreciate the contribution of the system to the performance of the people, whether you're talking about students, teachers or, or you know, school systems. So I thought that's where we could focus today.

 

0:03:49.8 AS: Yeah, you know it strikes right at the heart of everything that we believe, as particularly as Americans, but certainly spreading that around the world, that it's all about measuring, ranking, tracking. You know, when a parent puts a kid in school, what do they want to know? What was their grades? When a student's in trouble, it's 'cause of grades. And what a student wants to know, like everybody wants to know and rely on grades. So it's just so, it's so difficult. You know, I was talking with someone else talking about why Dr. Deming's philosophy hasn't been adopted as as widely as you'd hope. And I think it's part of, it's just because it's just sacred, the sacred heart of everything that we believe. And if you can measure it, you can track it, you can feed that back and give it to people and show them where they are and you deserve where you are based upon your efforts, and you've gotta move yourself from there. That is so ingrained. And I'm just curious, like what's the hope from your side that this can be seen. I think it can be seen if you stop and look, but it's so hard to implement.

 

0:05:18.7 JD: Yeah. Well, I mean, I think one thing that can be confusing is obviously Deming was a statistician. So he is, has no problem with using data to improve the quality of our schools or even an individual lesson that a teacher delivers, gathering some data on how students are doing and tracking that over time. There's no problem with that. There's no problem with that I don't think at the school level either. I think the problem comes in when you create a reward and sanction system around that data. And I think that's, Deming actually think he indicated that, that system of reward or sanction on the other side is one of the main constraints from being able to develop this win-win culture. Whatever level of the system that you're talking about, that student level, the educator level, or the school and system level. They're, all those grading systems are really reward and sanction systems. And I think when you take the data and use it in that way, that's when I think Deming is talking about the real problems, the manipulation, you know, the competition for top spots that leads to all kinds of strange behaviors, those types of things. That's really where he's focusing most of his attention.

 

0:06:49.1 AS: And if you had no constraints from governments or other outsiders and you were setting up a new school right now with zero constraints, the only thing was absolutely optimizing the learning of young people. How would you handle this - grading? How would you handle all of this? Would you do it in a different way? Would you just do it and de-emphasize it and say, oh, well, it's not so critical? It's just information feedback, or would you teach them how to use that data like Deming may, or like how he uses data? Or would you say, no, that's just that there's no redeeming benefit, if we're not required to do it, then we wouldn't do it?

 

0:07:43.4 JD: Yeah, I mean, I think it'd be some combination of the things that you mentioned. I mean, on just from a practical sense, Demings certainly understood that we live within the world that we live within. And so if that hypothetical school that you're talking about is a public school and I was in Ohio, I would obviously give the state test and take whatever data I could use and use that in a positive way that I can. So I'm gonna do the thing that I'm caught on to do as a educator in a public school system. So I wouldn't opt out or anything like that. I think in terms of how I set up internal systems, I think, ..GAP. Yeah, I thought a lot about this but I haven't maybe put to paper exactly how I would do it.

 

0:08:36.2 JD: I think...I certainly would use assessments. I certainly would track how students are doing on standards. I would involve students in doing so they could track that over time. In terms of grading, I don't know exactly what I would do. I would definitely de-emphasize that to the extent that I possibly could so that the emphasis is on the learning and not on the grades. That would be a key sort of guiding principle. There's certain things that I think are outlandish that schools do, where they do pep rallies or pep rallies or something like that to, as the buildup to state testing comes, I find those things ridiculous. So I would stay away from doing anything like that. Kind of how we've treated state tests in the past, even prior to discovering Deming was matter of fact.

 

0:09:47.2 JD: Like, this is something we're preparing for, we're gonna do our best. We're gonna try our hardest, we're gonna learn from the experience, we're gonna work hard on it, and then we're gonna move on. You know, that's the type of mentality we had. In terms of like the mindset of the school I led, there was a poster in the hallway that said, you get it wrong and then you get it right. So that was the mindset is, we learn from our mistakes. We talked about creating a culture of air in our classrooms so that students felt, you know, safe, I guess is the word I would use to, or willing to call out when they didn't understand something, or they did make a mistake, and then we work together to rectify that. So that's a little bit a long-winded answer. I don't have it all worked out. I have some ideas, but I think overall using data is fine. I think it's, when you get into the rating and the ranking, that's where the problems and the rewards and the sanctions, I think that's where the problems generally come from.

 

0:10:50.8 AS: Yeah. I mean, I'm kind of unconstrained in my Valuation Masterclass Bootcamp, because I'm not under any, there's no supervision of what I'm doing by anybody. It's just me trying to make a better experience for the students. And the idea of grading never really came into my process. It's interesting, John, that one guy who was a student of mine, he graduated and then I hired him to work with me to take care of the other students. One of the first things he did was come up with a matrix and a grading system, [laughter] just because that's what he knew. And now he uses that system and he has little points that he gives. That system doesn't have any connection to whether you're gonna pass the class or not, or there's no ranking or anything related to it. It's just that, okay, you only got six points out of 10, which means you haven't really done the assignment. He's clearly defined what are the things that you need to have done? And then he goes through and says, did you do them? So it's definitely a, I think it's a good feedback mechanism.

 

0:12:08.4 JD: Yeah.

 

0:12:10.2 AS: But, you know, whether it's valuable. I think what I'm trying to do is create the experience that young people are learning how to value a company through the process of learning and discovery and, and discussion and, and, and going online and going through my material, asking me questions, and then demonstrating, showing different things. And then they're slowly putting those pieces together. And I can also see that it takes time. You know, it can't all happen in one week. We have six weeks and where they're at at the end of the six weeks is so much further along than where they were at the beginning. But I guess my point would be if I was completely unconstrained, which I am, it's just that it wouldn't be pro or con grades and ranking, it wouldn't even really exist because it's not a core part of learning. Core part of learning is providing the environment, the excitement, keeping people on track, helping them see, okay, here's what you gotta do now, see if you can do it. You know?

 

0:13:18.8 JD: Yeah. And I think what I was gonna do today is bring this alive. You know, we've talked about grading of students and even the performance appraisals in some of the past episodes. So what I thought we would do here is, sort of, focus on why grading schools could also be you know, a barrier to joy in work, how this might play out. So I think generally most people are now familiar with, because in the public school system, we've had various types of rating systems. You know, each state has their own. And I think it's, what's important here is to look at examine if the ratings help the public differentiate between schools that are doing a good job of educating kids and those that are not, because that's the point.

 

0:14:09.7 JD: Right? At least one of the points. And on its face, it sounds simple, you know, up until this year, Ohio has like an A through F grading system for schools, and there are sub components and you get an overall grade A to F. Right? And so it sounds simple. Schools with more A's are better schools, except for it's not that simple when you go beyond like a surface level analysis. So I thought it'd be helpful to just zoom in on two schools located here in Columbus. One's called Jones Middle School. It's in the Upper Arlington School District, which is close by. And then Columbus Collegiate Academy, I'll call it CCA, it's also a one of our middle schools. Both the schools serve grades six through eight. They're less than 10 miles apart here in Columbus.

 

0:15:03.6 JD: So they're geographically proximate. And this analysis comes from an article I wrote in 2020. So it's from a few years ago. So the results are a few years old, but you know, I think they're fairly representative of how the schools have performed over the last decade or school or so. So let's start with just the grades that the schools received. So the schools get an overall grade. Jones Middle School has an A, CCA has a B, so you know, fairly close there, but Jones outpaces. And then there's the achievement grade. And that's basically looking at all the kids' test scores, how they do overall. Jones gets a B, CCA gets a D, right? So Jones has quite a bit better performance. And then there's a progress category. So how much progress did the kids make during that particular school year?

 

0:15:57.9 JD: How much growth did they make? Now this is interesting now, CCA gets an A and Jones got a B. So just to recap, the overall grade for Jones was A, achievement B, progress B, for CCA overall, B achievement D, progress A. So basically a higher percentage of students at Jones begin year on grade level 'cause they have that higher achievement grade, but they don't grow as much as the students at CCA once they're there. This difference between achievement and progress grades becomes even more interesting as you start to factor in not only the school characteristics, but also the neighborhood characteristics. So let's talk about inside the school just to start with. So in terms of student population, Jones and CCA are pretty similar in terms of students with disabilities. So those kids with special education needs tend to, as a general rule score lower on standardized tests.

 

0:17:02.9 JD: So those populations are roughly equal, but 100% of the kids at CCA are economically disadvantaged as defined by the state. At Jones, just 2.4% of the kids are economically disadvantaged. When you look at other report card measures such as attendance, chronic absenteeism, Jones has much better rates. So 97 plus percent attendance rate, just 2% of their kids are chronically absent. At CCA 93% attendance rate, 21% of the kids are chronically absent. But when you start to look at these, some of these metrics framed in terms of the poverty rates in the community surrounding CCA, these numbers start to take on a different meeting. And I think what they're, especially things like chronic absenteeism, that's all the rage right now, attendance, I think what you start to need to understand is these are indicators of inequity, housing instability, neighborhood violence, lack of access to healthcare.

 

0:18:15.5 JD: I think they're more an indicator of those types of things than they are of school performance. So as you start to think about things in those ways, what you realize is that the students at CCA are just as capable as the students at Jones, but they face sometimes overwhelming obstacles related to poverty. It's also interesting to take a look outside the schoolhouse. So the median family income in the census tract where Jones is located is $184,000. So the median family income in that neighborhood, so it's a pretty affluent area. In the neighborhood surrounding CCA in that census tract, the median family income is just over $20,000. So we're talking about an order, orders of magnitude higher family income in upper Arlington than in the neighborhood that CCA sits in. And then there's all types of factors. Some grounded in historical reasons that relate to this, but they're also compounded by funding disparities. So the per pupil revenue at CCA for this year is $10,600. In Upper Arlington, it's nearly $17,000...

 

0:19:35.0 JD: This Jones Middle School has almost no students living in poverty, yet gets $6,000 more in additional revenue per student than the students that attend CCA. So think of the implications of that.

 

0:19:52.3 AS: When you say they get more revenue, you mean the state or the government's providing them more money per student?

 

0:20:00.1 JD: Yes, all in. From all sources. So when you look at what the federal government provides, the state government, and then local funding sources. When you look at all those sources combined, this more affluent middle school gets $6,000 more students, dollars per student.

 

0:20:14.5 AS: Obviously, it's not based upon need. Is that based upon the grade or some other?

 

0:20:19.0 JD: Well, it's because the funding is heavily influenced by local property taxes. And because of the affluence of...

 

0:20:26.7 AS: They have the resources.

 

0:20:28.2 JD: They have the resources. And in Ohio, charter schools don't have access to local money. So that explains most of the gap. Most of the gap. But back to my point, when you think about CCA, having kids with more challenges, less money per student, less resources to pay for a facility, to pay teachers a competitive salary, extracurricular activities, all those types of things that we want to equalize are highly inequitable between those two schools. So then you start to ask yourself, well, what are the report card grades measuring exactly? Are those grades on those state report cards a fair representation of what's happening inside the school? Or can a significant portion of those grades be attributed to this larger context in which the school sits? And I think that's where you sort of put on this systems thinking lens and realize that, sure, what teachers and the principal is doing inside the schools, they are certainly making contributions to those state report cards. But you cannot ignore what is going on outside those schools and those neighborhoods when you're thinking about these grades.

 

0:21:54.1 JD: And so if you're sort of thinking about... Like a formula that would sort of lead to the school's results and you just... Let's just call it A+B+C+D+E=71, where 71 is the score that the school gets. Let's just call it that. And let's call the school's contribution letter F. A, B, C, D, A+B+D+C... Or A+B+C+D+E+F=71. The school's contribution is F. Well, that equation cannot be solved unless you know the values of A through E or at least some of those values. But what we try to do with this state report card system is that we assign this value to F, the contribution of the school, with no knowledge of the effects of these other variables.

 

0:22:57.3 AS: So it's, in other words, the contribution of the school is 100%. You mean you're responsible for your results? Is that what it means?

 

0:23:04.4 JD: Well, right. So if you're going to give me... If you're going to give CCA Main Street a D in achievement, that means the only thing that contributed to that grade was the school. But there's all those things that we talked about. Some, sort of, when you look at the variables A, B, C, D, and E, those other variables, you could look at things like, what are the state standards? Or what's the test design? What's the school funding? Household income? Home environment?

 

0:23:34.1 AS: Education level, maybe, of families.

 

0:23:37.2 JD: Education level of parents. Teaching methods. All of these things are variables that are outside of the school's control, or most of the ones that I just mentioned. But we don't, we don't see that when we look at these report cards. Right? You know, and just like I said at the beginning, despite all of that, I'm in favor of administering these state tests that are standards aligned, reported annually to the public. I actually think understanding how students are performing in a standardized way is actually... Could be useful information, I think. But when you extend those systems to the grading, the rating, and the ranking, I think that's a misuse of the information. Because too much of the rating and ranking comes from the system, as opposed to being directly assignable to the school or to the individual educators within that system. I mean, I think, if you analyzed report cards in this way, I think my opinion is that a reasonable person would conclude that the comparison between Jones Middle School and CCA is not a fair one.

 

0:24:56.4 JD: Because those students that arrive in those schools are not on equal ground upon enrollment. And I think our time would be better spent figuring out how to make things more equitable between those two groups of students than constantly recalibrating these rating systems that at best communicate confounding information, conflicting information.

 

0:25:27.5 AS: KPI experts around the world listening and viewing this are saying, "oh, come on, John. What? All you got to do now is you just got to break it down. And now we're going to do the KPI by adjusting for these factors. And now we're going to compare schools based upon that." Of course, what we've learned and I've learned over my life is that every time you think you're going to break it down and make it more comparable, it gets harder and harder to do that. And it just becomes less reliable and less useful in a lot of cases. Not completely. I mean, making some simple adjustments just for, let's say, yeah, I suspect that just one factor could probably represent A, B, C, D and E probably pretty well. Maybe that's the income of the area or the amount of funding that they got. One or two of those factors probably is enough to say, okay, we gotta compare schools that have these factors similar as a first step. But every time that I've ever gone down to go deeper into measuring, it just gets...it, it, the answer isn't there.

 

0:26:40.6 JD: Yeah, and I'm going to tie this back to joy in work. So if you think about that current school rating system, what we fall prey to is that fundamental attribution error that we've talked about before, where we have this tendency to underestimate the impact of situational factors on other people's behavior and overestimate the impact of individual factors, you know, when it works in our favor. But what happens is if I'm a teacher at CCA, there's a likelihood that I'm going to get blamed for the results. Let's say the achievement results. If I'm at Jones Middle School, there's a likelihood for praise. Because the school is doing pretty well. But in both cases, in both cases we're vastly underestimating the impact of the situational factors or the system on those results. So over time, I think this can have an impact on joy in work of educators working in these challenging schools.

 

0:27:49.3 JD: Even, even in the case where in many of these schools, like the one I just talked about with CCA, that there's solid evidence that staff and these schools are often getting better outcomes if you go beyond the surface level analysis. Because if you remember, they did quite a bit better on the progress, meaning they grew the kids more in a single year, even though they may have not hit the proficiency standard at the rate that Jones did, they grew the kids more. So you could make a solid argument that CCA is actually better, even though they got lower grades on the report card, right? I've often said, what would happen if you just switched the two staffs?

 

0:28:33.6 AS: Yeah. Problem solved.

 

0:28:36.3 JD: What would happen to the report card? You know. That's interesting. Obviously, it's never going to happen, but it's an interesting hypothetical experiment. My guess is a lot of teachers will find out that would go from Jones Middle School to CCA in a much more challenging environment would find out pretty quickly, that a lot of their methods don't work as well, right? So I think that these are the types of things that we're talking about. Imagine if you're at CCA year after year after year after year, get these lower grades. Right? And even if there's some evidence, like the progress score, who's digging in to find this? That score is often harder to find than the overall grade. That score is often not in the headline and what makes it into the newspapers. You know? And so you start to ask or you start to doubt yourself. You start to think about, am I really good at my job? Those types of things come in. And if you don't have someone there doing this deeper analysis, putting this in context, that's not easy to do.

 

0:29:52.1 AS: Yeah, when the pressure's on.

 

0:29:54.4 JD: When the pressure's on. And even if you're good at doing that type of analysis, sometimes people won't believe you because, well, that's not what I'm hearing. That's not what I'm... That's not what my family's saying. Those types of things. And then, and then, if you have those good teachers that at a certain point say, I'm just going to go somewhere where it's easier. Then those kids at CCA wind up in a worse place. And that's, I'm using CCA as an example, but I think this plays out at you know challenging schools all across the country all the time.

 

0:30:36.2 AS: Yeah, when you were talking about the morale of the CCA teachers, I was just thinking some brilliant bureaucrat would probably come up with the idea of why don't we post this grade right on the front of this school?

 

[laughter]

 

0:30:51.4 JD: Well, yeah, they're easy to find. That's for sure. These are all public, public reports. Sure. And in fact, actually, back during, I think, during the Obama administration, during Race to the Top, when it became really in vogue to rate teachers based on their progress scores, the individual teachers. The school report cards are easy to find, like a report card on any public school in Ohio or any public district. But in some cities, what started happening is they were, newspapers were getting a hold of the list of the progress rankings for individual teachers and posting those. I remember some of those were in the newspaper. And I think we've talked about this here as well, that what researchers have shown over time with these progress scores, these value added scores, is that some of the score is attributed to the teacher from before. Teachers that take on more challenging groups of students tend to have scores that are... Progress scores that are lower, all types of things And you want good teachers in those rooms. And what you're doing is disincentivizing that to happen when you have these types of rating and, rating and rating systems. So it's a tough thing.

 

0:32:15.2 AS: It's such an interesting topic. And I think, it got me thinking that we should start a new series on the Deming Institute podcast, which is, bad use of data. Like examples of, you know, here we have a misuse of data or just the simple thing of not making adjustments for situational factors and the misattribution. You could argue if you just improve that, maybe there's a little bit more meaning to this. But then, of course, there's also all the unintended consequences. And I just would imagine, I'm thinking about a book I have called the... By Terry Mueller, I think, or Jerry Mueller, which is the Tyranny of Metrics.

 

0:33:05.9 JD: Yeah, we got a lot of copies of that in our, right in this room where I'm sitting.

 

[laughter]

 

0:33:10.8 AS: Yeah. And I think that that would be kind of fun to bring out from the audience examples of what you're seeing.

 

0:33:17.4 JD: Yeah. Well, and one thing I didn't even mention that is also a key contributor here is, so let's say these two middle schools get this state report card. And another contextual factor is that most of the kids that go to Jones Middle School went to, I don't know the name of it, Upper Arlington Elementary School. And a very stable neighborhood. And of course, there's a few families here and there that will move in and out. But for the vast majority, I guarantee a vast majority of the kids that took these tests in sixth, seventh and eighth grade at Jones have been in Upper Arlington since kindergarten or preschool.

 

0:34:00.4 AS: Yeah.

 

0:34:01.2 JD: CCA Main Street, because of the nature of charter schools in Ohio, is a standalone 6-8 middle school. So that means 0% of the kids went to our elementary school during these years. And now whatever happened K-5 in a school, those kids school career, that certainly plays a big role in how they're going to show up when they enroll at CCA. So the only rule in terms of counting for CCA's test scores is that the kid had to be enrolled by October, let's say the first week of October. And they take the test in March. So six months later, let's say.

 

0:34:49.2 AS: Yeah.

 

0:34:49.6 JD: So let's say probably 50% of the kids at CCA, were brand new to that building, to that district, that school year, whereas the vast majority of Jones middle school students had been in that district for seven or more years. Because kindergarten is a year, and then when you're a sixth grader. So the time that they've been there, that's not taken into account either. And that may be the most important.

 

0:35:19.0 AS: Yeah. That's fascinating. So how would you summarize the one thing you want the listener, the viewer to take away from this.

 

0:35:30.5 JD: Yeah, I mean, I think it can be easy to start to think that data is bad. That is not the problem. You need data to help inform your decision making.

 

The problem comes when you then take the data and attach the ratings and the rankings to it, that's when the problem comes in. So you need to detach those two things.

 

We need to keep it public, keep it transparent, keep it known by all stakeholders, be it parents, the public, policymakers, students themselves. But it's the rating and ranking, that's the problem. That's the key takeaway.

 

0:36:11.9 AS: Great. Well, on behalf of everyone at the Deming Institute, I want to thank you again for this discussion. And for listeners, remember to go to Deming.org to continue your journey. You can find John's book, Win Win. W. Edwards Deming, the System of Profound Knowledge and the science of improving schools on Amazon.com. This is your host, Andrew Stotz. And I'm going to leave you with my favorite quote from Dr. Deming, which is totally pertinent to what we just discussed. And that is: people are entitled to joy in work.