Joint Assessment of ASCO, ESMO Value Frameworks in Focus on ASCO in Action Podcast

Subscribe to the podcast through iTunes and Google Play.

Dr. Clifford A. Hudis (CH): Welcome to this ASCO in Action Podcast. This is ASCO's podcast series where we explore policy and practice issues that impact oncologists, the entire cancer care delivery team, and the individuals we care for-- people with cancer. My name is Cliff Hudis, and I'm the CEO of ASCO, as well as the host of the ASCO in Action Podcast series.

For today's podcast, I am really delighted to have Dr. Lowell Schnipper, chair of ASCO's Value of Cancer Care Task Force as my guest today. In addition to his extensive service to ASCO, Dr. Schnipper is the Theodore W. And Evelyn G. Berenson Professor of Medicine at Harvard Medical School, Chief of the Division of Hematology/Oncology at the Beth Israel Deaconess Medical Center, and Clinical Director of the Beth Israel Deaconess Cancer Center. He is also an Associate Director of the Dana-Farber Harvard Cancer Center and a member of the Cancer Center's Executive Committee.

Now, to provide some background for our listeners, I do want to highlight a couple of related points. First, as early as 2007, ASCO, led by Dr. Schnipper, was already focusing on rising drug and health care costs. And two, as detailed in our five-year strategic plan, almost everything we do at ASCO is aimed at helping our members and all of society achieve high-quality, high-value care for all people with cancer. Examples of the latter include our Patient-Centered Oncology Payment model, our participation in the Choosing Wisely campaign, our entire CancerLinQ project, and ASCO's Quality Oncology Practice Initiative, or QOPI, which is now available internationally.

So our conversation today will focus on one longstanding project that's part of all of this effort. And this is one that's been a critical component of our efforts going back really, starting in 2007, ASCO's Value Framework. Here, I want to note that both ASCO and ESMO have developed algorithmic scales that are designed to evaluate the benefit of new cancer therapies.

Again, ASCO's is called the Value Framework. And it was developed primarily as a physician-guided tool to facilitate shared decision making by patients and their oncologists as they select among high-value treatment options for individual patients. In our framework, the clinical benefit of a specific treatment and its known toxicity are combined, and they produce a score that we call the Net Health Benefit.

So after years of building, testing, and refinement, we recently conducted an analysis that compared the output of ASCO's Value Framework against the European Society for Medical Oncology, or ESMO's tool, which they call the Magnitude of Clinical Benefit Scale.

Dr. Schnipper, you co-authored the assessment that we're going to discuss today. And that's why I'm so delighted to welcome you to the podcast. I want to thank you, in advance, for sharing your insights with us here today. Thanks for joining us.

Dr. Lowell Schnipper (LS): It's my pleasure to join you. This is a terrific opportunity to explain a bit about what ASCO is doing and put it in the context of what other groups are doing, in particular, a group like ESMO. We started this effort, as you were pointing out, approximately a decade ago, not the Value effort, but our emphasis, as a society, on the increasing cost of cancer care for our patients.

And we wrote several manuscripts detailing the importance of doctors being sensitive to cost and, of course, providing the patients with the best opportunity for high-value care. As the years evolve, and some of the initiatives that you've already mentioned, Cliff, we felt it really important to develop a formulaic way of approaching how we, as an oncology community, might assess the clinical value of the drugs we use to treat patients with cancer and convey that, then, to our patients in the context of shared decision making.

That's really the background of this effort. And the Value Framework itself has actually evolved over about three or four years in a number of iterations. And I'm hoping that as the discussion ensues, we'll be able to get into that in more detail.

That's great. So let's actually start with the main topic today, which is the comparison between ASCO's work and ESMO's. Why, exactly, did you decide to pursue this comparison and then publish it?

We became aware that ESMO was undertaking a very similar initiative, namely an attempt at developing an algorithm with which to assess the value, the clinical value, of oncologic therapies representing all of the European nations. That's about 27 nations. In parallel, but actually quite independently, we at ASCO were developing our own Value Framework.

And as one can see when reading about either of them, in many ways, they're quite different. But they have overwhelming similarities, specifically based on assessments of how much good a given therapy does for the patient when compared with a control treatment and, of course, how negative any of the effects of our therapies are to that patient population. And we integrated that, at ASCO, in our Value Framework in a concept that you rightly referred to as the Net Health Benefit.

Well, ESMO, we came to learn as we were readying to publish our framework, that they had worked on something very similar, looking at clinical benefit and toxicity somewhat differently. And they, of course, published theirs in the Annals of Oncology. And we published our framework in the Journal of Clinical Oncology.

These frameworks were met with an enormous amount of interest by every component of the oncologic community. And by that I mean patients, of course, are exceedingly interested in this. And we can get, I hope, to the patient perspective in a little while. But in addition, manufacturers were quite interested in how we were going about assessing the products that they bring forth and we put into clinical trial.

And of course, ultimately, the oncologist, him or herself, is the deciding factor in recommending or not recommending a particular therapy. So we elicited tremendous interest. And I didn't even mention the payer community which, of course, was very, very interested because of the rising cost of oncologic therapies.

In fact, I will say that one of the motivating factors for the ASCO community, and I think for the ESMO community as well, is the awareness that oncologic therapies are very, very costly, sometimes eliciting huge out-of-pocket payments for our patients. And now, I speak for the American health consumer, not so much those in Europe. And this becomes a major factor in the patient's and the patient's family's well-being, as in this risk we know for financial toxicity.

So we felt it very important to establish points of overlap and similarity, as well as points of difference with the ESMO framework in order to understand and improve each other's work. And so while we don't view this necessarily as a planned collaboration with which we'll modify each of ours after arguing about different points in a scholarly way, what we really wanted to do was see how each framework performs when looked at through the same lens of a number of trials.

And so that led to this comparison of, well, we started out with over 100 trials and focused on 97 that were being done in a prospective randomized clinical trial context for palliative purposes, meaning for non-curative situations. And that really is the background of how it came about.

CH: So before we dive into that aspect of the results, I think it might help to clarify, for just a moment more, the comparative effort on two planes. On the one plane, which you touched on, there is the actual design of the tool, what it takes into consideration, and what it weights. And on the other plane, there is how it actually evaluates specific treatment options. So can we just circle back, for a second, at a high level, what is it about the two tools that you would say is similar to overlapping? And what is it that is, perhaps, different, if anything, about the two tools methodologically?

LS: Both tools are premised on using a prospective clinical trial in which a comparator is compared in a specific clinical disease setting, clinical cancer disease setting, with a test agent, or a test regimen. And so both of them look at that particular subset. And, in fact, that is the case for these 97 trials that I just alluded to.

The ASCO clinical benefit, when comparing a test regimen to a control, is literally measuring the difference between the two in the hazard ratio, or the overall survival. So, for example, if the overall survival is measured in months, but no hazard ratio were given, if the overall survival for the control was 10 months, and the overall survival for the test agent was 20 months, in fact, we would register basically a doubling, or a very high degree of clinical benefit.

And if the hazard ratios are such that there's a very low hazard ratio for death, meaning a high likelihood of survival, then the ASCO Clinical Benefit Score would register a large number, accordingly. Our attempt at finding, numerically, the difference between a clinical comparator and a test regimen differs from ESMO where they have reduced categorical differences. In other words, they ultimately arrive at a score of 1, 2, 3, 4, 5 rather than measuring the quantifiable difference between the comparator and the test agent.

So we actually differ in how we go about defining the clinical benefit. We are trying to be a little bit more precise. And ESMO, I think, feels that that attempt at precision may not always be completely valid, because studies vary in terms of sample size and that could influence the variability around the mean or the median. So there's some technical differences, or biostatistical differences.

But basically, we each are trying to develop a magnitude of clinical benefit and have done so in our frameworks. We differ from ESMO in the sense that we have developed a rather elaborate toxicity scoring system. And that's largely because we have, in developing the framework, interviewed many, many patients or leaders of advocacy groups.

And they impressed upon us that it's not just grades three and four, or heaven forbid, grade five toxicity that worries them. It's often low level of chronic toxicity that also impacts on the quality of their lives. And we felt we needed to develop a tool that reflected the patient's perspective as it was explained to us.

And so we have a much more elaborate, and I will confess probably more difficult to pull out of the literature and score, than the ESMO toxicity assessment in which they more or less say, this is highly toxic, not so toxic, or very minimally toxic. So we feel that we needed to do that and that it makes sense because it reflects our patient's input. And again, this is a tool we keep coming back to that's to help patients and docs make decisions. So those are a couple of the ways in which we actually differ appreciably.

In addition, ESMO practically ignores progression-free survival, which, as you know, many, many prospective randomized trials use as a primary endpoint. I think we, at ASCO, completely agree it's not the very best surrogate endpoint. And as such, if [INAUDIBLE] namely, the better endpoint, is not the primary endpoint in a given trial, we will utilize the progression-free survival in our value assessment, but we actually downgrade its value.

So, in other words, if the value of a score for overall survival as opposed to the value of a score for progression-free survival differs, we indicate it by about 20% less for progression-free survival. So the differences in the toxicity parameters we felt were important to maintain and the difference in the progression-free survival measurement, or I should say downgrading its importance, that, too, was also difficult to at least-- well, I should say progression-free survival turned out to be a less valuable endpoint than overall survival.

And we felt quite legitimized in reducing it. ESMO agrees, but they basically give very little credit for progression-free survival unless the improvement in progression-free survival is associated with virtually no toxicity. So we differ a little bit in that. But again, we're circling around the same sets of variables, but coming at them somewhat differently.

CH: All right. So let's talk about the results for a moment. That background, I think, is really helpful to audiences who are trying to put these in context. But a couple of points that I would make, and then I have a question, or maybe one point I would make is, there have been a number of prior studies that have been conducted by external researchers.

They've looked at and compared the two frameworks. And they reported lower levels of agreement between ASCO and the ESMO frameworks than you and your colleagues are reporting now. What's fundamentally changed? I mean, how was your analysis performed? And why does it show a greater degree of concordance than earlier external investigators found?

LS: I'm glad you asked that. That's a key question. And of course, we were initially kind of dismayed when we saw published results in which there was some degree of disagreement between the ASCO and the ESMO scales. Well, it turns out that there was one, out of about four or five that were in the literature, that actually found a high degree of concordance, although it looked at a relatively small number of trials within the same disease area-- lung cancer.

So what we realized was that these are complicated frameworks to work with, that they're not trivial. And the reason I arrive at that is because we are blessed with a very, very bright, talented staff within ASCO who embraced the task of applying our Value Framework to these 103 trials. What we learned is that for people who weren't involved in generating the framework, but were called upon to apply it, they literally had to get onto a learning curve in order to produce consistent assessments after culling out the relevant data from a given paper.

So we are pretty sure that the low levels of concordance really had to do with people misapplying the scales in many, many cases. And the proof of the pudding, I think, is in the eating because while we didn't find perfect concordance, and that gives us additional food for thought, we actually did pretty well, because 65% to 70% of the trials actually came out with an agreeable scoring, meaning that what ESMO concluded was reasonable to recommend incorporating into a European nations Pharmacopoeia for cancer drugs, represented the upper half above, let's say, a Net Health Benefit score of 45 for ASCO.

So those 65%, 70% of trials actually scored pretty much concurrent with one another in terms of identifying drug regimens as useful. The outliers, those that differ, were actually another cause for our scratching our heads and, ultimately, teasing apart the differences. And we do understand some of the reasons why the scales are different, which isn't to say one is better than the other, but it probably just rests on bedrock assumptions in terms of incorporating various variables into the Value Framework.

CH: So that's great. A couple of follow-up questions, I suppose. And I'm just curious, either from the ASCO side or, if you know, from the ESMO side, based on this effort to compare them, is there any plan to actually tweak them to increase their alignment? Or is everybody happy that they're each fulfilling their planned functions and this little bit of discordance is OK?

LS: That's a great question. I can say that, at the very start, this goes back a couple of years, I had meetings with counterparts at our annual meeting, or at the ESMO meeting, just talking about whether or not it would be desirable to essentially arrive at a transatlantic consensus of what would be thought to be a clinically useful addition to our Pharmacopoeia versus not. And I think we are all motivated by exactly that, although I think we don't necessarily have a plan at the moment to try to develop a single uniform framework, at least not for the foreseeable future.

And the reason for that is that we are trying, in the US environment, to utilize our framework in some of the organs that we, as oncologists, typically go to in order to assist with clinical decision making. And so here, I'm referring to the JCO when we publish prospective randomized trials or, for that matter, other high-impact journals like New England Journal of Medicine or perhaps JAMA. We would like to actually begin a catalog of ASCO Value Framework Net Health Benefit scores as these trials accumulate over the next 6 to 12 months.

And that's because we'd like to actually get a feel for that which oncologists are comfortable using and recommending to patients and how our Value Framework stacks up against that. We're thinking that that could be a very, very helpful way of us understanding if we're throwing fastballs right over the plate, or are we missing the mark sometimes and do we need to make some mid-course corrections.

And as you can imagine, the Value Task Force, at the moment, is viewing where we are with the framework as pretty far down an iterative process that's not yet complete, because we need to incorporate this assessment into the real oncologic literature, see how well it performs with what doctors think and patients think are useful drugs. And at that point, I think, we'll be able to utilize it more effectively.

The other thing that I will say that we need to do, and we already have a sub-task force or a subcommittee working on this, is to essentially address the issue of, in breast cancer or in colorectal cancer, if there are three or four trials for a given specific clinical scenario, but each is using a different comparator, a different control, can they ever be compared if the test agents are the same, or the test regimens are the same?

And the answer is, of course, they really can't because you can't easily do cross-trial comparisons. And so we have been working with methodologists who are advising us on doing network meta analysis, a way of at least statistically looking at trials in the same patient population testing the same regimen or novel agent, but if using different controls, to essentially come up with what amounts to a reasonable sort of summarize, or summative control, against which we can then look to see how impactful a new therapy is or is not.

Statistically, it's never all that pleasant, as you know because you're a very experienced investigator, it's never all that easy to even think about cross-trial comparisons. But rigorous statistical techniques, I'm told, are available to at least help us do that.

CH: Yeah, it's interesting. There's a little bit of overlap with the synthetic control arm discussions we have in the context of CancerLinQ. But that's probably a topic best left for another day. I want to come back, though, to a core issue that I think circles all of our discussion and that is financial toxicity, which you raised earlier. And very specifically, in your dreams, how do you see that this process, both our tool and ESMO's for that matter, how do you see them actually having an influence on drug price, which is, I think, fairly summarized as the elephant in the room here?

LS: I'm glad you asked that, but I continue to struggle with what I would think would be an acceptable answer in its broadest dimensions. But my sense is the following. There has been incremental improvements in cancer therapy from the very start of our field. And you'll remember in Charles Moertel's work, the increment of 5-FU in colon cancer was trivial, a few months.

But now we know that patients with metastatic colon cancer can survive for 27 months, on average, perhaps more in many cases. So how do you judge incremental benefit? That becomes a real thorny, almost a societal issue because therein lies the rub. We know our patients are being almost asked to pay out-of-pocket quite a substantial sum for many of the novel drugs that we have.

Some of them, you and I would absolutely agree, are a slam dunk benefit. But we know, as well, that there are many that add very, very small degrees of benefit in comparison with a much more affordable, perhaps generic drug, or set of drugs. What I have hoped for is that if we arrive at what our peer group, the oncologic community and patients, feel is a credible value framework that market forces will be able to get to work and actually modulate what the price that's being asked for a given drug.

And that really goes back to the first thing we wrote in 2009 when we issued a guidance statement from the founding members of the task force. And that is that we recognize that oncologic drugs and, in fact, health care, in general, is not a market in the usual sense of the word. And the fact that it is a perverse market makes us worry, if ever, we'll get to a point where if we can show value, that will be reflected in the cost of the drug.

If you have an innovative drug that is a slam dunk, like trastuzumab, of course, that should have and merit lots different value than a drug that has a very small incremental benefit. And how that can be reflected by a value framework influencing a real world market scenario I'm hopeful about, but I don't quite know the path to get there.

CH: Yeah. I think that's actually a great summary of the frustration that so many of us feel as we think about how rational market forces should work and then what we actually see in our reality. So thank you for at least trying to put this in perspective.

Thank you. It's been my pleasure to participate. And I'm really happy that ASCO is producing vehicles like this to broaden the understanding of the efforts that we're undertaking.

As we wrap up, I want to just remind our listeners that the joint Value Framework assessment that we've been discussing is available online. You can find it at ASCO.org/value. And the ASCO Value Framework is, again, as I said in the introduction, just one part of ASCO's broader and multifaceted effort to help all of our members, and everyone, achieve high-quality, high-value care for people with cancer.

The other efforts that we touched on before and you hear about in the podcast series at various times are the Patient-Centered Oncology Payment model, the Choosing Wisely campaign, CancerLinQ, and ASCO's QOPI, or Quality Oncology Practice Initiative. For more information on any, or all, of these initiatives and for the latest cancer policy news and updates, I ask you to visit ascoaction.asco.org and you can find links out to these programs and more there.

Dr. Schnipper, I want to thank you again for joining me today for this ASCO in Action Podcast . And for everyone listening, until next time. Thanks for joining us.