Ep 71. Conviction vs. evidence: What’s driving math education’s worst policies with Thomas Briggs and David Shuck

This transcript was created with speech-to-text software.  It was reviewed before posting but may contain errors. Credit to Canadian Podcasting Productions.

A Substack version of this episode, written by Anna Stokke, as a guest writer for the Center for Educational Progress is available at https://www.educationprogress.org/p/what-to-do-when-research-shows-shuts

In this episode, Anna Stokke continues the conversation from the previous episode on what happens when requests for evidence are dismissed with the phrase “research shows.” She is joined by Thomas Briggs and David Shuck from the Center for Educational Progress to explore why ineffective or unsupported practices persist in education.

Thomas and David introduce an important distinction between misunderstanding and conviction: sometimes educators genuinely believe they are following best practices, while other times evidence is ignored or dismissed due to ideology.  They discuss examples including flawed advanced math placement decisions, an update on the New York math briefs controversy, concerns around programs like YouCubed, and the impact of San Francisco’s math de-tracking experiment.

The conversation examines how “research shows” can be used both in good faith and as a way to shut down debate, while offering listeners practical ways to think critically about evidence and education policy.

This episode is available in video at www.youtube.com/@chalktalk-stokke 

TIMESTAMPS

[00:00:22] Introduction [00:03:16] What is the Center for Educational Progress? [00:05:10] Two reasons ineffective educational practices persist [00:07:57] Examples of misunderstandings [00:09:11] Students being denied access to advanced math [00:16:14] Is conviction one of the biggest barriers to evidence-based education? [00:18:56] The controversy around the New York Math Briefs [00:21:43] NYSED response to Ben Solomon’s petition [00:25:27] Why did the NYSED’s response shift to political motivation?[00:27:41] The response from the New York Math Briefs expert [00:33:24] YouCubed’s education claims and flawed methodology [00:38:21] YouCubed’s updates and more underlying issues [00:44:23] San Francisco’s failed de-tracking experiment [00:51:57] Why tracking and ability grouping helps students [00:58:52] Final piece of advice for parents and teachers

[00:00:22] Anna Stokke: Welcome to Chalk & Talk, a podcast about education and math. I'm Anna Stokke, a math professor and your host. Welcome back to another great episode of Chalk & Talk.

In my last episode, I talked about what to do when you ask for evidence and get shut down by the phrase research shows. Today's episode builds directly on that. I'm joined by Thomas Briggs and David Shuck from the Center for Educational Progress, and we discuss why this happens.

Thomas and David talk about a really useful distinction, and that's misunderstanding versus conviction. Because bad practices don't always persist for the same reason. Sometimes people legitimately just don't know better.

They really think they're doing the right thing. But other times the evidence is there, but it's being ignored. So, we walk through some concrete examples.

Students being kept out of advanced math because of flawed placement decisions, an update on the New York math briefs controversy, research issues with programs like YouCubed, and the consequences of San Francisco's math detracking experiment. This conversation really made me think about how often people invoke research shows in good faith, but also how important it is to recognize when it's being used to shut down conversation and to make people aware that this is a real problem. I hope you enjoyed the conversation as much as I did.

Now on with the show. I am joined today by Thomas Briggs and David Shuck. Thomas Briggs is the executive director of the Center for Educational Progress, which is a nonpartisan think tank focused on evidence-based reform and academic excellence.

He holds an MA in philosophy from Georgia State and a JD, that's a law degree, from UCLA. At the Center for Education Progress, he's written on San Francisco's algebra debacle, New York City's gifted and talented policy, and the political forces squeezing serious education reform from both sides. Welcome to the podcast, Thomas.

[00:02:33] Thomas Briggs: Hi, Anna. Thanks for having us. It's an honour to be on the show.

[00:02:37] Anna Stokke: And David Shuck is a senior fellow with the Center for Educational Progress. He holds an MA in philosophy from Georgia State University, where he also taught critical thinking. And at the Center for Educational Progress, he's written on K-12 policy, the New York math briefs, and the assumptions shaping education debates.

Welcome to the podcast, David.

[00:03:00] David Shuck: Thank you so much. It's our pleasure.

[00:03:02] Anna Stokke: So, I'm excited to hear about your work. And so, let's start with some background. So, can you tell listeners about the Center for Educational Progress?

What is it and what do you do? So, Thomas, maybe you want to start.

[00:03:16] Thomas Briggs: Yeah, sure. So, the CEP is an education nonprofit think tank that advocates for excellence-oriented, evidence-based education policy. So, we publish policy analysis and research-backed reporting on our Substack, where we'll investigate policies that undermine student achievement.

And we'll also publish longer pieces that are grounded in science learning and the history of education and trying to basically paint a picture for our readers about why education got to the space that it's gotten to and how we can maybe get out of it.

[00:03:49] Anna Stokke: OK, super useful service. And right in line with the kind of stuff that I like to talk about on this podcast, I've read quite a few of your pieces and they're really good. And you have various authors on there.

Ben Solomon, for instance. Has Ben Solomon written for you? Yes.

[00:04:06] David Shuck: Yeah, he absolutely did.

[00:04:07] Anna Stokke: And Barry Garelick recently on conceptual understanding. I love both those pieces. You wrote that the Center advocates for effective evidence-based instructional practices, which requires uprooting ineffective practices.

Now, that in turn involves exposing flaws in bad research, something I also like to do, and figuring out why so many ineffective practices get adopted when the evidence supporting them is weak. So, let's unpack all that and let's talk about some of the things that you've found to be obstacles to better instruction. Now, you've described two different reasons why ineffective practices exist.

And I hadn't really thought this way. So, it's kind of nice to hear this written down and to talk about it. One of the reasons you think ineffective practices exist is just due to misunderstandings and resistance to change.

And the second is conviction. So, David, perhaps you can explain the difference for us.

[00:05:10] David Shuck: So, we understand these two reasons for resistance to changing ineffective practices as something like ideal types to help us understand both the source of disagreements that we find in education and also the strategies for resolving them. When it comes to misunderstanding, often certain things are very honest misperceptions that teachers will have. Teachers, administrators, policymakers all generally have a willingness to do better for their students, for their constituents, for stakeholders, and use the information they have to the best of their ability to advocate for change.

Sometimes the change doesn't work or the policies or practices they're promoting are ineffective, but it is based simply on the lack of information they've had or a problem higher up somewhere in the pipeline, perhaps in a teacher prep program that never gave them occasion to doubt the efficacy of a practice, perhaps in the conferences they regularly go to where the same practices are discussed. And it seems like, well, if this many people were discussing it, how could it possibly be wrong? How could this many people be mistaken?

So, we don't find cases of misunderstanding to be blameworthy in the same way as certain other cases are. Those are often easier problems to address because it is one where something like awareness and then advocating for clear and straightforward implementation strategies can resolve those problems. Conviction is the thornier problem.

When it comes to conviction, it's not simply a matter of there is evidence that a person has not come across before. Often the person who supports a practice out of conviction is a person who has encountered the evidence that counts against that practice before but continues to resist changing out of some kind of other motivation. That is not to say that they don't sincerely believe that they are trying to do what's best for, say, their students.

But we do take it to be a different kind of disagreement when someone is either honestly misunderstanding or simply used to a certain practice versus committed to a certain way of looking at education generally, a way of looking at how students should be learning or what their educational goals should be, and then reacting to inconvenient evidence by digging their heels in, changing the subject and doing whatever they can to not come to terms with the evidence.

[00:07:57] Anna Stokke: I'm very familiar with both of these. I'll give a couple of examples from the math point of view. So, an example of what I think is a misunderstanding would be a lot of teachers believe that time tests cause math anxiety, and that is not evidence based.

It's not backed by evidence. But I would say that it likely originated from people who are influenced where the problem is actually conviction. OK, so people who have told this, have stated that this is true, stated that it was backed by research when in fact it isn't, and they should know it isn't, you know, because in a lot of these cases, these are people that work in faculties of education and they really should know better.

OK, but on the teacher's side of things, it's a misunderstanding. Another example that's very common in mathematics is that a lot of teachers believe that conceptual understanding must come before procedural skill, also not backed by evidence, but is likely in many cases a misunderstanding. So, we're going to get into these in detail, and you have some great examples on both sides.

Let's first talk about algebra gatekeepers. So, you published a piece by Janet Johnson and John Whittle. Can you just remind me who they are?

[00:09:22] Thomas Briggs: So, they research and evaluate federally funded education grants, and so they're the heroes that dig into the nuts and bolts of how these really complicated, arguably over- or under-designed systems basically work and perform on the ground. And they're really doing that important legwork.

[00:09:44] Anna Stokke: So, you published a piece by them, and it's about students in the U.S. being denied access to advanced math classes. And they're not being denied because they lack ability, but rather because of how schools are controlling entry. Can you tell the listeners about some of the details there?

[00:10:04] Thomas Briggs: Yeah, sure. I think this story, which is a story about North Carolina, but I think it's not just about North Carolina because a lot of states have really similar systems and similar problems. But it's a story from North Carolina that basically found that around half, maybe over half of students that were based on the proficiency and the testing results that they had, they were not put into the proper math track based on their ability.

So, it was around 35,000 that were predicted to succeed in eighth grade algebra, weren't enrolled on that track. And unpacking why I think gets to sort of the ways in which really complicated systems and really opaque procedures can sort of function as these maybe non-ideological kinds of mistakes or like hard points in the system that can make it go awry that people don't really often have an awareness of. And so, fifth grade teachers were making placement decisions before the test scores came back.

But the recommendation forms, for example, to get into the nuts and bolts, right, those forms included test score fields. So, from the start, you have a system that involves a lot of maybe like qualitative tests, like teacher recommendations, a sense of fit for the student that isn't really rooted in an empirical reflection of their ability, right? You had all of these little veto points combining to basically prevent these kids from getting into the classes that would have served them best.

Just a quick caveat, like not saying all of those kids would have done well in those math classes. There are other reasons to not put kids in the most advanced math tracks, and certainly some of those were valid, but the raw numbers don't really look good if your main goal is getting kids in the appropriate classes for them.

[00:11:56] Anna Stokke: Yeah. OK, so they had the opportunity for kids to go into the eighth-grade algebra class, which would be considered the advanced track. But the way they were determining who would get to go into that track was essentially based on teacher recommendations, correct?

[00:12:12] David Shuck: Correct.

[00:12:13] Anna Stokke: Yeah, which is maybe one of the worst ways to do it because, you know, teachers are human. They have biases, right? I talked to Jonathan Plucker on the podcast about this, and he mentioned that really the best way to do this is some sort of universal screening, you know, that everybody gets screened and if they are determined ready to move into that track, they're automatically registered and then, you know, they could opt out, but they try to get students registered in that track.

Now, how is this an example of a misunderstanding and what can be done about it?

[00:12:48] David Shuck: It's not as if we think these teachers are doing their best to prevent students who the data suggests would be able to advance and succeed from taking these classes. The goal is not to restrict achievement. The teachers who are promoting and preventing based on qualitative assessments are by and large teachers who think that there are some things that data simply can't capture and that their expertise, which they have often gained over many years in numerous changing classrooms, that expertise captures something that simple numbers can't.

Like the caveat Thomas mentioned, there are often reasons that don't relate strictly to ability, why maybe a student isn't ready to advance. It could be how often they get their work in, whether there are issues at home that prevent them from being able to focus. There are reasons why some students might not be ready to advance, but it is very easy for teachers to over-index on those issues or think that in a given student's case, those obstacles might be outsized compared to the stakes of holding them back or not.

So, we think most teachers, when they understand the numbers of how likely these students who achieve these certain scores were succeeding in eighth grade algebra one and advanced algebra courses thereafter, we think this isn't going to be something that teachers are going to resist for ideological reasons. This was not a case where the decisions, like many other cases across the U.S., this wasn't a case where something like equity was being used as a justification for keeping students out of taking algebra. For that reason, even though some teachers might be used to the old ways, still prefer a vision of themselves as knowing something that the data doesn't quite capture with more awareness, with a better understanding of what the data actually does, what it actually measures, the benefits of universal screening, and how equity is actually advanced when we implement universal screening.

Because often the students who were held back were students of minority backgrounds and some teachers didn't realize this. We think once teachers are more aware of this, change is definitely possible, especially when clear institutional initiatives are laid out that are easy enough for teachers to follow.

[00:15:29] Anna Stokke: I get the point here is that in this particular case, they just didn't know. Now, we're going to talk about a case in a little bit where students didn't get tracked into the advanced track, and it probably had a lot more to do with conviction or ideology. They have to hang around to listen for that.

That's going to come up in a bit. So, the other issue, as you've mentioned, is conviction. When educators and administrators are actually mistaken about what good research and evidence supports, they may be married to an ideology and see any challenges to that ideology as threats.

So, would you see this issue as much more serious than the misunderstanding issue?

[00:16:14] David Shuck: Absolutely. Conviction makes so many of these discussions and debates harder to resolve because it no longer becomes an issue of just comparing the evidence base, seeing what we have the strongest evidence for, the willingness to try certain interventions and see what works. Because in the case of conviction, evidence is looked at in a different way.

With honest misunderstandings, largely people are receptive to better evidence, even if it goes against their priors. There's no reason why they should object to it based on their underlying worldview. But when it's a matter of conviction, then there's an active resistance to evidence.

Even just hearing evidence that doesn't fit with one's priors can encourage one to come up with other non-evidence-based reasons why the person challenging their view must be incorrect. This is where we'll get critics dismissing certain issues as politically motivated. This is where certain ideals like conceptual understanding or equity end up being wielded in ways that don't reflect how those goals are actually brought about because if only intentions were enough, it wouldn't be a problem.

Many of these people who do have convictions want to change the world for the better, want to ensure that students are being educated to the best of their ability. But the intention doesn't reflect knowledge of the practices and whether those practices are efficacious. And when people react and point that out, conviction lets one focus on the intention itself, forget about the details and continue to dig in and continue to misrepresent what the evidence actually says.

[00:18:14] Anna Stokke: Yeah, that's quite true. And we've run up a lot against that a lot. And speaking of which, let's talk about the New York math briefs.

We've talked about the New York math briefs in some detail on this podcast. I don't know if you heard that episode, but I had Ben Solomon on as a guest to talk about his petition to retract the New York math briefs. David, you wrote a piece called ‘Dropping the Ball’.

And by the way, that's a reference to Deborah [Loewenberg] Ball, who is a math educator who heavily influenced those briefs. Can you first please remind listeners about the New York math briefs and what that controversy is all about?

[00:18:56] David Shuck: And you're right, you had Ben Solomon on, on episode 54, I believe it was, to discuss his petition and these briefs. Terrific episode, like all of them are. In this case, we had the New York State Education Department setting out a numeracy initiative.

And the numeracy initiative had some great intentions. Explicitly, the goal was for the initiative to highlight the evidence-based features and best practices of effective math instruction in a collection of math briefs. Now, these math briefs would be commissioned by a suitably vetted math expert.

And then distributed by the NYSED with the stamp of approval showing that they are, in fact, evidence based and that practitioners in New York and elsewhere should feel comfortable that the instruction recommendations in these briefs have a strong research backing, have been rigorously tested in controlled trials, have meta analyses vetting their effectiveness, and don't show any over-reliance on something like overly qualitative studies or opinions or myths that originate in opinion pieces or elsewhere. Now, it just so happened that the briefs that we got were not the evidence-based briefs that they were advertised as. Instead, Deborah Ball and the Teaching Works team put together a series of briefs that relied largely on qualitative studies on a research base that was not well vetted, that did not reflect what we know from the NMAP final report on math instruction, the practice guides from the What Works Clearinghouse (WWC) put out by the Institute of Education Science.

And these briefs also perpetuated several math myths like some of the ones we've mentioned already, namely the myth that time tests cause math anxiety, the myth that explicit instruction is only or most effective for students with special needs, but not all students. A dismissal of the importance of repeated practice of math facts and standard algorithms, and a suggestion that discovery learning should be prioritized early at the expense of explicit instruction. So that's the background.

When you had Ben Solomon on, we didn't yet have the pleasure of seeing the response that NYSED put out. And our piece was, yes, in response to that response.

[00:21:43] Anna Stokke: OK, can you tell me a bit about their response then? Because listeners will be interested to know what the response was.

[00:21:50] David Shuck: The response included, it began with a letter by Assistant Commissioner J.P. O'Hare and was followed by a longer response by the author of the briefs, Deborah Ball. So, the letter that begins is an interesting artifact because as it just about as soon as they can, the issue becomes a matter of political motivations. In the very second paragraph, the guilt by association moved is pulled out.

J.P. O'Hare writes that the letter and petition were circulated by Families for New York, a group and led by an associate of the conservative Manhattan Institute. Then the letter characterizes the concern that the NYSED is lowering standards as a defense of a system that has long excluded many students. They then align the NYSED intentions with the intentions of SUNY (State University of New York) Chancellor John King, who was at that time being challenged by the Trump administration for certain commitments to DEI policies.

This was an attempt by NYSED officials to show that we share the beliefs of this political figure who clearly shows a commitment to equity and principles we endorse. But all the while, none of this is actually addressing the central question. Do the briefs involve effective math instruction or not?

They attack motivations that the petitioners are said to have. But when we actually look at the petition itself, the language is completely different. The petition itself explicitly mentions in the first paragraph that we bear, and I quote, we bear the moral responsibility to equip all children with mathematical knowledge to give them access to their future lives.

By promoting research-based teaching methods, we can close opportunity gaps in student achievement while raising achievement for all students. Now, if that is the language in the original briefs that shows a commitment to equipping all children, and the all children is emphasized in the original, and a commitment to closing opportunity gaps, then this is just straightforwardly a misrepresentation of what the motives were.

[00:24:14] Anna Stokke: Oh, absolutely. I mean, that's hilarious. What the argument is that if someone actually supported the petition and they have particular political leanings, then the person who wrote the petition must also have those political leanings, right?

That's the argument? And you both have philosophy degrees. And I told you I took a lot of philosophy in my undergraduate degree, right?

It's my second favorite subject, actually. So, you know, this is a ridiculous argument, right?

[00:24:45] David Shuck: Very much so. The response started out in this terrible place, and that's even before we get to the meat of Deborah Ball's response herself. That's just from NYSED Commissioner, or Assistant Commissioner JP O'Hare.

[00:24:58] Anna Stokke: So why would they do that? And by the way, I know Ben Solomon fairly well, and I actually helped with that petition, I helped set up the website. And, you know, I'm not political.

I just want to see better math instruction. And the same is true of Ben. So, it's absolutely untrue, this accusation.

But my question is, why would they do that? What's the motivation behind trying to say this is politically motivated?

[00:25:27] David Shuck: One of the biggest reasons is just because of the rhetorical force that changing the conversation brings. I understand it was probably very inconvenient for many members of the NYSED who had commissioned these briefs, expecting there to be an expert who would actually present them with evidence-based instructional guidance. It was truly inconvenient for them to receive a petition that charges those briefs with being not evidence-based.

In the least and actually contradicting existing evidence. So, one motivation is simply in order to save face, in order to justify their own position and cover for the fact that maybe they should have invited a different expert to issue these briefs. The political argument offers a cover and then distracts from the content, right. If New York parents are thinking, wow, maybe this petition comes from a political place, they're likely to discount the meat of the argument. That is a time-tested tactic for someone who is not able to argue on the merits of a given argument.

When you can't point to evidence, you change the subject. So that's one of the biggest. But another reason is that for the proponents of these ineffective practices themselves, conviction often extends a political lens to every issue in the subject, whether or not they're intentionally trying to make the most rhetorically powerful counter argument.

For teaching work personnel, for many who truly believe that, say, discovery learning is good at the beginning of instruction, even for students who do not have the background knowledge that is necessary for discovery and career learning to be successful. Many of them believe not on the basis of evidence, but on the basis of people they've spent time with and on the basis of their history, beliefs about conceptual understanding. Many of them believe that they're doing what's just.

They are promoting good instructional practices. And there's simply no way to see somebody who opposes them as being motivated by anything other than some kind of political or nefarious purpose.

[00:27:41] Anna Stokke: So, you know, Ben was pretty clear on those briefs, not being supported by evidence. He gave the counter evidence, et cetera. What I'm curious about is the expert.

I'd like to know what the expert, Deborah Ball, said.

[00:27:56] David Shuck: This is one of the points where I got, I mean, it was disappointing enough to read the response. But one of the big disappointments, especially for somebody with a philosophy background and spent many years enjoying heated conversations with others and getting through like really well-developed arguments that develop different points that all support the same fundamental argument in their own ways. What we see in Deborah Ball's portion is a reliable reliance on what I called in the piece, the scope and methods ‘2 step’.

It is a tactic where step one is to define the scope of mathematics proficiency so broadly that the specifics discussed become narrow minded. What this means in the case of Deborah Ball and the petition is that Ball writes that the petition is only concerned with the evidence and research for effects of practices on procedural fluency. And as we know from artifacts like adding it up in Ball's point of view, procedural fluency is only one portion of mathematics proficiency generally.

So, of course, any evidence that only applies to procedural fluency can't also apply to something like conceptual understanding. So first, that's the scope step. The second step is the methods.

If the scope is broader than the evidence the petitioners bring accounts for, then, of course, the methods both to research full mathematics proficiency and the methods to teach to these different components must also be broader as well. In each of the three criticisms she identifies in the piece, her first criticism identified that the recommendations are not based on research. This was in the petition that the briefs advance an approach to testing math that's been shown to be ineffective and that the briefs don't align with the science of math or the science of reading.

For each of these criticisms, she relies on the same two step argument. The evidence brought to bear is too specific. The methods for teaching all of mathematics proficiency need qualitative kinds of evidence like conceptual analysis, and they need different kinds of teaching practices, many of which look like discovery learning and inquiry approaches.

[00:30:28] Anna Stokke: Okay, yes, I know this argument very well. Okay, you had a really neat way of explaining it. I kind of think of it more as they just usually, if you have some evidence that supports procedural fluency, they like to come back at you with, yes, but math is more than procedural fluency.

Math is conceptual understanding, problem solving and critical thinking. And then, you know, these are all things that are not well-defined, impossible to measure. And so basically the way that you get those things that are not well-defined and impossible to measure is you have to use the approach to teaching that they want you to use, which is inquiry based or discovery-based learning.

So unfortunately, it sounds like the New York State Education Department bought all this.

[00:31:20] David Shuck: There is no indication that they did it. In the letters defending the briefs, they make no indication that any of the petitioner's concerns should be taken seriously. All are politically motivated.

And for Deborah Ball's part in her response, she stands by every original claim.

[00:31:37] Anna Stokke: I mean, they also seem to fail to recognize in a lot of these situations, the procedural fluency is what really leads students to the point where they can actually do hard problem solving, that this is a very false dichotomy that we're often get thrown at us.

[00:31:56] David Shuck: One of the biggest issues with bringing conceptual understanding in in support of ineffective practices like discovery learning and against explicit instruction is that even when there is fairly good research that tries to operationally define conceptual understanding, there's a very clear bi-directional link between procedural fluency and conceptual understanding. They are hard to tease apart, but even when we can, it's hard to identify something like understanding that is not closely reliant on being fluent in the procedures. So, if there's some kind of way to understand without being able to solve the problems or consistently get the right answer, one, we haven't found a way of reliably finding it.

But two, it is a mischaracterization to suggest that there are practices that can promote one without the other. Practices that improve procedural fluency help students understand.

[00:33:01] Anna Stokke: YouCubed is another case you've published articles about. And for listeners who aren't aware, YouCubed is Jo Boaler's math project at Stanford. Jo Boaler's a math education professor that's fairly well known.

So, can you tell us about the problems that your authors discovered with YouCubed's methodology?

[00:33:24] Thomas Briggs: Yeah, sure. So, we had the pleasure of Rahim Nathwani writing a couple articles for us. But in his first article, he called attention to the fact that a paper and a chart in a paper that YouCubed was using to demonstrate its good results in Healdsburg Elementary School in California, he noticed that one of these charts was weird because it showed proficiency over the last decade in fifth grade going from like 24 to 75%-ish.

But the publicly available data showed that the baseline was 44% proficiency for that year. And so, he was like, OK, well, that's a it's kind of a big difference, right, that obtains to the potential efficacy of the program that's being advertised with that paper and with that data. And nothing was disclosed about what accounted for that difference, even though normally excluding students would push numbers up and not down.

So that wasn't really an option. And when Nathwani pulled actual cohort data, he actually started to see that recent cohorts showed declining proficiency the longer they spent in the district. And that was peculiar.

And so, part one got a good reception and it prompted a response. So after part one, they sort of quietly revised the paper. The original chart, I believe, said Healdsburg Unified School District.

And then the new chart, I believe, just said Healdsburg Elementary. They changed a couple labeling aspects of the chart but didn't really disclose the actual error that should have been disclosed. But after some exchanges with a teacher at Healdsburg who was a co-author on that paper with Bowler, it was revealed that the normal research method was sort of inversed.

So instead of trying to dispassionately look at how YouCubed was affecting math learning across cohorts and across grades, what happened was they noticed that a particular grade seemed to be doing really well after a little while. And they said, OK, we need to study this because there is something good that we can draw from this. Right.

And there are a lot of terms for that in social science research. Right. But you generally don't want to try to fish the results you want out of the sort of pool of data that you're trying to analyze scientifically in an unbiased way.

And so basically what happened is just a ton of confounders made the gains appear to be meaningful gains. There was a successful charter school that was closed, a higher performing charter school. There was a big fire and which led to school disruptions.

But there was also an English learner decline. And so if you have a lower percentage of students that have to be classified as English language learners in the lower grades, they're probably going to perform stronger on math and reading, etc., in the higher grades. And the last weird element that's just worth mentioning is that third and fourth grade dropped proficiency.

So it was just like a really confusing picture that the actual data painted when the original claim was that, you know, our implementation of this math system was what's driving these increased proficiency numbers.

[00:37:12] Anna Stokke: OK, so backing up a bit. So let me get this straight. So YouCubed was being used in a district, correct?

It was a full district. And there was a claim. Was this a paper that they published or was this just on their website?

[00:37:27] Thomas Briggs: It was a paper on their website, I believe. I don't want to say it wasn't published in a peer-reviewed journal. Right.

But this gets to the back to the sort of evidence problems in this field. Right. But it was presented as good research.

[00:37:48] Anna Stokke: Yes. OK. And teachers and parents wouldn't know the difference.

That's a lot of the time. That's the issue. But your author found that the numbers didn't seem to match up.

But there was some sort of inflation going on. And so, your author wrote on your Substack. And when that piece was released, YouCubed quietly changed the wording in the paper to make it more accurate because it clearly wasn't accurate before.

Is that correct?

[00:38:21] Thomas Briggs: One thing they did was they changed the label. So, on the chart. So, the original chart was labeled H-U-S-D Healdsburg Unified School District.

But really, it just showed data from one specific school, which was Healdsburg Elementary School. What they did was they updated some of the data. A key chart was replaced.

But really, they now introduced a sort of second problem, which was before there was just these numbers seemed wrong. But what the update revealed was actually that there were significant confounds underlying the entire story that are minimally cast out on the original claims and probably represent a good part of the explanation for that successful story. And so there was the closure of the charter school, as I mentioned before, and then also the English learner population change.

[00:39:20] Anna Stokke: OK, that's rather interesting. And is there anything else you want to say about that and the articles that you published on the Substack about YouCubed?

[00:39:30] Thomas Briggs: Well, I think there are a million different important takeaways I could list if I had infinite time. But a big one is just sort of how opaque and disordered the information environment is in this space. YouCubed is at Stanford.

Stanford know how to do this. They've done a couple of studies, right? But when you have these ideologically motivated researchers that are presenting schools, districts, teachers, officials who are often, I mean, COVID introduced, you know, stressors on school systems that they haven't seen in a while, right?

So often these are teachers or school officials or administrators that are pressed for time. They're dealing with new circumstances now. And there are all these sorts of unpoliced corsairs of curricular providers offering what they claim to be research-backed programs that work.

But really, it's just an informationally degraded environment that's hard for people to pierce.

[00:40:32] Anna Stokke: Yeah, and I think you made an important point there about this being out of Stanford. And I would remind listeners that Lucy Culkins was at Columbia. And these stamps of these institutions like Stanford and Columbia, when it comes to education, it doesn't mean that the information you're getting is accurate, unfortunately, as you've just shown us with this YouCubed stuff, right?

And that's not the first issue that's come up with some of that work. So, we do have to be really careful. But there's sort of this celebrity business in education.

And I think that's part of it.

[00:41:14] David Shuck: Yeah, we could quickly mention that this is the same YouCubed who just recently released an email attempting to discredit the science of math. And Greg Ashman made a really nice post drawing special attention to this line that they are opposed to all that we know to be correct. And then they list a series of math myths.

Also, the paper they link to by Raymond and Gunter is one that repeats similar tactics we've already discussed. It begins with a political charge linking the science of math movement somehow to, quote, the rise in authoritarianism across the globe, unquote, and then goes on to charge specifically that the authors of one of the fundamental science of math documents don't address any of the potential purposes of math other than procedural skill development. It's another move to, in this document, like any other, attempt to clarify the scope to make it easier to argue that this difficult to measure conceptual understanding is really what matters.

[00:42:29] Anna Stokke: Yeah, I found that really interesting and that YouCubed email was forwarded to me as well, and that was very telling because, you know, there's no evidence provided that all these myths are true, but YouCubed was claiming that we know these to be true, right? So, it's almost pulling off the mask. The mask is coming down a little bit that they're almost even showing you that this really is ideology and not evidence.

And yes, the paper they linked to was really wild. But I mean, I've gone through this with the NCSM statement against the science of math, and Sarah Powell and I talked about that on the podcast. And now there's a new paper written by another group, which is perhaps even worse than the NCSM statement, to be honest.

The science of math people are doing good work. A lot of people feel threatened by that work, particularly because of what happened with reading, and they'll just keep moving along and that's fine. Let's talk a bit about San Francisco.

So, I don't know if you know this, but when the California math framework was being debated, I had Brian Conrad on the podcast to talk about that. And I think we talked a bit about the San Francisco detracking experiment then. And I also had Jelani Nelson on, and he told me about the detracking experiment as well.

At that time, I'm not sure what stage we were at, but parents were suing. And I don't think it was really worked out at that point. And I think there's been some sort of resolution.

And so maybe we can wrap that up because you've written quite a bit about that. So, let's talk about it. Where did the arguments come from for detracking middle school algebra in San Francisco?

And they did it. So, what was the result?

[00:44:23] Thomas Briggs: Detracking has a really long pedigree in the education philosophy and research space for all the wrong reasons. The founder of the Center for Educational Progress, he writes under Tracing Wood Grains online, and his name is Jack Despain Zhou. He wrote a short piece for the Critic Mag about just the bad citations in keeping track.

It's phenomenally lazy with its citations, and it misreads all of the important papers that it cites to sort of advocate that tracking harms students. And it's, you know, you mentioned earlier that things like fame or star power or maybe we call it like prestige are like really do a lot more lifting in this space than they should. And so back in 2014, SFUSD and the school board basically voted to eliminate eighth grade algebra.

They wanted to put all the students in the middle schools on the same track. And the arguments for this are just sort of the same type of arguments. That we've seen, right. So, there's a moral argument that this would somehow be better for minority students or that this would somehow induce a sort of better character in the students overall by not having them compete or, you know, push too hard or something like that. And then in the meeting minutes, parents, you can see they very understandably were like, wait, what if my kid's really good at math?

What are they going to do in eighth grade? And as expected, detracking math and not offering algebra in eighth grade gives opportunities to give deeper learning about other aspects of math. Right.

And so that argument's been here for at least over a decade. It popped up in 2014. Jo Boaler endorsed the move.

This guy named Phil Darrow, who works, I think he does like California standards and then pretty prominent ed policy lobbyist. He consulted on the design and the political strategy. And the argument given was that they needed to do this to comply with new state standards, which is like a really weird argument when you look at all the surrounding districts and Oakland also detract.

But besides that, like none of the surrounding districts did. So, it just wasn't something that needed to be done because of state standards. And that's like odd that that reason was given when it just wasn't true.

And over the course of the decade that it took for this story to envelop, like every single link in the chain features some quintessential bad ed policy or bad argument or bad scholarship or reporting. So just to give a couple examples, when I think like four or five years in, they started showing some early promising results, right? Op-eds were written about how it's working.

And they argued that enrollment was up, that test scores were up now on advanced classes. But Tom Loveless had a great piece diving into the details of these claims. But basically they weren't good claims.

The UC system didn't rate the class that they scored more highly in as like an advanced class. Right. So, there wasn't data that it would that this policy change had produced better, ultimately better advanced learners.

Because that would be a compelling point, right? If in 11th and 12th grade, kids who were on the same track in middle school were actually doing better in AP Calc, right? OK, that would be like a good or that would be one good reason to maybe detrack in those years.

But, you know, they weren't doing better in AP Calc. They were doing better in classes that the UC system said weren't advanced classes. And there was like a weird PR campaign.

There's so much money spent trying to convince people of just bad education policies in the US. It's really striking. And in 2023, a Stanford study came out, I believe it was 2023, that basically showed advanced enrollment declined.

The paper says it almost got back up to where it was. But what happened to the students during that? Right.

Like you can't redo high school class progression once you graduate. So those kids are now that much farther behind in math. And throughout this time, it's important to remember that this policy was being advertised as a kind of experiment or like rigorous test.

Right. Because what they did was they set up a couple of different kinds of tracks at the different schools. But there wasn't great bookkeeping on, you know, whether requiring students to take two math classes was better than one.

Right. They had these various convoluted, different schema they were implementing across the district. And so, it was an unpopular, bad policy change carried out in an irresponsible way that was misrepresented and misreported by the people advocating for it.

And after a referendum to bring it back and a school board recall and dropping enrollment in the district by like 10,000 students over this year, which a whole bunch of other stuff happened. Right. It wasn't just because of math.

But after all that, I wake up a couple of weeks ago and see that a bunch of people are now misreporting practically on what the school board actually voted to do now, because they're not really bringing algebra back in a robust way. Only two of the 21 schools will have a stand that will have a functional standalone algebra track for next year. So, you know, like the New York Times and even more places seen as more conservative papers, right, are all sort of credulously reporting that algebra is back.

And I'm like, not really. Like it's kind of back in a couple of places. And there are these weird barriers of entry that they still demand and still keep trying to set up when they do bring it back.

[00:50:48] Anna Stokke: Oh, such a sad situation. And I mean, you've got to wonder what people are doing. Like they wanted certain things to be true, so they made it seem like it was true.

And in the meantime, how many kids lost out there, right? It's just you don't get that time back. You don't get that schooling back.

[00:51:11] Thomas Briggs: What particularly grinds my gears is that they implemented a bad, ideologically motivated policy that alienated parents and that harmed students, took these algebra tracks out of the middle schools. But then when they vote to bring it back, Superintendent Suit calls that a major milestone that continues to be rooted in equity. And at the same time, the moral responsibility to make sure you fix your own bad decision well is now what you use as like evidence that, well, we have to be careful about how we fix this because it's, you know, hard to reschedule all these students.

And it's like you caused the problem. You know, this problem didn't exist. And now you're hiding behind your responsible governance in order to not fix the problem in the way you should.

And I just think that's really wrong.

[00:52:05] Anna Stokke: And about the ability grouping or tracking, just to put that out there, you know, there is some evidence that ability grouping or tracking improves outcomes for high performing students, and it doesn't harm outcomes for students in low performance groups. Now, I talked about that on my episode with Jonathan Plucker. He talked about evidence.

I talked about that on my episode with Montse [Montserrat] Gomendio about high performing systems around the world. But tracking and ability grouping are criticized on the basis of inequity and stigma. OK, when that doesn't really actually seem to be accurate.

So, do you think it's actually possible to convince people that it is, in fact, likely better for students?

[00:52:57] Thomas Briggs: I think it is. And there are different ways you have to do it, depending on what sort of people you're talking to. Because if you're trying to move the sort of research or academia zeitgeist, then one thing that my colleague Jack, who's on leave from the organization right now, he dove into criticisms of tracking in DC schools when they were integrating in the 50s and 60s.

And what's strange is that the common line taken in the academy is that teachers say tracking harms the students put in the lowest tracks. Right. That was what was sort of the big takeaway from these big collections of teachers reports that were run in DC back in the day.

But when you go look at the actual like teachers reports, not the secondary reports on the teachers reports, you go back to the teachers reports, you find that the most common complaint in the lowest track was that the students couldn't do the material still. Not that it was treating them like they couldn't do anything and that it was like, you know, dooming them to some uneducated state. The biggest takeaway from the teachers was that it was still too hard for them.

Right. And so, I think there's the sort of evidentiary base point that, you know, is going to be persuasive to some. And also, just we have to keep hitting this point.

We have to develop a body of work that proudly presents the best evidence, which currently is some form of ability grouping or tracking is good. Right. And just build education, sciences and movements and successful schools loudly declaring that these are the policies that we use and they work.

When it comes to the more ideological angle, though. I don't think there's any short battle to win here, except that if you don't have effective practices and you don't lift all boats, you are harming all students. You are not helping some of them.

Right, and yeah, standing up for scientific practices, excellence as an educational value.

That doesn't just mean only the top one percent of students should get our care. That's it's a value that informs an orientation toward every student about the goal of the thing that they're there to do and that you're there to do. Right.

It's not some exclusive principle. It's a principle that should be applied to all students. And that, I think, is worth emphasizing because people like to see educational rigor and excellence and then say, oh, well, that's only going to help the most talented.

And it's like, no, it helps everyone.

[00:55:49] Anna Stokke: David, why does this happen in education, do you think? When it wouldn't happen in medicine, like a failed drug trial would prevent a drug from going to market. But education programs either don't get tested, or they just keep going even when they don't work.

Or there's that example with the YouCubed program and the very poor methodology. Do you have any thoughts on why this happens in education?

[00:56:14] David Shuck: It really is an incredible disconnect. Here, I'm pretty influenced by Douglas Carnine's work, both in 2000 with his Why Education Experts Resist Effective Practices and more recently, 2025 is Stopping the Pendulum. In his 2000 piece, he discusses how research-based professions, on the one hand, don't reform themselves voluntarily.

It's often in response to a crisis of trust. Certain other professions have had these crises of trust. When medicine adopts a new practice, it doesn't go back.

Doctors and surgeons have not stopped handwashing, have not stopped using stethoscopes. When we have the thalidomide disaster, the profession changes and progress happens. In education, there aren't the institutional guardrails in place to prevent the pendulum from swinging back and forth.

We go from traditional math to new math, then we have a back-to-basics movement. Then the NCTM puts out standards. In reading, we go from whole word to calling whole word, going to whole language, then going to phonics, then balanced literacy.

There aren't the guardrails to prevent this, like a shared knowledge base, evidence aligned teacher prep programs, competence-based licensure, or certain accountability mechanisms. It is also very hard to measure something harder to see. But we don't do a good job distinguishing between trust in character and trust in ability or competence in education.

Teachers are consistently rated toward the top of our trust ratings in America. When Gallup asked them, they scored a 61% at high or very high, second only to nurses. But whereas nurses have those institutional guardrails in place, and so we can relate things like trust in their character, trust in their honesty and being morally stand up, for teaching, you can be an honest teacher, you can do what you sincerely believe to be best, but that doesn't necessarily correlate to we should trust that you came from a development pipeline and program that let you know what the effective practices are.

[00:58:32] Anna Stokke: All right. So, you think it has a lot to do with the trust factor and that there aren't really great guardrails in place. So, Thomas, we'll give you the last word.

So, for parents or teachers navigating this and hoping to make shifts, what's the most practical advice that you'd give them?

[00:58:52] Thomas Briggs: Starting on the more practical side, right, I think for teachers, the research ed community, Zach Groshell’s work, your podcast, Math Academy, Karen Veitz's work, the word mapping project is an exciting new project. And I'm forgetting like a hundred people, but there are resources for teachers who want to ground their practice and evidence. The problem is pipeline and cultural diffusion and availability and compliance requirements, preventing good practices from maybe making their way in when people want them to make their way in.

And just being aware of those best resources is most important. And then for parents, like just inquire about what your school does, ask what the curriculum is for whatever grade level your kids are going into for math and reading, right, be curious about this stuff. And with those resources, they are possible to make their way through those docs and those websites and those pamphlets from the school, even if it can seem opaque at first.

And then a quick word about just the broader education ecosystem, because that also is in need of repair or a turnaround. Public momentum matters and speaking out in favor of common-sense policies, especially in spaces where that will make you unpopular or that you might get publicly maligned for doing that. Like speaking up in favor of acceleration in a Facebook group might have a bunch of moms and dads yell at you, you know, because that's supposedly bad for kids.

But speaking up in those places does make a difference because these bad ideologies do thrive on fake consensus. And so, so just, yeah, got to speak out.

[01:00:42] Anna Stokke: Yeah, I like that advice. You know, we do have to stand up for this. It's education.

It's important. It impacts lives in good ways and bad ways, right? And it's children.

So, I agree with you. I think that's something to keep in mind and to really take away from this conversation is ask questions and speak up and don't be afraid to do that because it does matter.

[01:01:06] David Shuck: I would only add that is exactly the lesson that we should take away from the piece you've written for us. Be proactive, ask, evaluate and get informed. Those are the central ways to combat the problem.

[01:01:19] Anna Stokke: Absolutely. And thanks for linking back to that. So, I am going to post a link to your sub stack in the show notes.

And I really want to encourage people to check it out and subscribe because you're doing really great work. And we need more of it in this space. And so happy to have you on today.

And I love this conversation. It was so fun. So, thank you so much for coming on.

[01:01:48] Thomas Briggs: Thank you. Thank you. Pleasure to be here.

[01:01:51] Anna Stokke: Thank you. Thank you so much for listening. If you enjoy this podcast, please consider showing your support by leaving a five-star rating on Spotify or Apple podcasts.

Don't forget to subscribe on your favourite podcast app or on YouTube so you never miss an episode. You can stay connected with me on Instagram, Facebook, TikTok, X, Blue Sky or LinkedIn. All links are in the show notes and check out my website annastokke.com for more information. This podcast is funded by a grant from La Trobe University and from the Trottier Family Foundation through a grant to the University of Winnipeg to fund the Chalk & Talk podcast.