Ep 50. Universal screening in math with Amanda VanDerHeyden

This transcript was created with speech-to-text software.  It was reviewed before posting but may contain errors. Credit to Jazmin Boisclair.  

You can listen to the episode here: Chalk & Talk Podcast.

Ep 50. Universal screening in math with Amanda VanDerHeyden

Timestamps:

[00:00:00] Introduction  

[00:04:17] Overview of universal screening 

[00:07:51] Universal screening in math vs reading 

[00:11:36] How to find validated screening tools 

[00:19:32] Should screeners match the curriculum? 

[00:23:31] 145 math skills that forecast success and when to teach them 

[00:29:24] The problem with making screeners too easy 

[00:31:37] Is Acadience a good screening tool?  

[00:32:43] Comparing reading and math screening systems 

[00:37:09] Recap: What to look for in a good screener 

[00:40:12] Frequency and timing of screening 

[00:44:18] The growing importance of math screening 

[00:48:45] Addressing teacher concerns with screening results 

[00:52:31] Conclusion and final thoughts 

[00:00:06] Anna Stokke: Welcome to Chalk & Talk, a podcast about education and math. I'm Anna Stokke, a math professor and your host.  

Welcome back to another episode of Chalk & Talk. My guest today is Dr. Amanda VanDerHeyden. If you've been listening since the beginning, you might remember Amanda from one of my earlier episodes. I'm so thrilled to have her back, this time to help answer some of your questions about universal screening. 

In fact, I received so many questions on this topic that what was originally planned as a short mailbag episode turned into a full in-depth episode on screening. Amanda is a recognized expert on using screening tools to improve math instruction in schools. She's the creator of SpringMath, a K to eight math intervention program that includes screening, interpretation of the results and class-wide, small group, individual interventions.  

I'm excited to share that SpringMath has recently partnered with my favourite math program, JUMP Math. That means that SpringMath will now be available in both the US and Canada. You'll find links on the resource page to learn more, or you can go to JUMP Math's website to learn more about it.  

In this episode, Amanda discusses the value in using universal screening to identify students at risk of academic difficulties and how this fits into effective math instruction. She emphasizes the importance of using validated research-based screeners, screening at multiple points throughout the school year, and ensuring that screening is paired with responsive interventions.  

She also shares resources for finding reliable screening tools and cautions against using imprecise or oversimplified screening measures. This is a practical conversation that I think will be helpful to teachers, school leaders, and anyone interested in making sure every student gets the support that they need in math. I'm so excited to share it with you.  

Before we get started, I have a couple of quick notes. Chalk & Talk recently surpassed 200,000 downloads. Thank you so much for listening and supporting the show. If you find it helpful, please take a moment to leave a five-star rating on your podcast app. It really helps others discover the show. 

And if you know someone who could benefit from the conversations we have here, whether it's an educator, a parent, a policymaker, or anyone passionate about education, please share it with them. Your support helps bring these important discussions to even more people. Also, if you're enjoying Chalk & Talk, you might want to check out another podcast I've been enjoying lately:  Teachers Talk Radio. They've got a network of around 30 teacher hosts and they publish shows daily on all the usual platforms like Spotify, Apple, and YouTube.  

They cover anything you could think of that's teacher-related. Recently, I listened to Tom Rogers interview Daisy Christodoulou about AI and assessment. It was a fantastic episode and I learned a lot from it. So, give them a follow wherever you get your podcasts, or visit them at ttradio.org. That's Teachers Talk Radio. Now onto my episode with the amazing Amanda VanDerHeyden. Without further ado, let's get started. 

I am thrilled to welcome back to the podcast Dr. Amanda VanDerHeyden, and she is here to help answer some questions today. Amanda is a researcher and an expert in using screening measures and research-based methods to improve math instruction in schools. She has also built screening measures that are used in schools all over North America, and she's a founder of SpringMath. 

 Welcome, Amanda. 

[00:04:16] Amanda VanDerHeyden: Hi, thanks. 

[00:04:17] Anna Stokke: So, for these mailbag episodes, I like to tackle the questions that I receive most frequently from listeners. And something I've been getting a lot of questions about is universal screening for math. And Amanda's here to help me with that because she is an expert on that. So, here's an example of a question, and this one comes from Sandra. 

“I have a very good understanding of structured literacy instruction based on the science of reading. That said, I am still trying to understand how best to support teachers with math instruction. I'm curious to know if there are any universal screeners for numeracy for elementary students as there are in literacy.” 

So now keeping in mind we have a diverse audience, I'd like to just start with some basic questions first. So, Amanda, what is universal screening? 

[00:05:04] Amanda VanDerHeyden: Well, universal screening really comes from the medical world, and most people are accustomed to this when you have a newborn and you go for well baby visits and all the way through age 12. Every time you go to the doctor, and even as you become an adult, you're getting on a scale, they're taking your temperature, later as you become an adult, they're checking your blood pressure. 

Why do they do these things? Because there's a range of, when you test in the normal range, whatever's considered normal range, it's an indicator, it's a sign that you're okay, you are thriving, your baby is growing as they are expected to be growing. If something is not in the normal range, then that could be a fluke, it could be what we call a false positive error in the screening process, it could be nothing significant, but it also could be something really important.  

So, it doesn't tell us what is wrong, it simply tells us, whoa, Houston, there may be a problem. You might want to track this a little more closely, which this happens, of course in the academic world, or you may want to do some diagnostic assessment to rule in the presence of something that might be something that needs to be treated more cautiously by a physician.  

In academics we used to just sort of rely on parent concerns or a teacher nomination of a child as having an academic learning problem, and what we know is that that is highly unreliable. 

It turns out humans are not great about identifying who needs intensified instruction and they're also uneven, so some teachers might be pretty good at it, but that doesn't mean all teachers are good at it. So having brief sensitive measures of progress that tell us kids are on track or not on track, that's a really important warning sign to take a look at children who might need more.  

And in fact, this becomes the basis for multi-tiered systems of support or response to intervention. The entry point to that is always universal academic screening. 

[00:07:00] Anna Stokke: So, it's kind of like diagnostic testing. When you go to your doctor for instance, and they give you, they do the blood work like once a year or something like that, and it can kind of signal that there might be something that they want to look at and pay attention to before things, say, get worse. 

[00:07:17] Amanda VanDerHeyden: Yeah, and I guess I would say like a little tiny bit tweak to your analogy is more like if you're overweight or your blood pressure is high and then your doctor says, “let's do a blood test.” The blood test is diagnostic assessment.  

The screening is the first signal that something might have been wrong, which would be like, “I feel bad,” “my chest hurts,” “my weight is high,” “my blood pressure is high,” or like with a young kid in a well baby visit, “they didn't gain any weight between the second month of life and the fourth month of life.” That's a big thing to say, “whoa, let's look and see what's going on.” 

[00:07:51] Anna Stokke: So, when I get these messages from people, they tend to know a lot about universal screening in reading. And so, I think that's why they're sending the messages because they're looking for something similar for math. Is universal screening and math similar to universal screening and reading? 

[00:08:09] Amanda VanDerHeyden: Well, okay. I have a couple reactions to that. They say they know a lot about universal screening in reading, and they might, but they might not. And universal screening and reading is just sort of more ubiquitous, right? More systems are engaged in it, are doing it, but that doesn't mean just because they're collecting the data points that they're doing universal screening well, even in reading.  

Because someone has to be actually reacting to interpreting those data, identifying if there is a high base rate of risk, that sort of thing. And in fact, you know, Matt Burns and I published a paper several years ago in School Psychology and demonstrated that teachers sort of have this very individually determined, so it varies by teacher, appetite for academic screening in reading, our study was a reading study, and some teachers do a whole lot of it, and some teachers do less of it sort of naturally.  

That doesn't mean when I say “do it,” they're collecting the data points. And what Matt and I identified in this study is some kids were getting reading screening up to 12 times per year, and it was not associated with their level of performance in reading, meaning kids who had more risks were not being screened more frequently, it was not systematically related to student performance. It was instead determined by the teacher.  

So, we dug a little deeper and we looked at year-end reading proficiency across two consecutive years, and children who actually received more reading screening, and again, this is independent from their risk, those children actually had worse outcomes over time. 

So, there can be a cost to unnecessary screening, which I would define as sort of weighing the cow and expecting the cow to get fatter. If you're going weigh the cow, you need to be prepared to respond to what you learn or you have not actually completed the screening. 

[00:09:56] Anna Stokke: Right. There's got to be follow through is what you're saying.  

[00:09:59] Amanda VanDerHeyden: This is a big problem in the United States for sure. But yes, there's, we can definitely connect, we can make some analogies to reading, which will help this be more palatable to teachers. But teachers who are listening should understand that, you know, just because you collect an oral reading fluency data point, which does tell you a lot about a kid, you have to be willing and able to act on that for it to actually be a screening. 

Like one of my favourite quotes, Mark Shinn says, “Don't screen if you won't intervene.” 

[00:10:26] Anna Stokke: That might be one of the reasons there isn't a lot of universal screening, say in some Canadian provinces. It's going to cost money to intervene, they're going to have to provide evidence-based resources to intervene. There's going to be a lot of things that come after that. 

[00:10:42] Amanda VanDerHeyden: They're learning a lesson from the United States because in the United States what we've done is we fed this giant appetite to screen, and when I work with systems all the time, I'll say, “Okay, what are you using for your screening assessments?” And oftentimes they'll name two or three. 

And I'm like, “Why are you doing that?” You're collecting the same data using. Measures for which the scores are highly correlated. They're not giving you unique information about children, and you're really making a relatively simple decision. So what you really want is the briefest most cost efficient, meaning it takes the least amount of time to know is the child at risk? Thumbs up: yes, thumbs down: no.  

That's it. That's the only decision you need to make. And then allocate your additional assessment time to the children who you determine to be at risk, or the classes of children that you determine to be at risk so that you can make better decisions over time. 

[00:11:36] Anna Stokke: You have to have a good screener first of all, right? Are you willing to share some? 

[00:11:42] Amanda VanDerHeyden: It won't even be my opinion, although my opinion is going to track very closely to this, but I would send people to the National Center for Intensive Intervention in the United States. That website is www.intensiveintervention.org. They operate some tools charts. Have you heard of those, Anna? 

[00:11:59] Anna Stokke: No. 

[00:12:00] Amanda VanDerHeyden: Okay, so they actually have intervention tools charts too. I mean, SpringMath is very proud to ever be rated on those tools charts because their rating process is the most rigorous, their technical advisors are named right on the site, these are the reviewers for the various tool charts, and they really are the best in the business.  

I mean, these are people who really do know this technical work, this particular brand of science. These are world-class methodologists. The rating criteria are published. They are, so you should know as somebody who might submit to this site exactly what your rating will be on the other side when you submit because the rating criteria are that clear. 

They are extremely rigorous. There is a process that's very transparent, so you know that when you submit, you are really submitting very raw, actually raw data. You're in some of the cell tables that you have to submit for the screening accuracy analyses, you are actually submitting raw counts so they can duplicate what you report as a researcher.  

And then they, you get an initial review, you have an opportunity to rebut that initial review, but then the way they rate you is final and it's out of your hands. It looks very simple on the surface. So, you can scan the list, you can filter the list by grade level, and you can filter by reading. You can filter by behaviour, you can filter by math. 

And this allows you to see which commercial screenings are rated with full bubble ratings on specific criteria, and which are rated on sort of half or empty bubble criteria, and then you can click into the actual assessment that's listed there and you can read the actual data. It is beautifully transparent and accessible. 

One of the things I get frustrated with in the United States is now every individual state wants to publish their own set of approved screening tools. To my knowledge, no state has ever used criteria that are more rigorous than NCII and it's a little frustrating for people like me because I'm like, why don't you just accept the NCII ratings because they are superior in terms of their review practices to what any state can really duplicate and roll up.  

That's my frustration because getting states and state review teams to actually understand the actual research evidence you're trying to share is often difficult to do, but intensiveintervention.org, click on the tools charts, click on academic screening, and you can filter by grade, you can filter by content area and you can see which are, which are the best.  

I will tell you in reading, you just cannot beat oral reading fluency. You can get so much from oral reading fluency, you know, it is a one-minute assessment. You can repeat it if you want to. Although you can make valid screening decisions based on a single trial, which is a beautiful thing, and it's not what we thought in the early days, but we now understand that. 

And in math I do have some specific ideas about math screening. I believe that there is no oral reading fluency equivalent in the space of math. Instead, there are brief computation skills that are extremely meaningful at specific windows of time, given the typical sort of course sequences or skill progressions that we see used in math. 

[00:15:17] Anna Stokke: What are some specific ones you'd recommend in math? 

[00:15:21] Amanda VanDerHeyden: Well, obviously I recommend mine. I mean, that makes me a little bit biased, but I would also say, you know, my particular approach to math screening is different than many of the others that are listed on the site. So, you know, ours, we're using mastery measurement. It's a form of curriculum-based measurement, which is just like oral reading fluency as a form of curriculum-based measurement. 

But mastery measurement says we measure you on this particular skill right now, and when we expect you to be at mastery on that skill, we change the skill, and that's different. So, when you have something like oral reading fluency you can use that and it's often called, so it's a, like a capstone behaviour, a capstone skill. 

It's a general outcome that you can target all kinds of things like very specific sound pack blends or, you know, whatever you're targeting in reading, it can be very, your diagnostic tells you what to target when you're in the land of intervention, but oral reading, fluency improvements will gain in a somewhat linear pattern from within a season. So fall to winter is linear, winter to spring is linear and you can track this growth and that little metric remains useful to you sometimes, you know, for years.  

But in math we don't, we don't have anything like that. And there was an effort to, we all tried to build quasi general outcome measures in math, and usually what that looked like was a measure that had multiple skills on the same page, okay. Always timed assessment, of course, which is superior to accuracy metrics and there's articles I can send you to for that for screening. But in terms of mastery measurement, we are emphasizing, this is the key, is that you can track linear gains over time, you can make very sensitive decisions about who is on track and who is not on track, who is at risk, who is not at risk, but that measure is not as useful to you for as long of a period of time. 

There's multiple skill math measures that might have addition, subtraction, you know, three digit addition and subtraction with and without regrouping. So they mix up the problem types on the page. The problem with those is that the way they were always designed, and the only way they could be designed was to contain problem types for which children had never been instructed yet. 

And this theoretically would reflect and does reflect growth as children gain instruction in the different problem types, but the number of problem types you can use on a math assessment is constrained to approximately five and you can use some specific population, like there are some studies that show the way you set those up on a page matter, but at the end of the day, they all have the same problem, which is there is a tremendous lack of sensitivity at screening because only a small handful of items that a kid can encounter are actually things they've ever been taught how to do.  

So, in other words, the entire group that you are screening with that measure early in the instructional time period that you're assessing, everybody's scores will be low. Well, when everybody's scores are low, you have a constrained distribution. This is absolutely a hard and fast limit on being able to detect and use that measure to sensitively detect who is really at risk because everybody looks roughly the same. And then the other problem is sensitivity related to progress monitoring.  

So, if the measure contains five problem types that children, four of which they've not been taught how to do, yes, as they are taught how to do them during the course of the year, you'll see this very gradual upward trend. But it is a very, very subtle, low rate of improvement, meaning it's hard to know if your intervention is surpassing that or not. And let's say you've got this mixed probe with all those types of problems that are like, you know, multiplication, division, some fraction work, whatever, but you're actually intervening on addition because you, because children don't know how to do that. 

So, you have to go way back to an earlier skill that's not even reflected on that measure. So, using that measure to track whether or not the intervention is working is not sensitive enough. Does that make sense? 

[00:19:32] Anna Stokke: Yes. I mean, I think it's kind of related to something that I wanted to ask about, because a lot of times when people write, they'll say, for example, I'll just read another one. So, this is from Jennifer: “I'm wondering if you have any recommendations for a universal screener for math that matches the BC or Canadian curriculum.” 

So, I've actually received quite a few from Canada like this, and the person says, “I have been exploring Acadience Math, but noticed it does not match our curriculum.” So that would be an issue, right? 

[00:20:02] Amanda VanDerHeyden: Well, okay. I think this is like, again, we can go back to reading. There was a time that every teacher in the planet believed in the United States. “How in the world can a one-minute oral reading fluency, words read correctly per minute on a passage, be useful at all as a reading screening. That's not what we are teaching in reading. We're not just teaching kids to read connected words in print or phrases or sentences under one-minute timed conditions.”  

So it is something that takes a little bit of processing and thinking as teachers to understand that a useful screening assessment doesn't have to look like everything that's important instructionally. 

In fact, you don't want it to, it can actually lose its power. So, for example, let's go back to the sort of heightened weight things that we check for little babies when they go in for their well baby visits. Well, how much a baby weighs is not telling you everything that's important about child development. 

It's simply a sign that development may be off, and it's a very powerful sign because if a baby is not gaining weight, there could be a number of things that are going on with that baby, right? And academic screening works the same way. It is not a summative year-end test of mastery of everything that has been taught, nor is it an upfront pretest of the things that will be taught to see what children do or do not know.  

And sometimes you'll hear people suggest that as well, “Maybe that's what we should do for screening. We should take everything they were supposed to learn last year. We'll assess them on that. And if they didn't learn all of that stuff, we know they're in trouble.” 

Actually, that is a really. well, I'm going to say labour intensive and probably error prone way to conduct screening. You're much better off to say, all right, let's see, like in math, if you are in, you know, second grade anywhere, you know, I would say North America for sure, but around the world, you know, there are certain keystone numeracy understandings that we can test with you that if you cannot do, if you can't do those things well, it tells us there's a potential, a number of things you do not know how to do in math that are going to get you in trouble. 

Okay, so like, let's just use a classic one. Like let's say you're second grade, in the fall and children cannot add and subtract zero to 20. If children cannot add and subtract zero to 20 by, you know, winter of second grade, so in that fall instructional period, the end of that is winter. If they can't do that, we know they're really in trouble because the second half of second grade is really going to be about adding and subtracting, understanding place value, and if you haven't mastered sums and differences, you're already in trouble. 

We actually can, you know, this, this stuff is pretty powerful. We can forecast, like, based on those early behaviours that you're going to be in trouble when you encounter multiplication, you're going to be in trouble when you encounter fractions, which shouldn't, that's not a surprise to you, right? 

[00:22:58] Anna Stokke: Not at all.  

[00:22:59] Amanda VanDerHeyden: Because when kids can't do fractions, this is often like the first sign everybody goes, “Oh, Houston, there's a problem.” But at fifth grade, the fact that kids can't do fractions, those kids in kindergarten could not rapidly discriminate quantity and associate symbolic numbers with quantity, and they could not combine quantity, separate quantities, you know, the things that are important in early numeracy. These, these measures are very strong forecasters. 

[00:23:31] Anna Stokke: I have a question about fraction addition. Like at what grade level or age level would these screeners want to see that kids can actually do fraction addition? 

[00:23:44] Amanda VanDerHeyden: Okay, so here's, here's what I want to say too, and it's kind of related to curriculum and different expectations maybe in different countries. So my work has existed since before we had common core, even we didn't even have state standards in the United States when I started doing this work. And so what I want to say is like, you know, state standards were not handed down from God. 

They were not identified as, you know, some codified real thing in nature. These were created by humans. And often you can imagine like the sort of sausage making behind the scenes politically, like the geometry people at the table want the geometry skills listed. 

And you know, there's some, these were just created by people and they're fallible. So none of us should say, “Well, let's choose everything we do based on a set of standards that some people got on a committee and agreed about.” If the California math framework doesn't teach us that, what will, right?  

But at the end of the day, we can all take a step back and go, if you want children to learn and become proficient in basic algebra skills by grade eight, then there is a way to parse out the skills that children need to get there. So, for me, from kindergarten to grade eight, there are 145 key skills that will take you there. I mean, we articulate these on our website. I can send you our graphic and our list of skills by grade level, happy for you to share that, it's public facing.  

I might not have it totally right, this is my own work, people may disagree with me. That's okay. But I like to think about it that way because when children truly master, meaning they can, they can easily and accurately give the correct answers and understanding for 145 skills from numeracy to algebra, then those children are on track to thrive. 

And most of the advanced math performance we're going to ask them to do is going to be combining or using those foundational understandings, those 145 skills. So, what happens is like you get these teachers in grade five trying to teach a more advanced skill, and what they do is they go, okay, “kids can't do this. Well, let me hammer how to do this thing.”  

And what they need to do instead is break that down into what are all the component skills that are necessary for a kid to be able to do that thing. And that's what you build to fluency. That's the 145 skills, which by the way, addition with fractions is one of the 145 skills for us, of course. But when you think about what goes into that, so children need to know how to—in fact, I can send you our little decision tree and show it to you, but because we've mapped that out, I've mapped that out.  

But basically, addition of fractions requires that kids be able to understand the concept of shared common denominators, right? You got to put the fractions on an equal footing, as we say. So that involves a lot of understanding around multiplication, the identity property of multiplication, and then children need to understand the concept of adding, and then they need to understand the concept of simplification if you want them to give you a simplified fraction on the other side. That's it. 

But these, it turns out these are really foundational skills that you really are working on with kids that what's necessary to do successful edition of fractions in grade five, you're beginning that work as early as kindergarten. 

[00:27:06] Anna Stokke: Okay, so you're saying in a way that the screener really shouldn't have anything to do, or it doesn't have to have anything to do with what we'd call in Canada, the curriculum. You'd call that the state standards. 

[00:27:18] Amanda VanDerHeyden: Yeah, I don't want to say, doesn't have anything to do with, shouldn't be driven by. Okay, because I'll tell you what I noticed in Canada that I think is problematic. And not, I mean, I know it's different in every area, but in Ontario we could talk about Ontario. I think part of the problem is the expectations are rather light K-3, and then everything is expected grade five, six. And it doesn't work that way.  

You have to, you know, repetition is the key to learning and everything builds. I mean, I quote you all the time, Anna, I love how you say that that math is relentlessly hierarchical, and you return to the idea and you add a little bit on, you return to the idea and you add a little bit on, and it's cumulative. 

That's how math works. So yes, we don't want to be adjusting whether we measure kids are on track or not based on a set of standards anywhere that may change, you know, may change with the politics and change with other things, but may not be exactly right. But we do want, this is my caveat, we do want to give screenings in math that are rigorous. 

And so remember when I talked in your first question about the lack of sensitivity. Okay, well most commercial screening tools that on math screenings that come out of the United States have really, they are aware of this problem. They know they have constrained distribution problems. You're trying to conduct academic screening at the beginning of the year when you have time to actually intervene. 

If that measure does not become sensitive until you're at the end of the year when all the skills that are reflected on the page have been taught, then you are identifying kids too late. It's just not useful as a screener. Most commercial screenings in the United States have tried to address the constrained distribution problem at initial screening, so that lack of sensitivity by using easier measures. So then suddenly these measures are not really reflecting grade level content or grade level expectations.  

And the problem with that is teachers see what's on the screenings. This affects what they think we should be teaching, okay? So if you just make the screener easier, what you do psychometrically is you force a more normal distribution of scores, but our data show, and I've done a lot of research in this space in academic screening, that you will actually make worse decisions. 

So you can't use this easier measure, force a normal distribution, lob off your bottom 20% and identify kids accurately who need intervention to avoid academic failure because you often, in that context you have, you may have in most places in the US do have a high base rate of risk. Now, what do I mean by that? 

You've got a lot of kids, most kids look at the national assessment of educational progress scores in the United States. Most children are not proficient in math. So that base rate threat is not something you can ever avoid in academic screening by giving more measures, giving an easier measure, waiting it out.  

When you have a high base rate of risk you must use a process to lower the base rate of risk to improve your screening accuracy. Even the most, you know, sort of Superman measures in screening, the highest performing, most accurate measures in screening, if you use them in a context of most of your kids are at risk, you will miss kids. You will make the wrong decisions. 

So I use some slides all the time to show this story using a metric that we use in the research world called negative post-test probability. It is systematically related to your prevalence or base rate of risk in your environment and that it dramatically climbs, meaning you're missing kids who are going to fail the year-end test. 

Your screening will miss them. In a context, even with a 90% sensitive, 90% specific measure, you are missing a very high number of children who, who need a high percentage of children who need intervention to avoid academic failure. You'll never detect them.  

So, you have to use a process that lowers your base rate of risk, and then you enable your very accurate screening measures to actually function the way they are intended, and they can become very accurate for you.  

[00:31:37] Anna Stokke: Can I ask about a specific test that people keep asking me about? It's Acadience. 

[00:31:43] Amanda VanDerHeyden: Yeah. Okay, so people like it because it's free, all right. And I like the Acadience people. People don't know the history. Acadience comes from DIBELS. So Roland Good was one of the original, and Ruth Kaminski were the original builders of DIBELS in the United States, the University of Oregon. 

And then there was a hullabaloo come apart. I don't really understand all of it and don't need to or whatever, it's unimportant, but there is still DIBELS proper that does exist, you can use that. But there's also Acadience and Roland Good and Ruth Kaminski went with Acadience. So, these are very sort of, I would say very similarly designed. 

Similarly, you know, philosophied measures, academic measures. But, you know, I hope I don't offend people when I say this out loud because I grew up in the world of curriculum-based measurement too, which is what those measures are, and what I'll say is most commercial screening systems that use curriculum-based measurement research, which is most commercial screening systems in my mind, were really screenings that were built for reading.  

These were reading researchers. These people were publishing 20 reading papers to one math paper. Math, to me, was always sort of an afterthought. And I can remember like when we submitted the SpringMath measures for consideration for the Texas list, when I say we, that means me, I don't know, 10 years ago, we have the strongest screening data of any math measures.  

Okay, but we were rejected and we were not put on their approved list all those many years ago because we didn't also assess for reading. Like, well, because we are SpringMath. If you want to do reading, just go do oral reading fluency. Those are a dime a dozen. There's millions of 'em, right? Millions of ways to do that. But so sometimes conventionally, what people say is adequate and should be used is driven by things that have nothing to do with screening accuracy, right? 

Like, if you want to screen well in math, the first thing you got to make peace with is there's no world reading fluency. It's not going to look the same way that you do it in reading. So I'm going to put Acadience in that category. I'm going to say Acadience really is a system that was mostly designed like many of the others and to really do reading extremely well. And math is a little bit more of an afterthought. And I think some of what, you know, here's what I can say that's good about it, it's free.  

What I'll say that's objective about it, I mean, well also, here's what I can also say is good about it. They're really nice people. These are great researchers who really are involved in, and, Stephanie Stollar was a leader at Acadience for a long time, and I just love her. 

So, it's not personal, but what I'll say is, you know, go to the NCII site intensiveintervention.org and, and here's the thing, if your screening system is not listed on that site, that's a red flag. Because if they had the data, they would be. I mean, I have a conflict of interest. SpringMath is a commercial product. 

Anybody who is a vendor of a tool understands it's really important to be on that list. So, if they're not on that list, it's, there's a good chance it's because they don't have the data. Now, if they are on that list, then you don't have to take my word for it, or Anna's word for it or anybody's word for it, you can actually scan what's there and try to make a decision.  

Level two in your decision about what to adopt is how much time does it take? How expensive is it? How connected to the intervention solution is it? Because in my mind, like we will, I think our screening measures are the best around. We won't sell them separately from the rest of the MTSS process because as Mark Shinn said, if you, don't screen if you won't intervene. 

So for us, it's always in tandem, “Okay, you have a class-wide problem here, let's start class-wide intervention.” And that's part of, for us, that actually becomes the second screening gate. So Acadience is free, is pretty good for reading, for sure. For math, I just encourage you to go look at the data. 

[00:35:44] Anna Stokke: I know what you're saying, Amanda. Another assessment I've been getting a lot of questions about is MathUP. Is MathUP on that list? 

[00:35:53] Amanda VanDerHeyden: MathUP is not on that list. 

[00:35:55] Anna Stokke: That's a red flag, right?  

[00:35:57] Amanda VanDerHeyden: To me, it's a red flag. Again, submit your data. It's not personal. Submit your data. These are stewardship decisions that teachers are making, and what gets underplayed sometimes is, again, let's go back to the study I did with Matt. What was meaningful there is there's such an investment of time and bandwidth and energy sometimes to conduct these academic screenings and doing the screening alone is never goint to help kids learn more. 

You have to take very specific improvement actions based on what you learn. If you're not harvesting your universal screening data at the grade, the school, the class, the student level, but then using it in program evaluation to drive and evaluate your improvements and gains over time you are not doing universal academic screening.  

And if you've got some measure that nobody's ever heard of and you are buying that training teachers, encouraging teachers to spend time collecting those data, if it doesn't link directly to specific instructional actions that you can then use the same measures to evaluate whether or not you've had improvements, then at the end of the day, you are not being a good steward of the investment of the instructional resources you're responsible for as a leader. 

[00:37:09] Anna Stokke: Okay. So, I'm going to recap what I think is what I'm hearing here. So, first of all, we can go to the website and we can look up the universal screeners there. And we would want to pick one that's highly rated.  

[00:37:23] Amanda VanDerHeyden: Yep. 

[00:37:23] Anna Stokke: Right? That seems to be the most reasonable thing to do. And worry less about it matching a curriculum because there are certain skills, just like in reading, there are skills that any student should have that we know would be correlated with later success. And the same is true for math.  

[00:37:41] Amanda VanDerHeyden: It's actually better than trying to say, “this is what I'm teaching this week, so I'm going to use that as some” —you know, there really are important technical criteria to be useful as a screening device. And we, you know, as psychologists, it is part of our ethical mandate, it's our, it's an ethic of our practice, it's a tenet of our profession that we should not collect and report data using measures that have not demonstrated they are technically adequate for the purpose for which they are being used.  

That, if you are a psychologist and you are making up measures using measures that do not have adequate published technical data, then you're actually engaged in a form of malpractice. That is just the truth, you know. And I hear this all the time, like even sometimes on the Science of Math Facebook page, which can be great.  

And sometimes you get people like, you know, Sarah Powell and Matt Burns actually answering questions in real time and it's very cool, but sometimes there's an awful lot of non-science on that page too. 

And sometimes people will say things like, “oh, just go build your own measures.” Actually don't go build your own measures. It's technically complicated, it takes years if you're going to do it right. Children should not be subjected to unconsented experimentation. If you intend to use the scores to make decisions, you need to adopt something that has been shown how to work, that it does work. 

Like, I mean, being somebody who has built measurement systems my entire career, I will tell you when we built the assessment generator that we use to build, to run academic screening and math K to eight, we generated and tested over 45,000 problems. Just in that process, I mean, this is 10 years of research really just on the assessment, so to say to systems “just go build your own.” 

Like “you can find any worksheet, any kind of problem arrangement, print it off the web.” I mean, when I debated the guy at the Educational Writers Association and we were debating conceptual versus procedural, which blah makes everybody crazy, he said that in response to an audience question. 

He's like, “oh, just find any old practice materials off the web. There are a dime a dozen.” I said, whoa. I respectfully disagree. I mean, there are well constructed problem sets and there's a lot of technical know-how and skill that goes into that. 

[00:40:12] Anna Stokke: Now that we figured out how we choose a universal screener, I think that you've made that very clear, how frequently and what times of year should a teacher use the universal screener? 

[00:40:25] Amanda VanDerHeyden: You know, I'll tell you what we do. Because you know, again, I've published in screening more than I've published in anything. And I'll say because it is the starting plac, it's so important. Sometimes when I go to wonderful meetings like researchED, I do think, “Oh my gosh, that's the missing piece here. Nobody's really talking about assessment.”  

You know, assessment, formative assessment. It begins with screening. Who's in trouble? Who's not in trouble? And then is our instruction, meaning making them not in trouble anymore. That's the goal. That's what we want to be after in this academic, sort of iterative, assess, teach, assess, cycle that we all use. 

So we want to do screening in the fall and we want to do screening in the winter. And we do not really need to conduct screening for the purpose of risk identification and intervention delivery in the spring, but we often, and we do collect a data point in the spring, a screening data point in the spring because we are using it for program evaluation. 

Okay, screening all gives us a dual purpose: it's identification of risk at the system level, at the individual student level, and it also gives us adata point to, because we collected on every kit every time, it gives us a really important data point to evaluate program health over time, okay? And if systems did this, leaders would make much better decisions about where to invest and how to spend their limited instructional investments. 

So we want to collect fall screening data, winter screening data, most importantly, the first question we want to ask in screening is, “do we have a class-wide problem?” And you could define, here's our definition for it: A median score that is in the risk range at the class level. Okay? 

If you have a median score that is in the risk range at the class level, you have a class-wide problem. You need a second screening gate. If you have a class-wide problem, it does not matter how accurate your screening is, you're going to make bad decisions. Okay, you're going to miss kids and you're going to identify too many kids. So, what you want to do then is you want to use class-wide intervention as your second screening gate. 

That's what we do. So what we do is, do you have a classified problem? Yes, no. If they, in most places are yes, if the answer is yes, then we want to route you into class-wide intervention and now we are harvesting in real time the growth of every kid in that class grouping, given the same dosage of the same standard protocol intervention layered on top of the same core curriculum delivered by the same adult, with the same fidelity.  

And when we do that, it's really powerful. If, as your peers in that class grow, everybody grows. The median hits mastery. Now we don't have a class-wide problem anymore. If you are still personally in the risk range, our accuracy at saying “you're in trouble” is superior to any screening accuracy decision we could have made with a static screener. 

So fall screening, if you need class-wide intervention, you go into class-wide intervention. Class-wide intervention is always better than static screening, so once you have class-wide intervention in place, you're going to harvest that to determine who is at risk, this is how we do it.  And every subsequent screening, winter and spring then is for program evaluation. 

It's not, it's just to harvest, like general growth and improvement. If you do not have a class-wide problem by some miracle, then we'll use your static screening scores in fall and winter to sort you into intervention groups, okay, or individual intervention and spring screening for us, we collect it, it's rigorous by design and we only use it for program evaluation. 

[00:44:18] Anna Stokke: Is universal screening in math used less, is it done less than in reading, would you say?  

[00:44:25] Amanda VanDerHeyden: About 50%. But what's happening now, so in other words, about 50%—most schools are conducting universal screening and reading and have been for some time. But about, I would say about half of those schools are conducting, and that comes from some somewhat dated studies now, about 50% of those schools were actually conducting universal screening in math. 

But here's what's changing. So in the United States, you know, the science of reading took off. The science of math is sort of trying to grow up in its shadow and, and say, “Hey, what about”— because learning is learning is learning. It's really, in effect, it's all the same science. I publish in both spaces. 

Matt Burns publishes in both spaces, is a good example. Robin Codding, my colleagues do. “Behaviours, behaviours, behaviours” Ben Solomon would probably say, like I say, so at the end of the day, this, in the United States, there are, there's lots of legislation either pending, recently passed that is going to be requiring numeracy screenings, math screening in many states. 

So it's coming, it's coming. And what's going to be tough for places is to figure out how to proceed. And I wish, I always wish there were an easier way to help people avoid the mistakes that can be avoided in screening. Like, for example, don't select a system of screening just because it happens to give you reading too. 

It might not be adequate for math. Be sure when you select the math screening. That, yes, you look at the technical data, but you also want to look at how much time does it take to administer because that goes into your cost, you know, if it's, you know, CBM and math is usually two minutes for most students, two minutes per measure. 

We collect about three to four measures per screening occasion. So all in, you're under 15 minutes for a whole class. You can do it as a whole group and then. The most important thing is you need to be sure that your math screening is connected to a mechanism that will address a high base rate of risk. 

And for me, that means classified intervention is going to be your second screening. Our measures, if people want to know what I recommend for the static screening piece, those are public facing. You can navigate to those right on our website. It’s springmath.org, click on “How it works,”, scroll down and click on “Screening assessments,” and we give you a table for K through grade eight, fall, winter, and spring, we tell you exactly which measures we use.  

Now, of course, ours meet technical adequacy criteria, they're generated. They're not, they're not static in the sense that you always get the same problems in the same order, we actually use a generator for that. And you didn't ask it, but I will say it, for high school, we do not begin with academic screening in MTSS, and that is true in reading and that is true in math. 

And the reason we don't do that is because by the time children get to high school, which in the United States we call grade nine, this is like the really the 10th year of school because kindergarten is our first year, then in grade nine we have so much data about children we can really use how they have done on year-end state test up until that point.  

Children generally have already been tracked by the time they reach high school in the United States. So, we have remedial classes, we have advanced classes, that sort of thing. So, we don't really need to begin with universal screening, but I'll tell you how we do it because we do, SpringMath is K-12. 

In high school, we start with a static sequence of class-wide math intervention. We navigate this with the leaders directly to determine do they want to offer this in their remedial tracks, or do they want to offer this whole-school? And based on those data, we can very sensitively determine which students are in trouble. 

So, we don't, we do not begin with static screening in high school. Unfortunately this is one of those things where it is a little easier to sort of cover this with graphics, you know, and show people real data. However, you know, there are some kind of short articles I could send you that are practitioner-friendly that, you know, I've written and other, maybe other people have written that will help maybe help some of this make more sense.  

It, it's just tough in math because most screening researchers love to study reading, and most screening papers, chapters books will over focus on reading. And math is a little bit of an afterthought.  

[00:48:45] Anna Stokke: And I just want to mention another piece from Jillian and Lauren that I received and they said “Our junior teachers are concerned that some of the results they're getting from an assessment, a universal assessment, don't correlate to the Ontario curriculum, or the assessment doesn't correlate, and they were alarmed when it showed a high number of students at risk in their math skills, even though they are performing well on curriculum-based measures and in-class assessments.” 

So what do you think about that?  

[00:49:16] Amanda VanDerHeyden: Here’s the thing, I mean, it’s important to, so teachers get caught up sometimes about, especially math. They'll go, “Oh gosh, that's a really hard measure. If I give that measure to my students, they're not going to do well.” And this can make teachers a little squeamish. So, you have to do a little bit of conversation about this is, screening is low stakes. 

It is a low stakes assessment decision in the sense that you're not going to take a screening result and say, “this child is fundamentally flawed forever,” right? “There's something wrong with the child.” That's not the point. The first purpose of screening is to evaluate roughly, you know, are they on track or not? 

So it's important that you use a level of rigour that forecasts that if kids can do it at this particular time in their mathematical skill development that they're going to be likely to attain the future skill expectations. If this paces ahead of your curriculum a little bit, instead of being afraid of that, have some curiosity, embrace that. 

What does that potentially say about your curriculum? I mean, the higher goal here is not to adhere to the curriculum necessarily, although I understand the importance of that. The higher goal here is to open the door of math proficiency to all students. 

[00:50:38] Anna Stokke: Yeah, exactly. That was really important, what you just said about we want them not to be afraid of the measures, like, we want kids to succeed in math.  

[00:50:48] Amanda VanDerHeyden: Because, so think about it. All the other commercial math measure people, they understand the lack of sensitivity. So, what they've done to deal with it is they've all made their measures easier. Okay. Now, in our SpringMath, and teachers are like, “What the heck? This is so much harder than Acadience.” 

And we're like, “It doesn't matter. It's just an entry point, but it's an entry point to grade level success.” Like it's a meaningful, that measure is not meaningful because, like, okay, let's just push this to the margins. If a measure is too easy and we use it as a screening, yes, we will get a normal distribution of scores, but guess what? A kid can pass that screening and still be in trouble because the measure was too easy.  

That's the worst type of error to make in screening. So, when you, I think you said early on like, well, the curriculum, it doesn't, we don't care if it's aligned to the curriculum. Well, we do in a sense, we want it to be rigorous grade level expectations, but the higher kind of goal that we're after is, here's where all kids start, here's where we all want them to land, let's pace that out in more equal bites per year and make sure kids are tracking, roughly on track.  

And we get this all the time. Sometimes our class-wide intervention sequences have skills that technically or, you know, at the end of the sequence are a little bit ahead of grade level expectation. That's by design because what we're doing is smoothing out an expectation that jumps dramatically across years. So that's okay. We can do these things because we get a better result. 

[00:52:31] Anna Stokke: Okay, Amanda, I think we've answered these questions and really, you've done a fantastic job and I'm so lucky to know you and to have you to talk to and to come on and share this expertise it's really helpful for teachers and the people listening and I really appreciate it. 

[00:52:49] Amanda VanDerHeyden: Yeah, my pleasure. And I'm happy to send you a couple of Research Gate links for a couple of kind of practitioner-friendly articles on math screening in particular. 

[00:52:59] Anna Stokke: You bet. We're going to put all that up on the resource page today, so thanks, Amanda. 

[00:53:03] Amanda VanDerHeyden: All right, thank you. 

[00:53:08] Anna Stokke: If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Chalk and Talk is produced by me, Anna Stokke. Transcript and resource page by Jazmin Boisclair, subscribe on your favourite podcast app to get new episodes delivered as they become available.  

You can follow me on X, Blue Sky or LinkedIn for notifications or check out my website annastokke.com for more information. This podcast received funding through a University of Winnipeg knowledge mobilization and Community Impact grant funded through the Anthony-Swaity Knowledge Impact Fund.