top of page

Ep 24. Raising student achievement with Dylan Wiliam

This transcript was created with speech-to-text software.  It was reviewed before posting but may contain errors. Credit to Jazmin Boisclair.

You can listen to the episode here: Chalk & Talk Podcast.

Ep 24. Raising student achievement with Dylan Wiliam

[00:00:00] Anna Stokke: Welcome to Chalk and Talk, a podcast about education and math. I'm Anna Stokke, a math professor, and your host. Welcome back to season two of Chalk and Talk. During the break, my podcast surpassed 50,000 downloads. I want to thank you for listening and to encourage you to please continue to share the podcast. I've got lots of great episodes coming up.


This is episode 24, and I am thrilled to kick off this season with the amazing Dr. Dylan Wiliam. He is renowned for his expertise in formative assessment and has advised teachers, school leaders, and policymakers all over the world. He also has a background in math and math education. A lot of Dylan's research has focused on exploring strategies that are most likely to increase student achievement.


His recommendations are both practical and grounded in evidence, so I was excited when he agreed to come on the podcast to share his insights with me and my listeners. We begin by discussing PISA scores and the need to improve math proficiency. We then focus on Dylan's two primary recommendations: a knowledge-rich curriculum and investing in enhancing teacher effectiveness.


We also discuss assessment, both formative and summative. If you're not sure what that means, don't worry—we discuss that, too. Dylan provides some concrete examples of how educators, whether in school or post-secondary settings, can effectively use formative assessment to support student learning.


Along the way, we discuss math specialists in primary schools and what sorts of things teachers can focus on to have the largest impact. Dylan's explanations are clear and straightforward, and he provides a wealth of practical wisdom for teachers, policy makers and parents. I learned a lot from this conversation, both about teaching and improving education systems.


And I hope you do too. Now, without further ado, let's get started. 


I am honoured to have Dr. Dylan William joining me today, and he is joining me from Savannah, Georgia. He is an emeritus professor at UCL Institute of Education. He holds degrees in mathematics and math education and a Ph.D. from King's College London. He taught in urban public schools and directed a large-scale testing program in England and Wales. He has taught math education at the post-secondary level, and he has also served in several roles in university administration. He directed a math teacher education program, served as dean of the School of Education and assistant provost at King's College London, and was Senior Research Director at the Educational Testing Service in Princeton, New Jersey.


He has had a successful research career focused on supporting teachers to develop their use of assessment in support of learning. He has written over 300 books and publications, including the bestseller Inside the Black Box, coauthored with Paul Black, which reviews the research evidence on formative assessment. And he has worked with teachers and school districts all over the world on developing formative assessment practices. In fact, his work with a school was featured in a two-part BBC documentary called The Classroom Experiment, which I just watched recently. And I'm a huge fan of your work, so I'm really excited to talk to you today.


Welcome, Dylan. Welcome to my podcast.


[00:03:49] Dylan Wiliam: Great to be here.


[00:03:50] Anna Stokke: So before moving into educational research, you were a math teacher. So what grades did you teach?


[00:03:57] Dylan Wiliam: In England, you are either a primary elementary or a secondary school teacher. So if you're a secondary school teacher, you'll be teaching basically grades six or seven through to 12. So I taught 11 year olds up to 18 year olds.


[00:04:13] Anna Stokke: You've taught young children, you've taught what we call high school students, and you've taught math education at the university level. You're well versed in what happens in math education then. So you wrote several books. And one of the books you wrote is called Creating the Schools Our Children Need: Why What We're Doing Now Won't Help Much (and What We Can Do Instead). And I'd like to discuss some of the recommendations in that book.


So in the book, you discuss the need to focus on the things that will make the most difference for student learning. So before we get into that, should we be worried about raising student achievement? And particularly in math, should we be worried about that? Because I care a lot about math.


[00:04:55] Dylan Wiliam: I care about a lot about math as well, but I'm much more concerned with the lower end of the achievement range. So the PISA studies, this Program for International Student Assessment that rolls around every three years shows that in countries like Canada, like the U. S., like the U.K., something like 15 to 20 percent of children, of young people, don't reach the level of numeracy that's needed to participate effectively in society.


Those students are at a fundamental disadvantage. They're going to get ripped off, they're not going to understand the financial decisions they're taking. So for me, it's actually I mean, I love mathematics, I want to convince students of its beauty and elegance, but I also want to make sure that every single student leaving our schools is ready to thrive and flourish. And they can't do that unless they have a sound understanding of mathematics.


[00:05:46] Anna Stokke: Just to say a little more about that, performing below level two on PISA corresponds roughly to innumeracy, and I think we've seen increases across North America and in many other countries in the percentage of students performing below level two. Is that correct?


[00:06:02] Dylan Wiliam: Yes, I think that the long-term trends are actually not that strong. They're basically flat. And so you have some countries doing very well in 2000 when PISA first reported, then some declines and some recoveries. Other countries have gone up and then down. Several provinces in Canada had high achievement, but a lot of those have actually declined since then.


So, there's lots of people making up stories about why this is. But the real problem is you've got too few data points and too many variables. Nobody really knows what's causing this. We all have our hunches, and my hunch is that the kinds of things that we're doing to improve schools aren't helping.


And that's a problem because we do know what would help.


[00:06:46] Anna Stokke: Yes, for sure. And the other thing I look at are the top levels, the students performing in levels five and six on PISA. And at least in Canada, there's been a decrease in the percentage of students performing in those top levels too. And in some provinces, quite a large decrease actually. So we seem to have more struggling students and fewer students who excel.


Both of those trends, to me, are really concerning. 


[00:07:11] Dylan Wiliam: Absolutely. And I think that it's very hard to figure out what's going on. The thing about Canada is you have such a diversity of approaches to education generally where you have some provinces that are really in favour of inquiry-based instruction and others that are much more traditional, using formal examinations regularly throughout the province.


So I think it's hard to work out what's really going on. There's something problematic where we are failing our weaker students and we're not stretching our highest achievers. And that's a worry for me because those are the people who, that we will need to make advances in science and technology and engineering.


That's also a concern and other countries seem to be doing that much better.


[00:07:56] Anna Stokke: Yeah, those are our future STEM leaders. It's a concern for me as well. And, the thing I always say to focus on again is the long term trend, which is, I think what you're saying too, what's happening over time. We don't want to see more students struggling and fewer students excelling.


And when we're seeing that, we really need to, look at what's going on. And this is something you've studied a lot. You've looked really closely at what things schools and systems, education systems can do to improve. Let's start by discussing the two main things that you think to substantially improve educational achievement.


As I understand it, the first is a knowledge-rich curriculum, and the second is investing in teachers who are already in the system, and in particular, creating an expectation that all teachers continue to improve their classroom practice But before we do that, before we discuss those two things, what were your criteria for determining that these were the two most important things to focus on?


Was it research evidence, cost-effectiveness? What did you consider when you were thinking about that?


[00:09:04] Dylan Wiliam: It's just that. It's the research evidence with a view to cost-effectiveness. So we can always wish we had more knowledgeable teachers, so people point out that there's reasonably strong correlation between teachers’ content knowledge and how well their students do. But that's not relevant if you can't increase a teacher's content knowledge.


So I come down to this very simple idea, which is, let's 25 hours of professional development for every teacher every year. What use of that 25 hours is going to have the biggest impact on student achievement? So it's being really focused on “What can we do with the teachers we've got?” We could wish for more knowledgeable teachers, we could wish for more inspiring teachers, but nobody's going to improve a system by replacing the teachers with other people. 


You've got to work with the teachers you've got. And the question then is, what should we have them focusing on? What should we have the system focusing on?


[00:10:04] Anna Stokke: So the idea is to use your time as effectively and as efficiently as possible, with the research evidence in the background, with the research evidence in mind. 


[00:10:13] Dylan Wiliam: Right. 


[00:10:14] Anna Stokke: So let's start with the knowledge-rich curriculum. So what do you mean by that? What is a knowledge rich?


[00:10:22] Dylan Wiliam: I think I would summarize it by saying over the last 20 years in Canada and in many other rich countries, we wanted to give students -  to improve their ability to think. And so we've done lots of work to give students practice of thinking. We've had initiatives around thinking skills. And those haven't worked because, actually, students don't need more practice in thinking, they need more things to think with. 


So it's about equipping them with the concepts that they need to analyze problems. And, you know, we see this very strongly in science. The fact is that most of the models that we teach in secondary school science are not things that students will come up with on their own. We need to pass on the labours of past scientists say, “Look, you grab a door handle on a cold day. Your initial reaction is, ‘The cold is coming into my hand,’but then your training as a physicist kicks in, and you say, ‘Ah, no, that's not what's happening. What's happening is that metal handle is draining the heat away from my hand, and because metal conducts better than wood. That's why a metal handle feels colder than a wooden handle.’”


That is the result of training students to think like scientists, giving them these models to think with. And that's what I think we're lacking. We expect students to come up with these models themselves. And the real problem is this: our teaching is basically designed to be successful with high working memory students. In other words, we teach in a way that suits some students quite well. So the work of John Sweller has shown that exploration followed by consolidation works well for students with a relative degree of expertise. But actually, formal, explicit, step-by-step teaching works better for lower-achieving students.


And I think the real trap, and certainly, I fell into this as a teacher; I taught math using problem-solving because I loved the problem-solving approach. It took me a long time to realize that I was actually teaching in a way that suited my higher-achieving students better than my lower-achieving students. I was increasing the diversity, the range of achievement in my classroom because I was using approaches that tended to work for the students who were already good.


I was just basically increasing the variance. And that's natural in a way because most teachers are high working memory people. We are in the job because we succeeded in the academic enterprise. And so we teach in the way that we were taught, the way that we like to be taught. And the hardest transition I think that any teacher has to make is, your students are not like you. 


Your students may not actually benefit from the kinds of things that you think would work for you as a learner. And so that's what we need; we need to build up knowledge systematically, and we need to focus on the knowledge step by step. And I think we've often kind of resisted that. I mean, everybody accepts that in coaching in baseball or football. You build skills up step by step. And I think we've lost that. And it's basically because we think that the best way to get good at something is to practice the thing that you want to get good at.


So, if you want students to solve problems, the best way to do that is to have them solve problems. Actually, it's not. For a small minority of students, it may be, but for others, step-by-step instruction on the kinds of things that you do when you're solving problems makes students into experts more quickly than having them thrash around and flail and try to figure out solutions.


Because then, as John Sweller points out, when kids make their own connections, misconceptions are common. When the teacher builds up the knowledge much more carefully, step by step, misconceptions are relatively rare because the instruction heads that off.


[00:14:13] Anna Stokke: So you mentioned a few things there. I think you were talking about the expertise reversal effect a little bit, right? So the idea that for a novice learner, you need a lot of step-by-step systematic scaffolding instruction, right? Like a lot of direct instruction, but for an expert, meaning someone who has a lot to work with in their long term memory, so they already have a lot of knowledge, that approach might not be as effective. 


But what a lot of people are doing is they're using those approaches, those inquiry-based approaches that might work really well with experts, and they're using them with novices. And so the novices aren't learning well. Is that, is that a good summary?


[00:14:56] Dylan Wiliam: Absolutely, and that's what I did. I increased the variance of achievement in my classroom by adopting pedagogical approaches that were more effective with higher-achieving students. 


[00:15:05] Anna Stokke: I try to pay attention to the arguments I see out there, and a lot of times the people that are promoting inquiry-based approaches, they will say that they're really pro-equity. and in fact, it's the opposite. Inquiry-based approaches they favour, as you say, they favour people with strong working memories,  or students who already have knowledge that they may have acquired, say through tutoring or their parents. I mean, it's interesting because it's quite contradictory, calling yourself pro-equity in many ways.


[00:15:33] Dylan Wiliam: Absolutely, I've just been thinking about this recently. You know, all these people, they, they want to make sure that whether you're poor, or whether your family's rich, whether English is your first, second, or third language, what colour your skin is, should have no impact on your chance of being successful in mathematics.


And so people take those kinds of variables very seriously, but they don't take seriously the fact that some students, even though they've done nothing to deserve it, get given brains that soak up math, and other students get brains that don't actually soak up math quite as quickly. You know, the old eugenics argument was, well, let's not try to educate them, I'm saying that's not a recipe for a good society.


So the question is, what do we do faced with that diversity of processing speed, learning speed, that John Carroll identified in the 1960s? And Benjamin Bloom had a very clear answer. He said, “If we're getting a normal distribution of results, we're not doing our job properly.” Because if you treat students with different learning speeds the same, you will get a normal distribution. Our job as teachers is to destroy the normal distribution, to destroy the bell curve. We need to make sure that the students who need more support to thrive and flourish get that support. And they need to be taught in a way that works for them.


And that, I think, is a problem We've yet to face up to.


[00:16:57] Anna Stokke: And the other thing I think I sort of heard you touch on is what we refer to as the curse of knowledge, where it can be really hard when you know something to recognize that it's someone who doesn't know it can take a long time to learn it and that they need a lot of help to actually learn it.


And so we think that, you know, “I can solve a problem, I know how to solve problems, I have all sorts of techniques for solving problems and I want students to solve problems so that's where I should start.” Right? And we forget how much work it took for us to actually get to that point where we could solve the problems. Is that right?


[00:17:38] Dylan Wiliam: Absolutely. I used to, as you said, train teachers, train math teachers. When I used to interview these people for a place on the program, I would ask every single one of them, “What would you say to a 12-year-old girl who, when you explain that -1 x -1=+1, say, “Well, why?” In interviewing literally hundreds of students, I only ever got one good answer. 

Most of the time, they would just say, “It just is.” You know, “there are things you have to get used to in life.” And so, what I discovered was that a lot of the time, people who have been successful in mathematics, have been successful because they'd never thought about it.


They had just got successful at completing the dance of the numbers or the, or the letters, and didn't question it. And so, I think we do want students to understand what they're doing. But I think that, that understanding needs to proceed hand in hand with procedural fluency.


[00:18:37] Anna Stokke: Sure. And the other thing is you can teach understanding. I think a lot of people think in math that understanding comes through discovery, but you can teach people why things are true. And the teacher should actually know why the things are true in order to teach the students. 


[00:18:54] Dylan Wiliam: Yes, and I'm happy to acknowledge that if you discover something for yourself, It may result in a more permanent remembering of that. I remember I spent I don't know, a couple of months of Sunday afternoons working on the problem of which numbers can be expressed as a sum of two squares. And, you know, when I cracked it, I was just elated. I was over the moon. And so I will never forget that process.


But the problem is, many of our students don't have the time to discover things for themselves. And if we just let them discover things for themselves, they won't cover enough of the mathematics they need to learn. I'm willing to acknowledge that it may be better, I'm not sure about that, but let's say, let's stipulate that it is. The question is, “What do you do about the students who learn too slowly for that to be a recipe for success?”


[00:19:44] Anna Stokke: So we want to make sure that students have a lot of tools in their toolbox so that they can effectively solve problems. 


[00:19:51] Dylan Wiliam: Not just solve problems, but also think mathematically. I'm not worried about the minutiae or the intricacies of calculus, but I think this idea of rates of change, those kinds of issues actually sharpen our way of thinking. And so it's those tools for thinking when we, when we look at a problem like inflation. Understanding when, you know, when inflation is increasing, you're talking about basically a second derivative, but that kind of understanding comes from the big ideas of mathematics. 


[00:20:25] Anna Stokke: Let's move on to teacher quality and investing in teachers, which is something you've talked a lot about. So I've heard you mention that teacher quality is the single most important factor impacting student learning. Can you elaborate on that?


[00:20:42] Dylan Wiliam: Well, I think it's the single most important controllable factor. What determines how well students do at school? It's actually the students. So, these data come out slightly differently in countries like the U. S. and Canada, which rely quite a lot on coursework grades. And they can come out very differently in examination-based systems like we tend to have in Europe.


But Ian Deary at the University of Edinburgh found that you could basically account for about 80% of the variance in exam grades at the age of 16 from an IQ test at the age of 11. So basically, kids who do well - and it's not just because of IQ, it's also because those with high IQs actually are more likely to persevere. They're getting something out of their study, and therefore they're more likely to study. 


And so they're on a virtuous cycle, whereas students who find learning more difficult often get frustrated and they say, “What's the point? Everything I do I always get a D,” and therefore they try less. So, you know, we have to understand that. The variables we can control are a quite small part of the system. And my analysis of OECD data suggests that the school effect in Canada, the US, the UK, is about 10%. 


In other words, 10% of the variance in test scores is attributable to the school that the student attends. The other 90% is basically outside the school's control. So 10% doesn't sound like very much, but it is the only bit we get to play with. So that's why I focus on teacher quality. The fact is that some teachers are more effective than others. We've got some pretty good exam data from the Gates Foundation that shows that when you take teachers who are effective in one context and move them to a socioeconomically different context, they are still more effective.


So there's something that some teachers carry around in their heads that makes them more effective. And so the question is, what do we do on that one? One approach is, let's try to replace departing teachers with better ones. Then let's accelerate the process by firing the less effective teachers. The trouble is that it's very difficult to work out who the most effective teachers are because good teachers lay foundations for learning that are not realized for several years to come.


So that's how I come to this idea of the most important lever for improving outcomes for young people is to invest in the teachers we've already got. 


[00:23:13] Anna Stokke: So what are your proposals for improving teacher effectiveness?


[00:23:17] Dylan Wiliam: Well, we're looking for low-hanging fruit. I'm now more convinced that AI will make a huge difference in probably 20, maybe even 10 years time. But right now, it looks like there is nothing that we can do that's going to have a bigger effect than making, helping teachers make their teaching more responsive to student needs.


So I walk into classrooms, and I see a math teacher asking a question, and six students raise their hands, and the teacher chooses one of those students, and that student gives a correct response, and the teacher says, good, and moves on. Something that I used to do for years. The point I'm making is that if you're making decisions about the learning needs of a diverse group of learners based on the responses of the most confident and articulate students in that classroom, you cannot be making good decisions.


So the idea is that teachers will make better decisions if they have better evidence about what's going on in students heads. And by better, I mean better in two ways. One is deeper. So rather than, you know, asking students, “Which is bigger?” This is elementary school students. “Which is bigger, 0.25 or 0.3?” Why is that worth asking?


Because many young children believe that 0.25 is greater than 0.3 because 25 is greater than 3. So it's asking questions that elicit useful information and then getting information from the whole group, rather than just the usual suspects. So some people advocate cold calling. Cold calling, hearing from students who haven't raised their hands, is better than hearing only from students who've raised their hands, but it's still unlikely to be a representative sample.


So that's why I argue that we should be using universal response systems. And, of course, you can buy these so-called classroom clickers and all these expensive bits of technology, and I'm saying, no, don't do that. Just use finger voting. Just ask a multiple-choice question and ask the students to signal 1 for A, 2 for B, 3 for C, 4 for D, 5 for E.


And just read the class and make a decision about what to do next. You don't need to record it, you just need to make a better decision about what instructional next steps to take on the basis of the better evidence you have. And so, working on that aspect of assessment thinking about how we give feedback, activating students as owners of their own learning and as learning resources for one another, those things that we collectively call formative assessment.


And it looks like it's the most powerful focus for teacher professional development. A randomized controlled trial published a few years ago randomized 140 high schools into two groups. Half of them were just given cash, half of them were given resources for teachers to meet together once a month to work on their formative assessment practices.


Each teacher would decide, out of the menu of things that they'd read about, which one or two techniques are you going to prioritize. It could be wait time, it could be making statements rather than asking questions, which tend to elicit longer responses. It could be giving fewer grades and more comments about how to improve your work.


And the result of the trial, which focused on 9th and 10th-grade students, was that in the schools where teachers were doing the formative assessment, the students made 25% more progress over 9th and 10th grade, as measured by external examinations. the footprint of this was 1% of teachers' time. So by having teachers meet together once a month for 75 minutes, taking 1% of their contract time, it yielded an improvement of about 25% in student progress over a two year period. 

So that's why I'm now quite confident that it's probably the best bet. There may be other things out there that we don't know about yet that could be effective, but right now we don't know of anything that has a better prima facie case than helping teachers make their teaching more responsive to student needs.


[00:27:22] Anna Stokke: Let's dig into that a little bit more. So first of all, what is the definition of formative assessment?


[00:27:30] Dylan Wiliam: Well, to me, an assessment is formative if it forms the direction of future instruction. It's as simple as that. Benjamin Bloom thought about this as being short tests. And I know why he did that. Because, of course, learning is a change in long-term memory. In a lesson, you can't work out if the lesson has been successful.


You can't check the learning for a couple of weeks because what's important is whether it sticks. But what you can do is make decisions that are better founded on the evidence you have in front of you, but what's going on in students' heads right now? And so, I think the thing that Paul Black and I did in our 1998 review that I think surprised some people was that we were invited to review the research on classroom assessment, and we broadened it to include those day-by-day and minute-by-minute assessments that teachers make.


The sense that teachers make of students' responses to questions are themselves assessments. And they can be formative or they can be summative. So you can either decide, yeah, the students have got it. That's a summative conclusion. Or, this is what I need to do next. That's a formative conclusion. And the same assessment can be both summative and formative.


So let me give you an elementary school example. I give a young boy a test of his number facts from 1 x 1 up to 10 x 10. There's a hundred of them. I choose 20 at random and he gets 10 of them correct. So because I've chosen him at random, I can reasonably conclude he knows 50% of his number facts.


That's a summative conclusion. If I notice he's having difficulty with the seven times table, that gives me something to go on. That's a formative conclusion. So the same assessment, and even the same assessment data, can be interpreted formatively or summatively. So we think the best way to define formative and summative is not as kinds of assessments, but it's descriptions of the uses that is made of the evidence emerging from those assessments. 


So if I'm using it to improve instruction, it's formative. If I'm using it just to predict or to label or to conclude, that's summative.


[00:29:45] Anna Stokke: Okay, got it. And it's important to discuss the meaning of these terms, because as someone who actually came from outside education, a lot of times these terms people just throw them around and you don't necessarily know what they mean. And I used to think formative assessment really meant diagnostic tests.


Like diagnostic tests that you give at the beginning of the year to determine where the students are at. But what you're saying, it means it can be minute-by-minute checking in on the class, seeing where they're at, to seeing whether you're getting students understanding what you're doing to inform your instruction at the moment, correct? 

[00:30:24] Dylan Wiliam: Absolutely, but your example of diagnostic tests were also formative. The problem, of course, is that there's a really weird problem with diagnostic testing, which is in most of the domains that we actually use them.people would much prefer a diagnostic set of subscores. Rather than giving me a score for math, I'd much rather have a score for numbers and algebra and probability and geometry, for example.


Because that feels more useful. Turns out that the errors of measurement of those subscores are pretty large because they're based on far fewer questions. And so, in math, there is usually zero diagnostic information in those sub-scores. You'd be better off relying on the total test score, even if you want to make an inference about a particular sub-score. Everybody likes diagnostic assessment because we think that tests should do double duty. 


But in fact, in many domains, particularly in science and math, those sub scores have zero diagnostic value.


[00:31:26] Anna Stokke: So let's actually talk about some specific techniques that could help teachers, and I think the book that you recommend that would be really great is Embedding Formative Assessment. That's your book that gives specific techniques for formative assessment. So maybe teachers could use that book, but I want to say that I’ve been quite guilty of, you know, I teach and I think I’m doing a great job and I ask the whole class questions, and I teach at the university level, but it's the same idea, I should be thinking of ways to have formative assessment in my class too. 


And I ask a question and generally, you know, I might have 50 students in my class and I have maybe around 8 to 10 students who generally answer all the questions. And then there's the rest of the students. And you don't really know what's going on there.


So it could be that sometimes the students are too quiet to speak up, or it could be that the students are confused and I don't know and I move on because I think that I've gotten the right answer so everybody understands. So I’ve been trying to change that as well and one of the things that I've been doing, and I got this idea from Patrice Bain, who came on the podcast earlier is cue cards.


Every student in my class has these index cards. They have A, B, C, D,  I make up questions as I go along, multiple choice questions, and everybody has to hold up the card. One of the things you also mentioned was cold-calling. I'll admit, I'm very hesitant on cold calling. So you're going to have to convince me. I think, I have students that would literally drop the class if I started cold calling.  I don't know what to think about it, so maybe you can talk a bit about cold calling.


[00:33:12] Dylan Wiliam: Well first of all, I’m not sure that you know that your students would drop the class if you cold-called, I’m certain you believe that. There's a nice series of experiments by Dallimore and her colleagues where they looked at how, this is a college level, BS in accounting and management sciences. 


And they looked at the teachers who used cold-calling, and they actually found that students, originally, beginning of the study were really apprehensive. But the more they use cold calling, the more students participated, the more relaxed they were about cold calling. And in fact, the females and answered more questions than the males.


So this, this is a quite interesting research here. But yeah, absolutely, people are worried about cold-calling. So what we realize there's two reasons to do cold-calling. So Doug Lemov doesn't like the idea of random calling, because for him, cold-calling is a way, particularly in English or history, to bring in a student you haven't heard of. I think you might have an interesting perspective on this.


So you bring in a student who hasn't raised their hand because you think they might have something interesting or different to offer. That's a good reason. But I also value the increase in engagement. So, I actually haven't done this in a while, but when I used to lecture at the Institute of Education, I would lecture all our teacher training students, and we had literally a thousand of them, in a huge lecture theater.


And I would ask them about a study about the effects of feedback, for example, and I'd say, “What do you think happened in this study?” And there's five options, and I would ask them to use finger voting. And I would then pick one of them at random and say, “Why did you choose A?”, “You chose C, why was that?” I then interviewed the students afterwards.


I do this two or three times in a lecture, and I said, “Did the fact that there was a one in a thousand chance that I would pick on you for a response increase your engagement?” And every student I interviewed said yes. It's that dual role of getting information that you wouldn't get from the confident students, but it's also that idea that at any point you could ask a student to answer a question or to comment on another student's response. 


So that's one technique, is this idea of random questioning. But before we go into the others, I think there's another point I'd like to make, which is that our starting point here, in terms of professional development, is that each teacher should choose which techniques to work on.


So our process model is choice, asking teachers to work on the things that they want to work on, rather than trying to have every teacher do the same thing. Encouraging them to adapt them to fit their own classrooms and context. Allowing them to work on one or two ideas. For months, if necessary, and then making them accountable for making changes and providing support.


So the choice is really important, and so some teachers choose wait time, for example. Now, wait time is really interesting because every teacher I know knows they don't wait long enough after asking a question to give a student a chance to respond. And yet reminding teachers of that research has zero impact on practice.


It has less of an impact as reminding smokers of the harmful effects of smoking because it's not a knowledge problem, it's a habit change problem. And so one of the things I've been trying to do in my own teaching, now with, teachers, is I've been trying to move away from questions to statements. It turns out that if you make statements rather than asking questions, people tend to give longer and more thoughtful responses.


I'm not sure why this is, but my hunch is you can be wrong answering a question, you can't be wrong responding to a statement. So it's about having a mindset where I'm just thinking, “How can I get more out of my students? How can I get my students to share with me what's going on in their heads?” Because if I can't get that information, I can't teach effectively.

And so, for the last, basically, four years now, I've been trying to move away from questions and towards statements. “You chose A.” Rather than saying, “Why did you choose A?”, “You chose A. I'm interested in hearing a little more about that.” It seems a gentler approach. Making statements rather than asking questions. Focusing on wait time. Using multiple choice questions. One of the things about multiple-choice questions is that it's really hard to find good responses. So one of the things I like to do is to give students little 3in x5in little index cards, and you ask a question at the end of the lesson, so-called exit ticket or exit pass, and then you look at the most interesting incorrect answers, and you use those as distractors, as incorrect options, in your multiple choice questions.


You know, makes sure that the responses you're offering are things that students actually thought. Which makes the questions more revealing, and recent research has suggested that asking the question before you show the options increases long-term retention.


And very much more recently, asking students to rate their confidence that this response is correct when compared to the other options they've been offered, and interestingly, asking students to rate their confidence in their answer had zero impact on long-term retention. But asking them to rate it versus the other options seemed to increase the amount of engagement that the students had in answering the question, which, of course, then causes long-term retention. 


So, you know, there's lots of textbooks like this. There's, there's 50 of them in the book that you mentioned. People want the 50. Give me the list. And then they just choose one and try it once and go on to the next one. What I'm saying is, we need to be helping teachers accumulate. I've lost track of the number of times I've described a formative assessment technique to a group of teachers. And somebody says to me, “Oh yeah, I used to do that, it was good.”


“I used to do that, it was good, but something else came along, and I don't do it anymore.” So if we're interested in making this change sustainable, we have to give teachers time to make these techniques part of their classroom habits. So I'm very happy if a teacher works on wait time for six months.


Because once you get to that point, you never go back. And once you understand the value of wait time, you are generally horrified when you visit other people's classrooms and you see how little time students are given to think. So it's about changing that orientation towards, you know, “How can I get more out of my students? How can I get more coming back so that I can make better decisions about what to do next?”


And of course, them articulating their thoughts is also good for their own cognition. So it's that two-way process of me getting better evidence to make better decisions that results in better learning.


[00:39:54] Anna Stokke: And those are really great tips. And so you mentioned that the book has 50 ideas in there, try some of them and try to stick with them. I actually want to pick up on that point because. There's a lot of switching around in education from what I've noticed, so if people don't maybe stick with things long enough or they stick with the wrong things, and I've heard you say actually that it's self-indulgent for teachers to spend time getting better at things that actually don't help children learn when there's evidence about things that do help students learn. 


And I think that is actually a really important point because, and I don't always think it's the teachers who are doing this. I think it's sometimes district administrators bringing in professional development from people that are promoting methods that are actually unhelpful for learning. I mean, I see this all the time in Canada and so I'm wondering if you can elaborate a little bit on that.


[00:40:56] Dylan Wiliam: I think it's difficult because educational research is very messy. We don't have the clarity of evidence we'd like. So, the work of Robert Pianta at the University of Virginia, and Bridget Hamre, has shown that roughly 25% of students get reading in elementary school. Their progress is independent of the quality of teaching they receive.


So these kids just take off, they're called fast readers. For the other 75%, their progress in reading depends on the quality of instruction they receive. And so people who want to say, “Well, this approach really works,” can always point to a lot of students for whom it did work. And so, the stuff about inquiry-based versus explicit instruction.


You know, people will say, well, “inquiry-based is better”, or “explicit instruction is better.” We're now understanding, with a bit more nuance, that it depends the level of expertise the student has. So, in this situation, it works; in this situation, it doesn't work. And so, we have to be quite careful about what is our evidence base.


We get these things that are called aptitude treatment interactions. It works for some students and not others. We also have to be careful what kind of thing we're focusing on. So the work of David Geary contrasts biologically primary and biologically secondary knowledge.


So biologically primary knowledge is the knowledge of things that we are basically evolved to do. So, recognizing faces, speaking, and listening. You don't need to teach these things. In fact, they can't be taught. They're not innate, but they are learnable if students are exposed to a rich environment.


Other things, like writing and math, he describes as biologically secondary. And those things were not evolved to do, we need explicit instruction. So often, I see people arguing for play, but they're arguing for play in things that would probably be more beneficial with direct instruction, because they're biologically secondary.


On the other hand, I see people using direct instruction for things that really are biologically primary and therefore, you can't do much about it. So John Sweller suggests that transfer and generalization are biologically primary things. We are evolved to do those things. So what you need is not to teach people how to do this, but to expose them to an environment where they explore those ideas themselves.


So it's getting those things right. That's why the research is so inconclusive because everybody can find evidence to support just about any point of view in educational research.


I personally think that in most situations explicit instruction, direct instruction as Siegfried Engelmann called it, is highly effective. And in fact there's now research that shows that achievement causes motivation at least as much as motivation causes achievement.


The fact is that many teachers are faced with students who are bored and who'd rather be doing something else. And so, I think that our focus shouldn't necessarily be on achievement right now. The thing I'm convinced of is that 50 years ago, we could probably claim that students could learn everything they needed for the rest of their lives by the time they were 18.


Now we know that's not true. So, for me, it's much more important that our students leave us at the age of 18 with a desire to carry on learning. If you teach your students a lot of stuff but extinguish that passion for learning that every five-year-old has when they arrive in school, you have failed that student.


So I can see a rationale for inquiry-based instruction. First of all, in developing things like curiosity, which is biologically primary. And also, in just breaking up the pattern of study. So, for me, it's much more complicated because the aim is not just raw achievement right now. We also have to be making sure that our students are able to carry on learning and then also understand how to carry on learning themselves so they can learn how to learn. 


[00:44:59] Anna Stokke: You talk a lot about formative assessment, but what about summative assessment? Is summative assessment also important?


[00:45:06] Dylan Wiliam: It is. It's the backdrop. the nice thing about summative assessments, especially when they're externally set and graded, is they make the teacher into the student's ally. So let's take an extreme case. In the U. S., the grades that students get from their teachers determine which universities they get.


And therefore, students see their teacher as the enemy. The teacher gives you a B when you need an A to get into your college of choice. In the European system, also in Japan, there's an exam set by external people, scored by external people, and the teacher is your coach.


Like trying to get the athlete to clear the bar, a high jump bar at two meters. I've never seen a high jumper say to the coach when they fail to clear the bar and the coach says, “You failed.” I've never seen an athlete say, “Well, that's just your opinion.” It's an objective standard. And so the teacher, in that examination-based system can become the student's ally.


I think increasingly we're going to see AI making the traditional coursework grade-based approach increasingly impossible to implement fairly. It's now clear to me that AI can't detect AI. So I've tried doing things with AI checkers. I asked an AI package to write a an assessment policy for a high school, it did a pretty good job. 


I submitted that to an AI checker. It said there was a 75% chance it was produced by AI. I deleted the first sentence and then ran it again, and this time it was only a 4% chance. So basically, students can now submit their work to an AI checker, and it says, yes, this is going to get tagged as being AI produced, so what they can then do is put it through a paraphraser and students can keep on checking with an AI checker until this actually comes back as something that is not likely to get detected. 


So I think that whole coursework-based model is going to be impossible to maintain. And so we can have examinations, we can have controlled conditions, so students don't take work away from the classroom, but we actually have them do a piece of writing in a class where we can control access to AI, and the other approach we might be able to adopt is to have some way of judging the provenance of a piece of work. 


So we might ask students to submit drafts and see how they've taken the feedback they've been given and used that to improve. So that might also help. But I think exams are going to become more and more important because of the way that AI basically makes it impossible to authenticate work that students take out of the classroom.


[00:48:03] Anna Stokke: You mentioned an exam in, in the UK. Is it written at age 16?


[00:48:09] Dylan Wiliam: Yeah, so basically there's a, students will typically take 10 exams at the age of 16. English, math, science, history, geography, music, whatever, and then they will focus down. So in the last two years of high school, they will choose typically three or four subjects.


[00:48:27] Anna Stokke: And that's written by an external body and every student writes the same exam.


[00:48:33] Dylan Wiliam: There are competing boards, so groups of universities have their own examination agencies, but I think there's now, I think it's down to three now in England and , the rigour of their examinations is examined to make sure that they're not making the exams too easy to attract business.

[00:48:53] Anna Stokke: Okay. And what's the purpose of the exam? What do they use it for? What do they use the results for?

[00:48:59] Dylan Wiliam: Well, the exams taken at 18 were basically for university entrance. So, you get certain grades, so, you know, you apply to a university, and the university will say, Okay, yes, “we'd like you, and we will offer you a place if you get two grade As and one grade B in your three subjects at the age of 18.” These exams were originally written by the universities for university entrance.


The exams at 16 were originally a school leaving examination because many students have the right to leave school at the age of 15, then 16. But they are becoming a bit irrelevant now because de facto, education is extending to 18, even if kids get into work, they have a right to get some kind of work-based training.


So, there's a debate about whether these exams are useful because they're becoming an increasingly irrelevant punctuation mark in a 5 to 18 education system. 


[00:49:58] Anna Stokke: So, I mean, it's interesting because in some Canadian provinces, we really don't even have standardized, what I'd call standardized testing or very much of it. So, for instance, okay. BC, I don't think they have much at all anymore. There's not a lot in my province. I mean, for a university entrance it would be better if there was a more uniform exam that students took or just in general, I personally think it's a good idea because it checks how things are going, whether things need to improve and it's an accountability measure in some sense.


[00:50:35] Dylan Wiliam: Absolutely. And I think that the trap to avoid is what's happened in the United States, where they've gone down a kind of aptitude testing program with these, what used to be called the Scholastic Aptitude Test, which is now just the SAT. The problem with that is that I don't really want to measure aptitude, I want to measure math achievement. 


So I think that if we are going to go towards examinations, they have to be achievement-based examinations. Can students do the following kinds of things? Can they integrate 1 over 1 plus x squared or whatever, you know? That, to me, would be much more useful for the university to say, Yeah, this student is ready to go straight into our honours mathematics course, and this student probably doesn't actually have the background, and they will struggle if we take them. 


So that would seem to me to be much more useful to the universities, you know, in making these choices about which students to accept.


[00:51:26] Anna Stokke: You talk about teacher effectiveness and things we can do to improve teacher practice. And, I do want to ask you about one thing, because you've taught math education, you've directed a math education program, and I’m not sure what it was like in the UK, but here in Canada, it's actually quite common that a lot of primary school teachers actually struggle a lot with math.


And in Canada, primary school teachers teach all subjects. That's often the case up to even grade eight sometimes. And it has been suggested, some people will suggest, that it might be worth having math specialists teaching math in elementary schools. Do you have any thoughts on that?


[00:52:14] Dylan Wiliam: I used to believe that was true. I used to believe quite strongly that, say, for fifth and sixth grade, we should actually have math being taught by subject specialists. But I was really brought up short by a study from Houston where they actually discovered that the students who were taught math in 5th and 6th grade by their general teacher did better than the students who were taught by a specialist 5th and 6th grade teacher.


Now, it could be that those specialist teachers weren't that specialist. So I think that, again, these issues are very complex. Everybody decries the math capacity of elementary school teachers in most countries that I've visited. The interesting thing is, it's really hard to show a strong link between teachers math capability and the progress their kids make.


So probably the best work in this area is done by Deborah Ball and her colleagues at the University of Michigan, and they developed this assessment of mathematical knowledge for teaching. So it wasn't abstract content knowledge, it was do you know the subject at the level at which you're teaching it?


Do you know, can you explain why to divide fractions you invert and multiply? And what they found was that if you're taught by a teacher who scores one standard deviation higher than average on this test, you will make about two weeks more progress per year in math, which is not nothing. But given that the outstanding teacher, a teacher who is overall one standard deviation above the mean, produces 50% more progress, an extra six months of learning per year, it seems that either we haven't got a handle on what this capability is, or the effects aren't as strong as we thought they were. 


You know, given a choice, I'll take the teacher who's got the better math knowledge, but is it worthwhile improving their math knowledge? That is a question that is, as far as I'm concerned, still open. And we have one insight of this from an elementary school study that we did in England and it was just called, the report is available, it's called “Effective Teachers of Numeracy.” We looked to see what, which teachers in elementary schools were the most effective at teaching math. And we saw that the ones who were the most effective were the ones who saw math as connected.


So they didn't teach decimals and fractions and percentages as different topics, they taught them as being different ways of representing numbers that weren't integral. So, one of the things I would often ask kids to do is to is to put a half, 50% a half, 25%, and 0.36 on a number line. And, students are often surprised by that.


Because the idea, they've done number lines of fractions, they've done number lines of decimals, they've done number lines of percentages, but they haven't seen that all these are different ways of thinking about numbers that aren't integral. And so the teachers who were most effective were the ones who saw that kind of connection.


And in fact, we do know how to help them develop that kind of connectedness through professional development. So at the time in England, we had these what are called 20 day courses. So teachers could get day release, it's like, one day, a whole day release once every two weeks for a year and we worked on mathematical ideas with them, and that did seem to make a difference to how they taught math.


So it is, it is an example of changing teachers’ thinking about mathematics, that did seem to translate through into more effective practice in the classroom.


[00:55:54] Anna Stokke: And that makes sense. I mean, one thing I know for sure, you cannot teach what you don't know. We don't need studies to tell us that. If you don't know how to add fractions, you're not going to be able to teach someone how to add fractions. 


[00:56:05] Dylan Wiliam: Right.


[00:56:06] Anna Stokke: Beyond that, I do know that teachers need good resources. And I think that in a lot of cases that's not happening, like in some cases there aren't even any textbooks or any teacher's manuals and if math wasn't your strongest subject, to me, this isn't fair.


This isn't a fair place for teachers to be in, right? They do need really good resources.


[00:56:29] Dylan Wiliam: I agree with that, but I think I accept you can't teach what you don't know. The question for me is, how long ago did you first, did you first know it? So I think we have to accept that the teachers don't have the math knowledge we might wish, but good textbooks can provide that in a kind of just-in-time approach.


And I think what really distresses me is that even when textbooks are available, teachers are still downloading lesson resources from the internet. And the problem there is that they're just thinking about cool stuff for their students to do. And as I like to point out, a collection of resources, learning resources, is no more a curriculum than a pile of bricks is a house.


And the fact is that what we have is unsystematic development of capacity because we don't have clear learning sequences. And that's why I think textbooks are important. The problem in North America is that the textbooks are largely produced primarily from a commercial perspective. What we need to be looking at is what happens in Japan and Singapore, where there's a very strong input from government agencies about how to design really effective textbooks, so they tend to be thinner, they tend to be leaner, and just much better focused on what we know builds effective knowledge in our students. 


[00:57:51] Anna Stokke: So I'll ask one final question. So you've talked a lot about ensuring that teachers are constantly improving classroom practice. And we've talked about some of those things today. So if a district leader asked you what things can they do to ensure that that's happening, that teachers are constantly improving classroom practice, what would you advise them to do?


[00:58:16] Dylan Wiliam: It depends how much political room for maneuver they have. So, if it was a district that had a kind of step-and-lane compensation system, whereby teachers get an annual pay rise just for getting older, I would like to replace that by a requirement that a teacher demonstrates increases in skill or capacity.


So the idea, it's something for something. So the idea is, here's my proposal, and it may be too radical for many districts, every six months or so, a teacher would meet with a supervisor. The supervisor would say, “What are you planning on getting better at over the next six months?” The teacher would outline that.


The supervisor would say, “What's the evidence that that will make a difference?” So they'd have to engage with the research in some way. And then the supervisor would ask, “What evidence are you going to bring me in six months time to support your claim that you got better?” And so it could be videos of classroom practice, it could be test scores, it could be questionnaires with students.


But the idea is that the teacher chooses what to work on, provided they can say it's likely to make a difference, and they choose the evidence base that's going to support those claims. And then the supervisor says, “How can I help? What support do you need from me to do that?” And I got that idea from being a provost.


Because when you're a university provost, you end up line managing people who do work that you do not understand. So you say to the professor of organic chemistry, you know, “What's your plan for the year? How will I know you've been successful?” “Well, maybe six articles in refered journals and at least so many millions in grant income.”

“Okay. How can I help?” This idea that past three years of practice, and let's, let's support teachers closely in the first three years, but after three years, they should be ready to decide their own professional learning agenda, and we structure that, and we build a system whereby the teachers are responsible for making their own decisions about the ways that they get better, with a view to maximizing the benefit for their students.


[01:00:26] Anna Stokke: Very logical advice. I love it. Thank you so much for joining me today. I really enjoyed the conversation. There's going to be so many great things for teachers to walk away with from this. So I really appreciate you joining me.


[01:00:41] Dylan Wiliam: Thank you, it's been fun. 


[01:00:43] Anna Stokke: More in just a moment. As always, we've included a resource page for this episode that has links to articles and books mentioned in the episode, including a link to the BBC documentary on Dylan's work. I'll have another great episode coming out on April 12th.


If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Chalk and Talk is produced by me, Anna Stokke, transcript and resource page by Jazmin Boisclair, social media images by Nicole Maylem Gutierrez.


Subscribe on your favourite podcast app to get new episodes delivered as they become available. You can follow me on X for notifications or check out my website,, for more information. This podcast received funding through a University of Winnipeg Knowledge Mobilization and Community Impact grant funded through the Anthony Swaity Knowledge Impact Fund.

bottom of page