Ep 58. When a mathematician became education minister: Nuno Crato on transforming education
This transcript was created with speech-to-text software. It was reviewed before posting but may contain errors. Credit to Canadian Podcasting Productions.
In this episode, Anna Stokke welcomes Dr. Nuno Crato, research professor of mathematics and statistics at the University of Lisbon. He was Portugal’s former Minister of Education and led major education reforms from 2011 to 2015. His education reforms contributed ot Portugal’s strongest-ever results on international assessments like PISA and TIMSS. Nuno shares his belief that everything starts with the curriculum and stresses the importance of clear standards, accountability, and student support. He also explains how Portugal raised academic rigor while helping struggling learners. They also discuss what Nuno’s research on PISA and TIMSS reveal about the characteristics of effective education systems and the lessons policymakers can draw from Portugal’s success. This episode is a must-listen for educators, policymakers, and anyone with an interest in improving education.
This episode is also available in video at www.youtube.com/@chalktalk-stokke
Nuno Crato’s website: https://www.nunocrato.org
TIMESTAMPS
[00:00:22] Introduction
[00:05:12] Becoming Portugal’s Minister of Education
[00:08:51] Adjusting to the position of Minister of Education
[00:13:07] The landscape of the Portuguese Education System, early 2000s
[00:19:08] The importance of a good curriculum
[00:23:07] Using TIMSS data to structure a knowledge-based curriculum
[00:26:22] Using testing to ensure curriculum goals are met
[00:29:11] Concerns about project-based learning
[00:34:18] Standard assessments and evaluations
[00:39:31] Accountability markers and incentives for standard assessments
[00:43:08] Addressing testing and accountability criticism
[00:44:59] Becoming informed about evidence-based pedagogy [00:48:23] What PISA data reveals about effective types of instruction
[00:51:04] Does more money spent on education translate to better outcomes?
[00:52:15] What changes are more likely to lead to better outcomes?[00:54:58] Results of PISA and TIMSS in 2015 Portugal
[00:59:53] Pushback to education reforms and where it came from
[01:04:54] Advice to policymakers
[00:00:00] Anna Stokke: Welcome to Chalk & Talk, a podcast about education and math. I'm Anna Stokke, a math professor and your host.
Welcome back to another great episode of Chalk & Talk. This episode is available in both audio and video. Please give the show a follow-on YouTube in addition to your podcast platform. You'll find a link to my YouTube channel in the show notes.
Now, before we get started, I'd like to acknowledge new support from the Trottier Family Foundation through a grant through the University of Winnipeg to fund the Chalk & Talk podcast. This helps me to continue to share high-quality, evidence-informed content that connects research with practice to strengthen math education.
I've got a really great episode today. My guest is Dr. Nuno Crato, Research Professor of Mathematics and Statistics at the University of Lisbon and Portugal's former Minister of Education from 2011 to 2015. He led sweeping reforms that transformed the country's education system and dramatically improved its international standing in PISA and TIMSS.
If you're not familiar, PISA assesses 15-year-olds in reading, math, and science and TIMSS measures grade 4 and grade 8 achievement in science and math. Now in our conversation, Nuno shares how he was unexpectedly called to serve as Minister of Education during a financial crisis and why he accepted the challenge. We discuss his vision that everything starts with the curriculum, how he refocused schools on fundamentals like reading and math, and the role of standards, accountability, and student support. He explains how his reforms raised rigor while also helping struggling students. And he shares what his statistical analyses of PISA and TIMSS reveal about the characteristics of effective education systems, underscoring the broader lessons that policy makers can take from Portugal's remarkable improvement. I hope you like this episode as much as I do.
Now, without further ado, let's get started. I am honoured to have Dr. Nuno Crato joining me today. He is a research professor of mathematics and statistics at the University of Lisbon in Portugal. He currently serves as president of the Education Initiative.
That's how we'd say it in English. It is a nonprofit organization dedicated to fostering student success. He has a PhD in applied math and statistics.
His research focuses on stochastic models, econometrics, and the statistical evaluation of education policy, with particular attention to international assessments such as PISA and TIMSS. He was president of the Portuguese Mathematical Society for three terms, making him the longest continuous-serving president of the society. From 2011 to 2015, he served as Portugal's Minister of Education and Science, where he spearheaded major reforms that emphasized a knowledge-based curriculum, external student evaluation, rigorous teacher preparation, and school accountability.
And these changes contributed to Portugal's strongest-ever performance in international benchmarks such as PISA and TIMSS. He's also a very prolific author. His books include Figuring It Out, a math popularization book, Data-Driven Policy Impact Evaluation, and Improving a Country's Education, PISA 2018 Results in 10 Countries.
And recently he co-authored Developing Curriculum for Deep Thinking with some well-known educational specialists including Tim Surma, Paul Kirschner, and others. His contributions to education have been recognized with several awards, including the European Mathematical Society First Prize for Public Awareness in Mathematics and the European Science Award for Science Communicator of the Year.
Welcome, Nuno. Welcome to the podcast.
[00:04:35] Nuno Crato: Very nice to be here, Anna. Thank you for the invitation.
[00:04:39] Anna Stokke: Yes, I'm really excited to hear about all the things you've done in Portugal and around the world, really. So, let's just start with a bit of background just because things could be different there in terms of how the Minister of Education is appointed. So, my understanding is that in Portugal, the Minister of Education is appointed by the Prime Minister. So, the Minister of Education is not an elected official like they would be here in Canada, and they don’t necessarily belong to any political party. Is that correct?
[00:05:12] Nuno Crato: That's perfectly correct. Yes, that surprises my English friends because to become a minister, you have to be a member of the parliament. Not in Portugal.
So, the Prime Minister appoints the ministers, and the ministers are of his choice, and they have to be ratified by the parliament, and they have to be ratified by the President of the Republic, but essentially, it's pro forma.
[00:05:36] Anna Stokke: Okay, got it. Okay, so that's important to understand. So, let's go back to 2011. Let’s go back to 2011. You've been preaching about needing education reforms through books and opinion articles and press interviews, kind of like the stuff I do.
So, your views on reforming education were quite well known then throughout Portugal. And now in 2011, I understand that Portugal was in the midst of a severe financial crisis and you received an unexpected phone call from the new Prime Minister asking you to become the Minister of Education. That would be a really surprising phone call.
So, what was going through your mind when you got that call and why did you decide to accept the position?
[00:06:25] Nuno Crato: Well, I was really surprised, although some people were talking about my name, but you know, each time there are elections and there is a new cabinet, there are speculations about people. My name has appeared but had appeared before in other governments and so on. I never paid much attention to it.
I was in Berlin. So, I went for a conference in Berlin and when I received the phone call and I was really almost in shock and I said, well, may I think for a while? And he told me, yes, I'll call you tonight.
And I asked, may I talk to my wife? Of course you can talk to your wife, but don't talk to anyone else because this is the rule. If you don't accept, I don't want people who I'm going then to invite to feel they are a second choice. So don't talk to anyone, talk to your wife and we’ll talk tonight.
And that's what happened. So, I talked to my wife and what she told me and made lots of sense. She told me, well, “If you accept, it's going to be hell. If you don’t accept, you will never again in your life be able to talk about education because you had the opportunity to try to change things and you refused it. You didn't take it. So, it’s up to you.”
And I thought, and I said, well, I had in my mind a program of reforms in education for a while, because as president of mathematical Portuguese society, we had to give opinions, and we have to participate in debates and so on. So, we had a clear idea of the main things or some of the main things that had to be done. And so, I accepted it.
And that was it. And I was very happy at the end of the mandate of the tenure to see the results in PISA, in TIMSS, and see that the country improved. This was really my measure of success or actually measure of success for the country and for the kids.
[00:08:23] Anna Stokke: Absolutely. Lots to be proud of. And we’re going to hear all about it today. So now when you got this call, were you worried at all that maybe you weren't qualified? Like you're a mathematician by training. We don't know a lot about policy necessarily, right? Or making policies. We're definitely educators.
We know a lot about education, but, you know, you probably weren't a principal or in a school or anything like that or superintendent. So, were you concerned at all about that?
[00:08:51] Nuno Crato: I was a bit concerned but not overwhelmed by that fact because I always thought that a politician is a person. I had contact with almost all Portuguese ministers of education for a decade or more because we were invited regularly to talk, to discuss in private, in public and so on.
And I had an idea about the job. Of course, there are a couple of things I was not prepared, and I didn't prepare myself and I don’t regret it. Some things I regret, others I don't regret.
For instance, one thing that I don't regret is that my training as a teacher, as a professor is a training to try to understand what the question is, what's on the mind of the person who is asking the question and try to answer as clear as possible. In politics, things don't work exactly like that. So, when we have a debate in a parliament, this was kind of a drawback for me because I was looking at the person who was criticizing me or asking a question and I was thinking, okay, what's going on in her mind or on his mind? So let me try to be clear.
And that's not the way things work in politics most of the time. So, people talk, not to ask a question, but talk to appear on TV, raising that question, raising that point. Well, I can tell you a story that for me, it's very illuminating.
Well, I had many stories like this, but I had a debate at the parliament, and we had slightly increased because we were in difficult economic situation. We had slightly increased the number of students, the maximum number of students per class, raising from 28 to 30, something like this for high school. And I was discussing it and there was a lady, a member of the parliament who asked me, so do you think you can improve the learning of the students, increasing the class size? And I answered exactly what I think.
And that's what data tells us, is that increasing from 28 to 30 or from 30 to 32 or from 24 to 26, it's not significant, doesn't change things. And we have overwhelming evidence saying that class size, when we are talking about this, I'm not talking about raising from 30 to 70. That's a different story, but this type of increases or of changes are not significant.
That's not important part. There is overwhelming evidence about this from statistics of education, economics of education. It's like a well-known fact.
And so, I explained to her that. And she told me, well, so you are saying that you are improving the education because with larger class size, it works better. And I said, no, that's not what I'm saying.
And I explained again. And then she asked again, and then I explained again. And finally, I realized what was happening. What was happening is that she wanted to get on TV that sentence, the minister thinks that increasing the class size situation gets better. And she got it. And she was looking at her phone. And then my advisors pointed that to me. She was looking at her phone. And the moment that was on the online press, she stopped.
So as a politician, she was much better than I am, or than I was. But I don't regret that. You see, because I don't regret it.
Because I think that we should be frank. We should speak the things as they are. Of course, in politics, we have to be cautious about a couple of things.
And if the situation repeats, if I had that situation today, maybe I would not repeat the discussion and say, look, I have already explained this. Let's move on. Maybe that's what I would have said.
But that's okay. Just an example. Our professor is, has his mind by training in a different way than the politician.
[00:12:59] Anna Stokke: I get what you're saying.
When we get asked questions, it's because people genuinely want to know the answer.
[00:13:05] Nuno Crato: Exactly. Not because they want to make a point.
[00:13:07] Anna Stokke: Yes, exactly. So that must have been eye-opening. Let's go back to before you started. So, what was the situation like in Portugal's schools? Like what were the dominant ideas in education at the time? And why did you feel they weren't working?
[00:13:24] Nuno Crato: We had about a couple of periods in education in Portugal. So, we had the revolution of the 74, so democratic revolution. And we entered in what we can call a romantic period. No exams, the flexibility in terms of the curriculum, the joy of kids was the most important thing, this type of policy and this type of thinking.
And that lasted more or less until 2000, 2001, 2002. Because in 1995, there was the first teams, and the results were revealed in 96 and 97. And the results were appalling for Portugal.
We only had two countries below us in terms of math, fourth grade. That was really something that we thought a lot. This is not good.
We have to change. And then in 2000, we had PISA. And in PISA again, we were very low on the ranking of countries.
And that made everybody think, well, we are not in a good path. We have to change things. And then it followed a period until, let’s say, 2010, 2011.
That was what I call a pragmatic period. So, all ministers at the time were worried about the situation. And they were always all saying, we have to improve education.
We have to have a stricter curriculum. We have to have more evaluation, better evaluation. Although with differences and all this wrapped up in some romantic ideas for some people and not for others.
But the idea was this, was let's move on to improve the curriculum, to improve assessment. Let's see if these kids learn more. That was the frame of mind for everybody.
And so, I call this pragmatic because this was wrapped in some romantic ideas. And some ministers wanted to do this with the experts on education that thought exactly the opposite. So, for instance, they wanted to improve the curriculum with the help of experts of education that think that curriculum should not be improved.
So, they think some experts on education think that curriculum is secondary. What's important is discovery learning or activities or experiencing math situations. And so, some ministers looked for the help of this type of people and this type of mental framework to improve the curriculum.
And of course it didn’t work very well, but it worked because we improved slightly. We started improving our results in TIMMS and in PISA, not in TIMSS because TIMSS was canceled for a while. So, in PISA.
And so, situation was working. When I arrived at the ministry, I had the idea that we should do it but should do it better. Let's say, yes, we should improve the curriculum, but we should improve it better.
We should improve evaluation, but we should improve it better. And so that was my point of view. Let's work in a way that although we have a difficult economic situation, we do things that we can do with almost no money.
For instance, the curriculum is peanuts for a ministry. It's not an important expense on the opposite. And so that was my idea.
That was the idea of my team. I had a good team, people that thought the same way, and that helped me a lot to doing it. So, we essentially, what we did was these two things, improve the curriculum and improve the evaluation. So, we have these two things, but we added a third factor, which is support for students with special difficulties. I'm not talking about students that have some type of cognitive problems. No, I'm talking about kids that simply are behind because of their background, because of their socioeconomic status, whatever.
They were not lucky in previous classrooms and so on. I think this is very important because today this may seem trivial, but at the beginning of the 21st century, people were debating this dilemma. Well, either we improve the curriculum, we make the curriculum more demanding and do more evaluation, and then we lose the kids from poorer backgrounds, or we try to catch these guys, these kids, and then we lose the top kids.
So, the kids that are ready to progress faster. And so, this dilemma that was discussed at the beginning of the 21st century, and I attended many debates where this was the dilemma. Well, I think this has a simple solution, simple conceptually, but then easier said than done.
And that solution is we should provide special support for kids that are having some special difficulties at a certain time. So, we should have special times, we should have response to intervention programs, we should have this type of thing for these kids. So, I would say that what we did was these three things, curriculum, assessment, and special support.
[00:18:45] Anna Stokke: Okay. So, let's actually kind of dig into those things a little bit more. And I’ve definitely heard you say before, and I did some reading that you've written on things you’ve and you often say everything starts with the curriculum. What does that really mean? And what are characteristics of a good curriculum?
[00:19:08] Nuno Crato: Well, that's a good question. Well, what it means is that really everything starts with our goals. So, if we don't have goals, we don't arrive anywhere.
You know that Alice in Wonderland dialogue, the cat and Alice, and Alice's way is asking, oh, what's the way? And the cat answers, well, it depends on where you are going. And Alice says, I don't know where I want to go. And then the cat says, then take any way.
All ways are equivalent if you don't know where to go. And this is the point; we need to know where to go. And that's the curriculum.
I'd say the curriculum is the goals. For American and other colleagues that are used to a very decentralized system, you have to change a little bit your mind, thinking that Portugal, as Spain, as Italy, as France, and so on, but Portugal and France, maybe more than the other countries, we have a very centralized system. So, it's highly centralized.
So, the minister or the ministry decides everything. And so, decides everything about curriculum, decides everything about the assessment and schools follow. And so, the curriculum is a national curriculum decided by the ministry.
The ministry, of course, has or should dialogue with many people and have many experts in dialogue, but it's the responsibility of the ministry. So, when we are talking about a curriculum, we are talking about the organized list of ideas, topics, knowledge, and skills that a kid should acquire every year in every discipline. So essentially this is the curriculum.
We have two pieces of the curriculum following a little bit the Anglo-Saxon lead. One piece is what we call programs. That means it's a description.
So, let's go to math, for instance, high school math. We should say, well, students should know polynomials, should do algebra of polynomials and this type of things. That would be the program.
And then we have another piece that we call methods or goals or learning goals. Maybe the better, best translation is learning goals in which we say students should be able to define what the polynomial is of degree to start and then degree n. Then students should be able to multiply polynomials, add polynomials, divide polynomials, everything like this of degree n. So, all these types of things expressed in what we call learning goals, which are very detailed and need to be detailed because if we have things completely vague, then one school is very rigorous and next school is not rigorous. And there is this tendency that everybody has that if we are having difficulties, well, we should water down our goals.
This is human. I'm not criticizing teachers. This is human.
If I have lots of trouble explaining how to multiply polynomials of degree two, I avoid going to degree three unless there is a curriculum that says, no, no, you have to be able to multiply of degree n. What n means, n, of course, it's not put everything up to infinity, is give examples of n equals four, n equals five. These types of things should be very clear in the curriculum. If you don't have a national curriculum and if you have a regional curriculum or if you have a school curriculum, I'd say the same thing needs to be done at a different level, but it needs to be done because otherwise we don't arrive there.
We all need goals in life.
[00:22:52] Anna Stokke: Okay. So, you started by transforming, like making sure that the curriculum got rewritten. Just out of curiosity for math, did you use the National Math Advisory Panel Report or anything like that?
[00:23:07] Nuno Crato: Yes, we used that. And for elementary math, we looked at TIMSS very carefully.
[00:23:13] Anna Stokke: At TIMSS?
[00:23:14] Nuno Crato: Yeah, because TIMSS has a list, and there are studies by Bill Schmidt and other people who have studied TIMSS and have studied the coherence of curriculum in many countries compared to TIMSS questions and so on. We looked at that very carefully, and we adapt the curriculum to be more focused and better defined in terms of math progression. And it's a knowledge-based curriculum.
What we built was a knowledge-based curriculum. That means knowledge over knowledge over knowledge over knowledge.
[00:23:47] Anna Stokke: And you talked about the curriculum being, A, it's going to say the outcomes, the things that we expect students to know. And then you talked about sort of how they should demonstrate those things, right? And I've seen similar curriculums here in Canada, by the way. I've looked at them, and sometimes it's that second piece where you can kind of see that someone’s trying to dictate pedagogy.
So, there's sort of comes along with kind of the constructivist type thinking sometimes that children should have to display that they can do things in many different ways because every child has their own way of doing things, right? I don't suppose you ran into any trouble like that. You got people to work on the curricula who were on the same page as you, obviously.
[00:24:36] Nuno Crato: Exactly.
And with standard algorithms, this type of progression. And actually, we avoided giving pedagogical advice to teachers because we thought this way. If the goals are clear, then teachers will manage to get through these goals.
And actually, that happened. So we didn't need to say, oh, you have to have explicit instruction because teachers immediately understood, well, if we have to get to this point, then we have to be efficient and then we have to explain this and we have to make sure that kids understand it, that kids develop necessary skills to cope with these problems and so on. So, we avoided that.
For my English friends, this is a little bit surprising because they have the idea that we should kind of impose the science of teaching to teachers. We didn't do that. We, of course, we disseminated the best practices, and we disseminated the scientific results regarding teaching, but we were not very specific about that, except for reading.
The reading is a different story because reading, we needed to put some measures like fluency measures, because otherwise kids, they just arrive at second, third, fourth grade without being able to read clearly and fluently. And so, for that, we were more specific. We had help from many good psychologists, actually from France and from Belgium and so on as well. And we put these goals in the curriculum.
[00:26:14] Anna Stokke: Okay. So, you, Portugal has a great curriculum.
[00:26:17] Nuno Crato: No longer, no longer, but it had, no longer.
[00:26:21] Anna Stokke: Isn't this always the case?
[00:26:22] Nuno Crato: It's always the case. And if you look at PISA and so on, you will see that we had a great improvement up to 2015, where we obtained our best results ever, but then we changed course. And so, in a way, this is what sociologists or economists call a natural experiment.
It's like you have something you don't experiment actually, because we can't do this type of things with humans, but two different policies lead to two different results and a clearer policy, better oriented towards a knowledge goals worked very well. We were better than Finland in some measures, which is for a small European country like Portugal and Southern European country is really amazing. So, curriculum, that's the first part.
Now testing is second thing. And why do I say that everything starts with curriculum? Because curriculum tells us the goals and then testing should go for these goals. And testing needs to be well done as well, because if you test things towards memorization of facts, for instance, if you get your students used to have exams that just ask for dates or for places or for names of heroes or things like that, that’s what kids will do.
They will memorize these things. But if you have exams in history in which you ask for what were the main causes of the First World War, how the First World War was different in terms of from the Second World War, things like that. What are the great movements in history? And if you teach these to your students, then testing is good.
It's like, I think the best example is this. You want to learn English, not as a specialist in English literature, but you want to learn English as a tool. What's the best testing you have for that? Is English is spoken to read, to read, to write and to speak.
And so, teaching to the test is teaching to know in a way. So, this idea that we teach to the test, well, we teach to the test. That's not a problem if the test is aligned with the curriculum, which is not easy sometimes.
[00:28:45] Anna Stokke: I completely agree. There is absolutely nothing wrong with teaching to the test as long as these are good tests. By the way, on the curriculum changes, you mentioned in some of the pieces that you wrote something about moving away from vague subjects.
So, the schools were, my understanding is there were some vague subjects that they were focusing on. So, can you give an example? Like what were they focusing on?
[00:29:11] Nuno Crato: Project area, for instance, this was, kids should do projects and should have four hours a week for projects. Then they should do citizenship education, and they should have three hours a week for that.
Without a clear understanding of what are they doing in a project. Projects are great, but if you have a clear intention in terms of what we are teaching, for instance, you do a project for math, they are not so easy to do projects for math. They're not so easy.
You have to, what do you want kids to learn? That's what should be in your mind when you are doing a project. Maybe examples in other areas are easier than math. For instance, I came across in a Southern American country some time ago of a project.
The teacher was very happy. She was doing a project of colouring the map of South America. And then kids were colouring the map, and this was in fourth grade.
They were very happy because some would put soccer players in some countries and would put original things in other country and so on and so on. And then at the end, so this was in a debate, a workshop, at the end, people asked the teacher, what did they learn? Did they learn the capitals, the number of the countries, the borders, the cultures, the geography? And the teacher couldn ’t answer. Well, what does this mean? It means that the kids were concentrating on how to cut the soccer players, how to choose the right colours, how to choose the right glue to put things one over the other and without a clear goal.
And so, of course, they could learn a lot about colouring and paper cutting, but did they learn something about geography? No. And so, projects should be designed in a way that very clearly for the teacher, what the knowledge goal should be very clear for the teacher, and everything should be oriented for that. And it's not easy to do that.
And in math, it's not easy to do it, maybe even more difficult than in other areas. I can talk about projects that I do in at university, but that's for master's students.
[00:31:32] Anna Stokke: Oh, yes.
Like real math projects require a lot of knowledge, right? You can do a science project or some sort of other project where you do some statistical analysis. You can do things like that. But to do an actual math project is going to be almost impossible for a student, right? So, but I'm pretty sure listeners are going to be familiar with the type of vague subjects then that you're talking about.
We're very familiar with the project-based learning. And I agree that it often there's not a lot of learning going on. It looks good, right? You can display the projects and it's busy and it seems like fun, but at the end of the day, did students really learn? Sometimes not so much.
You know, like learning really needs to be, it needs to be focused. It needs to be targeted. If you want someone to learn something, you've got to actually give them the type of instruction that's going to make sure they learn that thing.
It's got to be very focused.
[00:32:33] Nuno Crato: Exactly. And people think there is this misconception that active learning or active methodologies are making things. And active methodologies should be being active here. So, the brain should be active. That means the brain should incorporate the new knowledge into old concepts, try to build, to construct in the mind a coherent representation of what we are discussing.
This is active and this can be done in a very quiet way, not necessarily cutting and moving from one side to the other of the classroom. Although sometimes it's good to have some movement, of course.
[00:33:14] Anna Stokke: Sure.
Yeah. And we’re not against science fairs or science projects and things like that. It's just, they really shouldn’t be the centre of learning.
[00:33:23] Nuno Crato: Because it's not possible. It's not possible. If you want, just think, I think physics is a good example for that.
Because if you look at the concepts, just think about a year course in physics in high school or so, because it's easier to understand. You have to talk about forces, about vectors, about masses, about weights, about all these types of gravitational forces, all these types of things. And if you do everything by a project, you’ll never end.
You'll spend a couple of centuries doing it.
[00:33:54] Anna Stokke: And you would almost need a one-on-one tutor to teach you the things that you need to know to do the project properly, which just isn't going to happen. So, well, let's go back to standards and evaluation.
So how frequently did you implement standard assessments? Like what grades and what did you do with the results?
[00:34:18] Nuno Crato: We have historically, we have elementary school divided in two cycles.
We have the first cycle from one to fourth grade, second cycle, fifth and sixth grade. Then we have middle school, three years. Then we have high school, another three years.
So historically, this is the way it goes in Portugal. And also, kids change schools at the end of a cycle. So not necessarily.
There are schools that go from kindergarten up to 12th grade. There are some schools like that, but most of the schools are not like that. And so even if they are integrated or if they cooperate, students typically change school, change establishment, physical building at the end of each cycle. And so, it makes sense to have evaluation at the end of each cycle because that's where things come to a closure. Now, by fourth grade, kids should read fluently and should have an understanding of basic texts. Okay, so that's the time to evaluate that.
They should know the basic algorithms. They should know the basic concepts in geometry. What is an angle? What's a triangle? What's a polygon? These types of things, closure.
That's evaluated there. So, for these historical reasons, for things having been divided like that, I think the way to go is to do at the end of each cycle an evaluation of kids. If we had just two cycles, if we had six plus six, as some countries do, we should do at the end of sixth grade maybe because it's the same teacher or the same school. So, it depends on the country. Now, what is this evaluation? This evaluation is what I would call middle stakes, not high stakes, not low stakes. In education, people talk about high stakes as evaluations that are consequential.
That means kids could fail or kids could be required to repeat the grade if they fail. These are high stakes exams. And then low stakes are evaluations that are just for information purposes.
Now, each country should look at history, look at the culture and decide what's better, what's best and what's the best combination for this. I think that one way or the other, more consequential, less consequential, we should have standardized assessment. Because if we have a national curriculum, we should have a standardized assessment.
Because this standardized assessment is going to make things uniform, is going to push everybody up, and it's going to help teachers in school to do their formative assessment. That means the type of assessment that they do just for information and for learning purposes. Because if we don't have standardized assessments, teachers would do whatever they feel is better, their best judgment.
But we are all humans. And so, we should have something as a goal, as a national goal or regional goal or whatever. It depends on the country, on the culture.
I don't know if I made myself clear. So, I'd say assessment is a system. We cannot say we have formative assessment that’s good and we should not have standardized assessment.
Or we cannot say we should have only standardized assessment. We should have everything. All this is a system that complements all the pieces there.
[00:37:48] Anna Stokke: So, in Portugal, though, your standardized assessments occur at the end of each cycle.
[00:37:53] Nuno Crato: Yeah.
[00:37:54] Anna Stokke: So, like what we'd say, the end of grade 6 and then grade 12?
[00:38:00] Nuno Crato: And then 9th and then grade 12.
[00:38:02] Anna Stokke: OK, so 6, 9 and 12. OK, got it.
[00:38:05 Nuno Crato: Actually, 4, 6, 9 and 12.
[00:38:08] Anna Stokke: OK, got it. And what do you do with the results? Do you publish them?
[00:38:13] Nuno Crato: We publish the results per school. We report the results per school, and we tell each student, so the results of the student.
We introduced what I'd say middle stakes assessment for fourth and sixth grade, which was, this is kind of difficult to explain, but they were consequential for the student, but very mild. So, what it means is that a student who failed math, who failed reading, who failed three other subjects would be retained for one year. In reality, almost no one was in that situation.
But even so, this improved our situation immensely because everybody looked and said, oh, my God, I have to study. I have to work. And teachers and parents and students were saying, I have to work.
Let's do it. Let's go. And so, a few, very few, very rare cases that students were retained, but because of the exams.
So, it was what I called middle stakes assessment.
[00:39:19] Anna Stokke: OK, so those were some sort of accountability measures, so to speak. OK, were there any other accountability measures or incentives or was that it?
[00:39:31] Nuno Crato: OK, this is for students.
Then for schools, there were other accountability measures. But the point is this. While some countries and some philosophies think that we should evaluate schools and not students, I think that we should evaluate schools through their results on students.
I mean, it doesn't make any sense to say, oh, this school is doing a great job. And I don't look at what happens with students, because the purpose of schools is to improve students' knowledge and skills. So, we should look at kids. We should look at their knowledge and their skills and their improvement. Now, this is difficult because we have different cultures. We have different backgrounds. We have different socioeconomic backgrounds. We have all these types of things. So, what we did is we constructed a system of measures, which was actually very difficult.
Every year we had a discussion with the statisticians of the ministry to fine tune this, in which we would grade a school for the improvement in results, for the alignment of external evaluations with internal evaluations, but essentially for the improvement in the results. Because you have kids, let's rate kids for 0 to 100 just for the sake of simplicity. So, you have kids that arrive with 20.
And after one year of schooling, they arrive at 40. This is much better than having kids that arrive at 90 and at the end of school are 91. So, it's important.
The improvement for kids is what's very important. So, this is what economists of education call value-added, and we measured value-added. And then what we did is that we gave more resources to schools that were able to show that they use well these resources to improve kids' results.
And so, this is a very sophisticated and very fine-tuned statistical exercise, but it worked well. It worked well because actually, and this is one thing that gives me some hope in teachers and hope in schools, is that once schools and once teachers have clear goals, they answer. They go for it.
Although they are hearing all these strange things about learning by discovery learning and project-based and active methodologies and all these types of things, once they have goals, they realize that kids are kids, they need to improve, let’s work for it.
[00:42:19] Anna Stokke: Yeah, if you do actually have to get someone to learn something, and it's going to be tested to see if they learn that thing, it's going to drive you towards using the most efficient and the best method. I do think that that's part of the reason that a lot of people don't like testing.
A lot of people would say that these kinds of reforms that you’re talking about, so like really rigorous curricula and testing and accountability measures and transparency, these things could perhaps negatively impact certain students, like students of low socioeconomic status or students that are weaker, they don't have a lot of resources at home. So how did you address that? What do you say to that criticism?
[00:43:08] Nuno Crato: Actually, we have numbers and it’s very clear if you look at the analysis of data from TIMSS and PISA and PIRLS and if you look at our country, each time we improved the situation, the number of low-performing students was reduced. Each time we had a less rigorous curriculum, the number of low-performing students increased.
So, the number of low-performing students and with the great oscillations, like 30 percent, that was the highest level we had when we had a curriculum that was saying this is made for everybody. When we say it's made for everybody, it's made to address the difficulties of lower socioeconomic backgrounds, well, the results are exactly the opposite. These guys are those that suffer most.
We have numbers for this, and I have papers published on this with Portuguese numbers. And it's a generic situation.
[00:44:11] Anna Stokke: We can sit here and theorize all we want, but if you've got data, that does the talking, right? And speaking of that, I do have a question about pedagogy. And I know you said that your curricula and your reforms, you didn’t dictate pedagogy.
However, people would probably come to you and say, “No, no, project-based learning is best because you were trying to get rid of some of these vague subjects”, as you call them, from the curriculum. And you're a mathematician. Mathematicians don't normally have training in pedagogy.
And you might look at that and say, “No, that doesn't make any sense.” But did you educate yourself? Like, did you educate yourself on evidence-based pedagogy so that you could counter some of these claims?
[00:44:59] Nuno Crato: I did. And that's what aggravates most people who discuss with me and Dana on the other side, is that many times I know more about Ausubel, about Piaget, about Vygotsky, about all these things that they do, because I read a lot about all this.
But what I think is very important is that people read modern things about education because the cognitive psychology has developed immensely in the last 20, 30 years. So, we have a lot about the testing effect, for instance, we know now from many experiments in cognitive sciences that by testing we improve. And so, this is a surprising factor.
But for instance, a student is reading a book. Think about history. Maybe it's better than math because math we don't read the same way we read.
But math is also a good example. A student reads a book about whatever, about geometry and says, oh, I understand all this. Oh, I know the angles.
Oh, I know the sum of the angles. I know the formula for this. I know that.
And that's not a good way to study. A good way to study is to go to the exercises because the moment you go to the exercises and go to more difficult exercises, if you can, the moment you go there, you start understanding that you are not understanding as well as you think. And by solving exercises, you learn a lot.
And that's the best way to learn is to test yourself. And so, there is lots of evidence about this nowadays. And there is also lots of evidence from the economics of education and statistics of education that show that once a country introduces exams, that country improves.
And once a country quits some exam or abolishes some exams, the country goes down. And this is type of studies that we couldn't have in the 20th century, but we can have now because we have PISA, we have longitudinal data about PISA, about TIMSS, about PERDs. We have all these external assessments.
And we have also data about each country when they introduced an exam, when they abolished an exam. And so, we know that more consequential types of testing improve a country's education. And that's like a general rule.
And that works in a stronger way for countries that don't have such a good education, the opposite of what we think. So, let's say you have a country like Singapore or like Estonia, they have a great system. For them, one exam more, one exam less, it's not that important.
But you have a country that is struggling. It's more important for this country to have exams than for Singapore or for other better developed countries in terms of education.
[00:47:56] Anna Stokke: That's really interesting. And I'm glad you brought this up.
So, because you’ve actually done a lot of statistical analyses on data collected from PISA and TIMSS, right? And people are always talking about PISA and TIMSS and correlation isn't causation. That is true. But can we draw any conclusions, say, about types of instruction that positively correlate with student success from that data?
[00:48:23] Nuno Crato: Actually, we have more than that, because correlation is not causation.
But once we have a timeline, and in TIMSS and PISA, we have a timeline now. And we know that if we make a change in the curriculum and then the situation improves, most likely that's causal. It's not just correlational.
And so, we have more information about that. But in terms of correlation, there is a very clear study by PISA that shows that the direct instruction is the most effective way of teaching, and that the opposite. So, student-led instruction is associated with poor results in education.
So, this is a very clear correlation. And there is also a very interesting correlation, because PISA tells us that we should develop competencies, we should develop skills, and knowledge is no longer important. Sometimes the OECD officers say that, and I think they are completely wrong, because their data shows the opposite.
Their data shows that countries who have a knowledge-based curriculum are countries that perform better, and not countries that have a vague skills-based curriculum. And we have lots of examples across the world. One of the best examples is Scotland against England.
At the same time, England decided to go for a knowledge-based curriculum, Scotland decided to go for a skills-based curriculum, Scottish results were down, English results were up. So, there is something there.
[00:50:03] Anna Stokke: I mean, I found that really interesting about the OECD as well.
So, they administer PISA, and PISA actually isn't really like a knowledge test in a way, right? So, whereas TIMSS is. Like, in my mind, TIMSS is a much better test. So, TIMSS would test like real mathematical skills, and whether students would be prepared for future mathematics, right? Whereas PISA is more of a problem-solving test.
They call it a mathematical literacy test. I probably would, yeah, I'd put it more sort of in the problem-solving domain, which is fine. But what's interesting about it is the data actually shows that it's like the knowledge-based curriculums, the direct instruction that actually cause students to do better even on the problem-solving test.
And its counter to what a lot of people want to be true, including the OECD people themselves, would you say?
[00:51:04] Nuno Crato: Yeah, including many of the OECD people. Yes, yes, yes, including them, because that's the reality. If you just think about anything, like if you study math for solving problems only, you will never develop your math skills, because you need to know a lot.
I don't know, it depends on the level of math you want, but even for high school math, you need to know a lot. You need to know what functions are, what is continuity, what are limits, what are this, what's infinity, what types of infinity we have. All these types of things you need to know, what rational numbers.
And you can’t do this through projects and through project-based learning, you have to know a lot, then you can start solving problems. And it's the same thing, you want to be a piano player, you want to play Beethoven sonatas, well, start with scales, start with basic knowledge, and then you will play Beethoven, but it takes time.
[00:52:02] Anna Stokke: So, a little more on the PISA and TIMSS data, and I'm curious about money.
So, does more money spent on education translate to better outcomes?
[00:52:15] Nuno Crato: Well, we have really interesting data on PISA studies. I think OECD did a great job there. Now we have, where it's due, it's due.
So, OECD did a great job there because they compared countries and the investment in education in purchasing power parity and cumulative for students and so on in many countries. And I'd say that main conclusion is that up to a certain point, so let's say up to a poor European country, more money is correlated with better education. But after that point, it's all the same.
You look at Poland, and you look at Luxembourg. Poland spends about a third of what Luxembourg spends per student, and Poland has much better results. So, money doesn't explain everything.
And actually, I think that there is something that costs almost nothing and that improves much better education than just poor money on education. And I think it's a curriculum, a good curriculum. And many, many countries don't start where they should start with a good curriculum and they just decide to increase the spending in education, buy more computers and so on.
That's not the solution.
[00:53:35] Anna Stokke: In terms of bang for the buck, bang for your buck, you'd say curriculum is probably the best one.
[00:53:42] Nuno Crato: I'd say so.
[00:53:43] Anna Stokke: Anything else?
[00:53:44] Nuno Crato: Well, assessment?
[00:53:47] Anna Stokke: Yes.
[00:53:48] Nuno Crato: And support to students with more difficulties. So, I'd say these three things are really very important.
Let me say something about PISA, because when PISA was set, PISA was set for 15 years old. So, it's not a grade based because it's not for a specific grade. And it made sense because at the time when this was designed, so we are talking about 1995 more or less, when this thing started being designed.
PISA aimed to know how well prepared our students at the end of mandatory schooling, because mandatory schooling at that time was essentially 15 or even low than that. Nowadays, most countries have mandatory schooling up to 18 years. So maybe PISA should be revised on that because it's no longer measuring what it intended to measure because things changed a lot.
Or maybe PISA should be more knowledge based than it is now. But that's for someone else to decide, not for us. But just to explain why we have this contrast between PISA and TIMSS, because they have different goals.
[00:54:58] Anna Stokke: So, let's talk about your reforms and what happened.
So, by 2015, Portugal's results in PISA and TIMSS improved dramatically. And so, before you started as minister, only two countries were below Portugal. And by 2015, 36 countries were below Portugal, including Finland, as you already mentioned.
And everybody just loved to talk about Finland. So, they're talking about Portugal now, right? They're not talking about Finland anymore. It's all Portugal now, right?
[00:55:32] Nuno Crato: I hope so.
And I hope we go back to the dramatic, great improvements we had. Let me just say something. You are compared, thank you very much, but you are exaggerating what I did in a way, because the two countries you are referring to is in TIMSS 1995.
And then the Secretary of State at the moment decided, oh, we have so poor results, better not to do TIMSS again. And so, we didn't do TIMSS, which was really very bad, very bad policy. And once we resumed, we resumed in 2011.
And that's when I started working at the ministry. So, the improvement was really due to a long period in which we have these pragmatic people that thinking whatever they would think, they would try to improve the curriculum. And so, there is a long line of ministers that try to do their best to improve the curriculum.
And if they didn't do better, it's for some problems that they had, but they were trying and they were improving the situation in Portugal. So, I think this is also important because usually things don't come just from a decision of some smart or some stupid guy or whatever who is there. Things come from a country's involvement in improving education.
But even that, let me tell you something. I think I’m exaggerating on your time but let me say something. In education, people think usually that things take a long time to change.
Well, it depends. And I think we should be very clear about this. If you are talking about, if you are improving teachers' initial training, for instance, suppose we have a policy and we say, okay, we don't have the best system.
Let's improve the system. And there are lots of things to be done because teachers still learn very outdated things in schools of education. Teachers' colleges still teach very outdated things, which is a pity.
Suppose we change it for better. And suddenly we have in the United States, in Portugal, the best teachers' colleges possible, the most updated on the best science of learning and so on. How long does it take to change the results of a country? It takes at least one decade.
[00:58:00] Anna Stokke: Yeah, that one takes a long time. It's a long game.
[00:58:03] Nuno Crato: It's a long game.
So, there are things like that that take a long time. But if you change assessment, the time it takes is one day. Because you change, you introduce an exam, next day, everything is changed.
Next day, schools look and say, oh my God, we have to pass this exam, or we have to pass this test. Let's change things. But if you abolish one exam, everything changes.
Next day, everything changes. People think, oh, okay, okay, good. We don't have this test now.
So, it depends on what you are talking about. If you are talking about the test, that changes things dramatically from one day to the next. If you are talking about improving the quality of teachers' preparation, it takes a long time.
[00:58:47] Anna Stokke: It's still an important thing to do, though.
[00:58:50] Nuno Crato: Very important.
[00:58:51] Anna Stokke: You know, I agree that it takes a long time.
It's a long game. But we're educating people to do a job, and they deserve to be well prepared for that job. And they deserve to be taught about techniques that will actually work in the classroom.
They should actually know the subject. So those are all really important things. The teacher preparation is really important.
But you're right, changing it, making changes in teacher preparation obviously will take a long time. So, I get what you're saying. Because I would imagine politicians think that way.
They're probably thinking about things that will make things change very quickly. So, then assessment's the way to go. And also, very important.
So, assessment, curriculum, those changes you made obviously had quite a positive impact. Did you face any criticism? Did you get pushback?
[00:59:53] Nuno Crato: Oh yeah, definitely. Because there are two types of oppositions.
One opposition is from people from outdated teachers' colleges, ideologues of education who are still thinking in this romantic view, non-scientific view, that project-based learning is the solution for everything and that student-led education is the way to go, and these types of things. So, we had the opposition from people who think like that. And we had the opposition from some type of unions.
And I thought that the first type of opposition was stronger than the second, but I realized it's not true. It's actually the opposite. So, people from ideologues of this education fads and education romantic perspectives, they don't have so much weight in real life, in political life.
They have weight in their schools, but not outside of their schools, at least in our countries. And unions, if they want to oppose, they have power to oppose. And so, unions are something that we have to deal with.
I dealt with different types of unions, some very radical, some moderate, some amenable to a discussion, others that just wanted to criticize. But the important thing is to have the support of teachers and support of parents and support of the country. And I came to realize that teachers, even though they have been hearing these things, I'm talking about Portugal, 2011.
I'm not talking about Finland, 2015 or Canada, whatever, in different time. I'm talking about Portugal, 2011. What I came to realize is that although teachers have been hearing these ideas for a long time and hearing that testing is bad and students' initiative is good, and content is not important, and skills are more important.
Although they have been hearing this for a long time, the moment they realize that there is a good curriculum, they change their mind and they say, let's go for it. So, I'd say that teachers were very supportive and we couldn't have done any improvement without teachers. We couldn't have done it.
And so, I'd say that focus on teachers, focus on parents. If I can, as an amateur politician to give some advice, I say that focus on parents, focus on teachers, because they are your allies. They are people that really are moving the needle, and they are much better than what sometimes people think. Teachers are fantastic. I have only good things to say about teachers.
[01:02:52] Anna Stokke: Are you the only mathematician who's been an education minister?
[01:02:58] Nuno Crato: Oh, that's funny, because once I was appointed, I received a message from a friend of mine. I still have that message somewhere. And he sent me a statistic about how many mathematicians have been ministers of education since that job exists from the 19th century.
And there were very few and long-lasting ministers stayed there for a month.
[01:03:23] Anna Stokke: Really?
[01:03:24] Nuno Crato: So, he told me, you have to stay for longer than a month, at least.
[01:03:28] Anna Stokke: And you did.
[01:03:29] Nuno Crato: And I did.
[01:03:30] Anna Stokke: Okay. So, we can definitely say you are the longest serving minister of education who was a mathematician, right?
[01:03:39] Nuno Crato: That's correct. Yes.
[01:03:40] Anna Stokke: Okay. Wow. That's pretty exciting.
[01:03:43] Nuno Crato: In Portugal. I don't know about other countries.
[01:03:46] Anna Stokke: Ah, in Portugal. You think there are some in other countries. Well, maybe someone will write to me and tell me, oh, there was this other minister of education.
Certainly, there are mathematicians who have been politicians.
[01:03:57] Nuno Crato: Yes. Actually, there is a book about that.
There is a book by Springer made by two Spanish mathematicians about math and politics, and where they made a list of, all to their knowledge, mathematicians that were involved in politics. I even wrote a small statistical story for them about some statistical problem I’ve solved while I was a minister. I had to use statistics for that.
[01:04:26] Anna Stokke: Oh, that's great. I mean, it would come in handy. No one's going to trick you on the statistics.
That's for sure. So, let's end with this. Many countries are struggling today with declining math and reading results all over the world, actually, except, you know, the East Asian countries are doing great. So, if you could give some brief advice to policymakers based on Portugal’s experience, what would it be?
[01:04:54] Nuno Crato: I'm going to sound a little bit repetitive, but I'd say make sure you have a solid curriculum.
And in math, that means that concepts are well-organized on top of each other, progressive, rigorous, detailed, and that students get a good grasp of the mathematical concepts and mathematical algorithms and mathematical procedures. So, I'd say curriculum and then assessment and get a great curriculum. Go over, look at Singapore, look at U.S. math panel, look at Portugal, look at England, look at a couple of countries and try to get a great curriculum. I say this is what my best advice.
[01:05:38] Anna Stokke: There you have it, people, the formula for success, for education success. You heard it here.
So, it's been an absolute pleasure to talk to you today. I've been a fan of your work for a long time. As you know, we've met previously, so it was just an absolute honour to have you on the podcast today.
Thank you so much.
[01:05:56] Nuno Crato: Oh, thank you very much for having me. It was a great pleasure. Thank you.
[01:06:00] Anna Stokke: If you enjoy this podcast, please consider showing your support by leaving a five-star review on Spotify or Apple Podcasts. Subscribe on your favourite podcast app to get new episodes delivered as they become available.
Chalk & Talk is produced by me, Anna Stokke. You can follow me on X or LinkedIn for notifications or check out my website, annastachy.com, for more information. This podcast received funding through University of Winnipeg Knowledge Mobilization and Community Impact Grant, funded through the Anthony Swaity Knowledge Impact Fund.