Louise Howson talks to Hadi Abulrub a senior lecturer in the Department of Mechanical Engineering who explores the literature around programme level assessment and how this should be encouraged.

Listen to the entire series here

Podcast Transcript

Welcome to the pedagogy podcast brought to you by the Bristol Institute of learning and teaching. Each week we look at a different piece of the pedagogy pie and see how we can inspire exciting new practice at the University of Bristol. We hope you enjoy this slice of teaching and learning engagement. My name is Louise Howson. I am a lecturer in academic development in the Bristol Institute for Learning and Teaching. And today I am joined by Hadi who is going to tell us all about program level assessment. So first things first, could you tell us a little bit about yourself Hadi?

Hello. Thank you, Louise. My name is Hadi. I’m a senior lecturer in the School of Civil, Aerospace and Mechanical Engineering. And I’m associated in the Department of Mechanical Engineering. My background is in engineering with management. So that’s what I teach and what I do I have a PhD in engineering and also an MBA engineering. So I always been fascinated by the interdisciplinarity of these topics together. Now I am the program director for the engineer with management program.

Could you summarise what you think the key messages which came from your reading into program level assessment?

The literature is really quite rich in terms of the program level assessment. And if you read in the literature, you will always find that, you know, program level assessment has been approached as being a very positive, of the student experience. And it’s always has been looked at as a way to transform the student experience. So really, that’s how I started looking at it positively and always encouraged me to start with the excitement of understanding the structure of assessment within a program as a kind of like, really, the derivative from the literature always talks about how you map the assessment along the whole of the programs, how do you have a structure around it, how you have a mix of assessment and ensure there is both a summative and formative assessment that comes through the program in order to ensure, of course, the integrated learning outcomes of the program and the units, but also to deliver on the skills as much as possible. But also there is a lot of talk in the literature around the quality of the assessment and how does it really provide, if you like, the employers of the assurance and the credibility of our graduates in one sense? How do we actually ensure that when we say, this graduate has attained or achieved these ILOs, or these skills at a program level that especially we are having a major and a decision through an assessment that really contribute to this. And that’s really what’s the problem Level Assessment is trying to achieve in in some of its areas, but of course, the topic is quite big. So I’m just really going to focus on that for now.

Yeah, what do you think about all the ideas about authentic assessment and things like that in terms of, of practice? So yeah, this is quite a large, large area? Definitely. Can you tell us more about the model of program level assessment you applied within your program?

Yeah, thank you very much for asking about the way that we implemented them. Let me start by saying that the program level assessment that we applied in our program is a little bit different from what you see in the literature, as in the essence of it, it integrated three units in one place for for assessment. But before I go deeper into this, there is an important thank you for collaboration. And for my both of my colleagues who are Professor Chris McMillan, who was one of the key drivers also behind the idea. And I worked with him quite closely. And also a professor Julian Booker, who was also part of the team who contributed and we worked all together to actually generate, you know, the force behind this because this is really a place where without the collaboration of team, it wouldn’t really work. So a big thank you for them to start with and for their support for me as a as a program director.

But really the VLA approach that we have is different in the sense that we are trying to create a truly integrated mechanisms for the students to experience the interdisciplinarity and the knowledge in one space. And that’s probably one of the probably the key problems that we trying to achieve through this alright, is that the students need to understand that the knowledge is not modular in the way that we really do it. And the way traditionally we’ve been assessing students is that you get this unit you get the knowledge; you’re having to type or three types of assessment depends of assessment that’s, summative or formative and then you finish, you know, if you pass you got the unit included for you, but that’s not really mimicking the real world as soon as they go out. I mean, even before they start, you know, they go for employability and the employer sometimes, you know, put them in conditions where they make a new project or small project. And what they will see is they can work together with different people from different fields, then there’s straightaway, a test for our graduates on that sense. So we wanted them to experience that rather than assessing them on that as such, and that’s the most important driver for us.

So what we did is we looked at three key units for our program. So our program has three pathways. And these are three pathways has got infrastructure system, intelligent manufacturing, and energy for sustainability. So by design and default, we actually add interdisciplinary program by the type of the students we get. So now for the teaching block one, they’re going to meet all together, and they’re going to be studying together in teaching block two, that’s where they go into a specialization. So that’s the opportunity in teaching block one, where we teach them three key units, engineering, design and technology led by Professor McMullen. And then we have uncertainty or risk management, which is led by Professor Volker, and then myself where I lead the unit towards strategic business management for engineers. Now, each one of these units has its own intended learning outcomes. But we borrow from these intended learning outcomes of these three units. So for example, the new design and technology around 50%. And we say 50%, it doesn’t mean six, that means the three who, you know if you know, they could be five, right? But what we’re saying is around the 50%, of the and that’s talking about deeper assessment, if you like, rather than the threshold assessment, right, so 50%, from engineering design technology, 30%, from business, strategic business, and 20%, from uncertainty or risk management, and we create a piece of work where the students need to deliver on a global challenge project.

Now, that’s where it starts to get interesting, because let’s just finish on, on what’s what’s we’re doing before we go into the idea behind it. And now, what we have is the assessment now coming into place, we assess the student, or by using a consultancy like reports, or they have to deliver consultancy projects, for delivering sustainable development goals, we give the students a problem that is quite complex, the problem itself is not defined, and the solution is not defined. So the first thing they have to do is frame the problem. And that’s a very different experience from them. So we preparing them for what we call a real world experiences. And after that, the marks from this global project, return back into the units and feed back into the three units. And that’s where the feedback goes into into there in terms of marking and in terms of achievement of ILOs. And that’s the interesting part because that’s where your actual assessment, your actual learning feed to actually be assessed and appear in your unit assessment and individually at the same level. And that’s what makes it quite interesting.

So is that a group project they’re expecting your students to do? I’m really loving the the idea of almost you collaborating in order to produce this. And then of course, you’re almost modeling the behavior that you want your students to do, because of course, you in collaborating, making sure that this is authentic, making sure that this is something which the students can really aspire to, you’ve gone through the process in a way. So you can really see that in your students, which is excellent to hear.

And also we have one thing that’s quite important is the way that we support this project, because that’s where we have 12 weeks, so roughly, I’m not gonna go into exactly but roughly around, you know, six, six, let’s say six times where the students has to meet on their own individually together, and then six times where they actually appear in the class. But when they appear in the class, three of us appear as well. And what I have noticed is it creates a completely different dynamic. First of all, the students can call into expertise in one time, there was times where there have been questions that “Hadi, in your unit, you do this, but in this unit, you we do that”, and I remember once actually I called my colleague, you know, to say, Can you please come along here and can you let us have a conversation around this disagreement, you know, from the students point of view, obviously, and then, actually, over the talk, the students has clarified their understanding, but they had both of us there. So the interesting thing is they have all the expertise in one place and they see that how we think and how we understand the complexity of the knowledge becoming in one place and is a very, very different dynamic to the class.

And that must be so fulfilling for them as well to kind of see the fact that you’re thinking in different ways. And you might go about problem solving in different ways. And of course, as you say, you’ve got quite a fuzzy problem that they have to define, and therefore there’s quite a fuzzy answer. So all of this, and then plus seeing the fact that there’s different ways to approach it must be really, really beneficial to them, and the fact there isn’t the quick fix in terms of problem solving. So that’s really, really great to hear. And this kind of links to the next question. So we’ve kind of talked about employability, we’ve thought about the skills they want. So was there a specific problem that you’d notice that you were trying to fix here? while doing program level assessments?

First of all, we have this challenge that’s coming up to us quite a lot, we need graduates who are capable of understanding the complex world more and more than before. And engineering before there were more more towards technically technical knowledge focus, while we want now the students is to move away from technical knowledge, focus, and become still strong and strength in their technical but with awareness of the sustainability and impact, right, because your solution might be local, but your impact could be global. And we have seen this many times where actually, your solution that you do might impact someone else somewhere, you know, completely over the globe. So we need our students to understand the extent of their behaviors. And that’s really inspired by the report from the UNESCO 2021 report, which is really about curriculum developments, towards engineering for sustainable development and SDGs. And in that report, you will see a framework for the curriculum development where they actually define the way forward is to go for complex problems where there’s the solution of the problem is not there. And it’s actually require interdisciplinarity and require also rethinking structurally of your program to change the way that you deliver it. So that’s the first thing what we need, we need to create more engineering who are aware of their impact on the environment.

The second thing is, which is extremely important is that the students sometimes in the traditional way, they go through the program, but they do not really have a real opportunity for integrating the knowledge until actually they probably reach their dissertation, or probably at the end, they start to say, you know, some assessment might say to them, okay, draw in your learnings and that kind of things. While this one is a front loaded abilities to experience that, and take them through, if you like, the challenge of mentally thinking about even defining the overall problems and actually define the solution, which is very, very different. So you know, putting that in front that is extremely innovative in the way that we bring that into the students at the start, and that change the way they learn, although as they go into their specialist units. And obviously, the final point is around skills and around employability, what we want is to integrate that in an authentic way, as well as being authentic assessment, but with authentic practical based problems, delivering on real life like situations, and that really where students develop their thinking, they were a bit of their knowledge and all of that.

There is a side thing that you notice in that space that they use to become a resource, and knowledge resource. And I like that idea. And I wasn’t sure about it before, but now I think I’m a strong believer in it. Because I’ve seen places where the students, they bring their experience from industry, I’ve got got students who have been three or four years experience, and actually, because although that we ask the students to deliver on the units that actually no limits, you can actually bring your own knowledge and strengthen your work out or outcome, right. And they start to teach their colleagues or the team about that tool. And amazingly, I’ve seen amazing work and amazing exchange of knowledge. And even sometimes we learn from the students. I’ve been in the places where students prove to me that actually, what I was thinking is the right advice is not the right advice, and they bring the evidence and I accept it. So it becomes me vulnerable becomes the student as a source of knowledge. And that is absolutely amazing. But you have to be ready for that sort of seated challenge if you lie, but if you are, if you are open for that, and you are really wanted to bring the the students centered into the education of the learning experience, I think we need to start thinking differently. And probably I would say there’s many different ways but that seems to be doing the trick, for us.

Yes, definitely. I think that I think that’s that’s an excellent kind of view there of looking at how students can bring in their expertise, but also know that you’re fallible as well in the fact that you don’t have to know everything all of the time. You don’t have to have the answers on the tip of your tongue as soon as somebody has asked you a question, you have to go through the process you have to review you have to evaluate and how to think for yourself in terms of your practice. And it’s lovely to hear that your students are learning from one another, which I think is really, really excellent. It sounds lovely. Are there any other benefits of this model that you have noticed? So we’ve really talked about collaboration, we’ve talked about real world problems that anything else which you have come across in terms of the benefit?

One of the things that also I do in the program is a skill development. So I bring the skills down to the students on a program level, I’m not going to go into too much details of that. But really, this project because it’s a bigger project, and it’s another project, it brings them a place where they can experience the skills, and hence, evidence, the skills. So I do ask him through that model to evidence, the skills. And when the evidence, the skills, what happens is that when they go to their jobs, they have really real life in terms of their education. But that real example is, is is a real life linked as well. And that is a fantastic place where, where by the end of it, the students actually can do it straight. What I am actually started with saying the challenge of not just learning outcome, but competencies and skills outcome that’s really integrated in a very inherent way, it’s midway in the program. So that’s, that’s one of them, there is a indirect benefit. And it depends how you define it, how you believe in it, and probably that will create a bit of debate, but let’s just talk about it is, is the notion of decolonization of the curriculum, because when we, when we talk about global challenge project, we ask our students to bring the solution from developing countries, and these are list of countries, there’s a list of them, we all know it, they have to choose from it. And suddenly, what you will have is always as someone from the global was there, right? And I’ve seen it, where now, these students who were from that places, we call them international students, right? But really, are they because No, they are the local students, right for that experience within that project. So suddenly, the local students become the lead, and while we call home or other students all in the other side of it, and also it stops us from just bringing our own examples and our own, you know, view of the world and allows them to actually form the solution from the experience of the people. So that is really fantastic. And I’ve had students who actually took ethical approval, just for that global challenge project, and collected data from the local people in order to feed into framing the problem, and framing the story behind it. And frame the solution, which again, shows you that what we talked about is that driving a different perspective and respecting the solution for any sustainable growth for the people from the people, if you like, you know. So it depends how you look at it, but it’s indirect benefit as well, you know, for for the students to stand there as the leading students.

It’s always good to have your students listening to other perspectives, other voices and and then opening up their minds like you are opening up your mind to having more students and having more student voice, I suppose in your sessions are getting them to open up their ears to other voices, other perspectives, which can only be a positive thing when it comes to problem solving, the more perspectives that you have, the more different ways you can solve a problem, which is lovely to hear. So in terms of implementing program level assessment, did you come across any challenges that you’d like to kind of share with colleagues if they were going to take on this kind of project?

Yes, what was important when is understanding university system, there are going to be challenges, it depends where you’re applying it in, under what circumstances. So make sure that you speak to stakeholders, your administration system, your quality people and all that, because you will face some because this is a non traditional way of doing things. But we were able to reach a solution and the university supported us and the system supports you and then because that’s what was very important; so do have that conversation because you might need to think differently a little bit from what you try to achieve. So that’s one. The other one is there is a need for the unit directors to have a very deep conversation about their their knowledge about the links between them. And one of the key things in when and where that knowledge will appear in the unit. The good thing about this project because as I said it’s a complex project, which means that problem is unknown. The solutions are unknown, we kind of like introduce it in the first week and they have to start working on it from the second week. But really you know you can think about the first three or four even that are still in the stage of a framing of the problem. So you have an advance, you know space where you can actually introduce quite good knowledge to the students. So by the time they come into the fourth week, that’s, you know, or a fifth week around that period, you already have introduced to them some concepts, some thinking, so that they start to explore the potential directions of solutions, right, and start to link all together, by the time you are in the middle of the unit and start to move probably in the week seven, eight, you know, the knowledge has started to feed, you know, into that. And, of course, as I said, you know, the talk between these. So that’s, that’s very important.

The other thing is crucial, is thinking about your assessment, and marking matrix, you know, how are you going to mark something that complex? How are you going to mark something that is delivering over three units, so you need to move away from that marking criteria, where before it was very unit focused, and go into what I call is an outcome focus. So it’s about the discussions about the criticality is about the justification of what can go on and on. And that matrix is actually I worked myself out when my colleague, we deal with the full matrix, and that full matrix include every band, or what does it really mean. And even what the differences between them in the objective way, if you like, is to the students can clearly see the difference in the band and what they are actually measuring against. And it doesn’t need to be a long list, it needs to be a list that really gonna say, the deeper knowledge and the skills that you’re trying to achieve in your program. So there’s a bit of a specificity to it. So that’s the second challenge.

I would say the third and most important is, do not forget that students are not equal in the sense, not all of them has got the same abilities, the same skills. So you need to ensure that you are supporting all the students to get along with that initial stage. Because you will see that probably the majority of I would say sometimes all the students would come to you, “wait a minute, you know, we used to, you know, are given a problem. And I mean, even if the problem is a little bit fuzzy, but we have a problem, we know we’re going this one is literally we don’t have the problem. So I’m asking you almost to frame your own kind of”. Of course, we have boundaries for that. So we give them a report to you know, to read and give them a framework. So this is a very much defined in the sense of a low carbon transportation that we give them a report from a consultancy company, which is about 25 pages they have to read. So there is a framing but really is not. You know, the thing that’s so you have to ensure the support mechanism is there. But as I said to you, because the three of us appear, that really give us quite good resource utilization. So remove that pressure on the staff, because we three are there. And you know, and we can do it in a very good way. So I think there are probably other challenges, but probably I would say these are my three top ones when I talk to my colleagues or when I talk about this that comes always ask questions. So I will say, you know, these are my top three.

Yes, lovely, and it’s really interesting just to hear you talk about things like the marking matrix, or therefore working together on that in terms of practice, but also sharing that with the students, which I hope would help them with that kind of real uncertainty, because obviously, you’re kind of opening them up to a fuzzy problem that they have to try and create boundaries for. So giving them something which is really clear and defined in terms of and this is how you’re going to be assessed probably goes a little way to making sure they feel slightly less anxious about it.

Absolutely. And also keep the conversation with them that we are not looking for the solution. We don’t need even sometimes to agree with you. But the point here is, what is your justification, what drive your solution, what drives that thing behind your approach, right. And what I want to see is a series of justification and links as logical and, driven by the themes of the program, but also the knowledge of the units. And that’s what we’re looking for.

Thank you very much, Hadi. Just before we go, is there anything else that you want to add about program level assessment, anything that you feel that we’ve missed?

The only thing that I would like to add is to communicate with Mike Lee, the the benefits in terms of the students words, you know, I just want to share a couple of things that what the students says about it, you know, the students enjoy the program level assessment. To the extent where they think it’s the feature of the program, right? I’m just gonna read one of them here. “It’s been interesting to see how the techniques layered in lectures can be applied in a real world application to deliver high quality results”. Some other students they say, bringing tough material to life is you know, which is really, really fantastic. And when you talk to the students, even the students who did not do flying you know, project when you ask them, would you do that again? When they say yes, we will do that, again, you let’s say that you didn’t enjoy that part of it, and also from our state examiner as they say that this is an example of an exemplary project because it really designed a holistic way of integrating the knowledge. So I really encourage people to think about it. And we will be happy to support anyone who first but it’s a fantastic way to think about integrating the knowledge.

Fantastic. Thank you very much. Hadi.