Chapter 7 – Hands-on assessment in a hands-off world: strategic implementation of inclusive assessment in an online environment

A month before the first pandemic lockdown, JISC published a paper on assessment with the conclusion that it needed to be ‘smarter, faster, fairer and more effective’ (JISC, 2020). The paper argued that assessments needed a technological overhaul by 2025 to become more authentic, accessible, appropriately automated, continuous and secure.

The radical overhaul that JISC were arguing for came much sooner than expected due to the pandemic. Universities were closed, and exams and assessments could no longer take place as intended. As the pandemic unfolded, assessment remained a core concern for students, with 74% of students worried about how it would impact their final qualification (NUS, 2020).

This chapter will explore how hands-on assessments were forced to become hands-off. Through our roles as senior academic leaders, it will begin with a strategic response to ensure there was equality and inclusivity in assessment design whilst maintaining quality and standards.

The strategic approach to pandemic assessments

Why were formal university exams traditionally taken on campus with written exam books in a digital age? Why were exams traditionally delivered in blocks ranging between 1-3 hours? Why were exams tending to follow tried and tested approaches of essays, short answers, laboratory tests, and multiple-choice formats? Although scholars had asked such questions for a number of decades prior to the outbreak of the Covid-19 pandemic (Elston & Johnston, 2002), there had been little significant change, despite evidence suggesting that traditional assessment methods were neither fit for purpose for employers nor fully beneficial for students in terms of addressing learning needs (Williams et al., 2014; Jorre de St Jorre & Oliver, 2018).

At De Montfort University (DMU), the formal exam schedule that was initially published for the 2020 summer exam schedule had increased to 313 formal exams and 41,441 student sittings. The reality of the pandemic meant that all summer 2020 exams were either taken online over a 24-hour window, or delivered in a different assessment format. While the Faculty of Arts, Design and Humanities (ADH) had very few exams, all of these exams had to be adapted for an online format, with the exception of the need for in-person exams in a few specific subjects, such as English, History, and Dance.

Key considerations

One of our key considerations early in the pandemic response was how to structure our academic delivery to ensure that it reflected the needs of all students. We developed a range of guidance that included a ‘top 10’ principles of online pedagogy, the first of which was a recognition that there is no such thing as a typical student (DMU, 2021a). As an advocate of Universal Design for Learning (UDL), we needed to ensure that all students had an equal learning experience. Consequently, we updated our learning and teaching glossary to consider online learning in the context of the pandemic (DMU, 2020b; Labbe & Blair, 2020). Given the significance of online learning and assessment, we invested considerable resources in providing students with an extensive IT handbook that set out the approach for teaching and learning and the IT resources and software available. This included 2,600 computers across 180 IT labs, studios and other teaching spaces, plus a further 1,000 computers in open-access and library spaces. The University established a service called ‘Your DMU Labs’ which enabled students to remotely access physical PCs in labs and open access spaces on campus that were not in use for face-to-face teaching. We also recognised that many students would not have access to computer facilities and established a laptop loan facility (DMU, 2020a).

Central to our thinking was the importance of accommodating different student needs with regards to technology and access. This played a large part in conversations early on for teaching, but became even more of a concern when considering remote assessments. Access to data or reliable internet connectivity was as much of a concern as access to a computer. It is no surprise that in the year after the pandemic, the JISC Annual Student Insights (2021) raised the digital divide as a critical issue, with 63% of students reporting that they had problems with poor internet connections, 30% had trouble when accessing online platforms, and 24% faced problems with mobile data costs.

Case study: Making Media at DMU

To illustrate this approach, we will use an example of Media Production, an undergraduate programme, which is described on the institutional website as being an ‘industry-focused course, which provides you with the technical foundations and practical skills required for a career in a rapidly changing media landscape’ (DMU 2021). Some assessment was designated to take place within the media production studios, either audio booths or television, and others relied heavily on industry-standard equipment that students could access via the university loans unit. Across levels, there was a range of assessment due that all needed to be thought of in a different way.

For a programme that relies on being industry-focused it was essential that industry expertise was brought into revising our assessments. Recognising that students would not be able to be assessed on using high-end equipment or studios, consultation with how students could meet their learning outcomes and needs of industry with adaptations was essential. Staff engaged early on with industry contacts and networks to see how production was adapting so that our students could adapt in the same way.

As with other sectors, the British Film Institute and industry organisations like the Production Guild and BECTU issued guidance in consultation with the Department for Digital, Culture, Media and Sport on the Covid safety guidelines that productions must follow. Students were able to see how production companies were responding to the challenges, witnessing live chat shows using video technology, which in turn enabled them to learn from an increase in their use of mobile filmmaking technology.

The mobile phone became central to the media production student’s toolkit. The mobile phone became central to the media production student’s toolkit. This enabled reiterating to students that quality came from knowing how to use the tools rather than the quality of that tool themselves. It helped that mainstream broadcasters were also using phones to capture and broadcast content, with one example being ITV’s Lockdown Stories where the cast filmed their scenes from their own homes, with directors advising them remotely (Craig, 2020).

More than anything else, this brings to the fore a key question around the digital divide and the gulf in learning opportunities that is widening for those students who do not have adequate technology. At DMU we sought to tackle this challenge by providing students with access to technology, by posting necessary items directly to them. We also sought to address head-on any potential implications of a technological divide by reframing the way in which  students would be assessed by making it clear that the technical quality would not be a barrier to achieving a high grade. Rather, composition of shot, framing and storytelling were identified as the key factors, thereby ensuring that students with different access points would not be disadvantaged.

What we found through this process was that the work that was produced was of an exceptionally high standard. Indeed, it seemed that in some ways the pandemic focused students’ attention even more on their craft, enabling them to focus less on the availability of big tech and more on their skill. One such example was that of a student who used a mobile phone to produce a six-minute short film which investigated the potentials of sound design to create atmosphere.  They used their housemates as the cast, and the locations were restricted to those where they were based. Despite these strictures, the production limitations did not impact the quality of the idea nor the work that was produced.

This example highlights the way in which a new spirit of creativity was unleashed by the pandemic, where the more structured learning environment of the classroom gave way to a landscape that offered students different opportunities which in many ways reflected more accurately the challenges that they might face after their studies. Such a focus on a ‘practical turn’ was central to students achieving success in their assignments, as emphasised in the reflections of Simon Walsh, the programme leader:

Some incredible creativity – and I believe that those who (particularly) graduated at the end of the 20-21 session will go out into the world knowing – really knowing – that you can have all the fancy kit in the world, but without talent, without ideas, without creativity – without the human holding the kit, or working the software – it will all come to nought. I think the experience, as difficult as it was, will have taught them all that valuable lesson: they are the important part of the equation. They say that bad workmen blame their tools; bad workmen also over-rely on them. Good workmen understand that they are the vital element – it’s not the parts, it’s the labour!

Discussion

At DMU we wanted to understand students’ views of the shift to online formal exams that had taken place in 2020 and their continued use in 2021. We undertook a survey between 25 June and 4 July 2021 to find out what had worked well and less well over the last two formal exam cycles. Of the 1,327 students who completed the survey, 90% were undergraduates (1,194). The vast majority of respondents were full-time students (1,280). In terms of what worked well, the top three factors that students ranked in order of importance were: choice to complete the exam when it suits (N=905, 68%), having 24 hours to complete the exam (N=742, 56%), and ease of process (N=690, 52%). A key theme that stood out in free text comments in favour of online exams included enhanced inclusivity, such as in relation to students with learning support needs, as well as being more flexible for students who have to commute or have caring responsibilities. Students also noted that online exams had a positive impact on their emotional wellbeing and mental health as they were less stressful. Looking to the future, while 75% of students considered that exams should be online, it is also the case that some modules may be required to be taken face-to-face. This would include, for example,  practical and objective structured clinical examinations (OSCEs) and similar practice-focused modules.

Three months into the pandemic, JISC and Emerge Education (2020) published their insights into student assessment during the pandemic. It concluded that although it was a positive move to find that assessment had become ‘faster, smarter and more accessible’, the changes made were not necessarily sustainable long-term. As we now look towards a future beyond the pandemic, we need to find a way that these lessons can be applied, and we hope that the sector does not simply follow the path of least resistance and return to pre-pandemic practices. It is in this context that we encourage peers to review their assessment strategies with inclusive approaches in mind, both relating to formal exams and more practical led assessments.

The lesson of the pandemic tells us that change can bring significant benefits in terms of how we think about our teaching delivery. But the pandemic also reminds us of the importance of not becoming stuck in established working patterns or ways of thinking. Just as it took the pandemic to really force the issue of teaching and assessing in a digital age, as we begin to live beyond the pandemic it is important that we do not forget the many positive benefits that these changes have created. This includes delivering more inclusive forms of assessment, flexible pathways for students, hyflex delivery options and dynamic pedagogies. Taking the time to analyse what has worked, and listening to students and colleagues who have delivered, will enable us to make those decisions. Staff concerns about online exams tend to focus on issues such as enhanced opportunities for cheating. But just as technology enables exams to happen in an online environment, it can also provide the framework to protect against cheating, through better assessment design and the use of proctoring software. As we have argued in relation to university leadership, there is a need for academic staff to think in an adaptive mindset about how they teach and assess in the future rather than reverting to established ways of working (see for example Blair & Jones, 2021, on this topic).

References

Blair, A., & Jones, S. (2021, September 8). University leaders need to demonstrate an adaptive mindset. THE Campus.
 https://www.timeshighereducation.com/campus/university-leaders-need-demonstrate-adaptive-mindset

Craig, D (2020, April 22). Sheridan Smith, Robert Glenister and Angela Griffin to star in ITV’s Isolation Stories. Radio Times.  https://www.radiotimes.com/tv/drama/sheridan-smith-robert-glenister-and-angela-griffin-star-in-isolation-stories/

DMU (2020a, March 24). Coronavirus update: Eric Wood news and laptop loans.
https://www.dmu.ac.uk/about-dmu/news/2020/march/coronavirus-update-eric-wood-news-and-laptop-loans.aspx

DMU (2020b, June). Teaching online glossary.
https://www.dmu.ac.uk/academic/centre-academic-innovation/staff-support-for-remote-teaching/teaching-online-glossary.aspx

DMU (2021a). Top 10 principles of online pedagogy.
https://www.dmu.ac.uk/academic/centre-academic-innovation/staff-support-for-remote-teaching/online-pedagogy-10-principles.aspx

DMU (2021). Media Production.
https://www.dmu.ac.uk/study/courses/undergraduate-courses/media-production-bsc-degree/media-production-bsc-degree.aspx

Elston, L. and Johnston, B. (2002). Assessment in Universities: a critical review of research. LTSN Generic Centre.
https://eprints.soton.ac.uk/59244/1/59244.pdf

JISC. (2020, February 10). The future of assessment: five principles, five targets for 2025.
https://www.jisc.ac.uk/reports/the-future-of-assessment-five-principles-five-targets-for-2025

JISC/Emerge Education. (2020). Assessment Rebooted.
https://repository.jisc.ac.uk/7854/1/assessment-rebooted-report.pdf

JISC. (2021). Student digital experience insights survey 2020/21.  https://repository.jisc.ac.uk/8487/1/Student%20DEI%20HE%20report%202021%20Final.pdf

Jorre de St Jorre, T., & Oliver, B. (2018). Want students to engage? Contextualise graduate learning outcomes and assess for employability. Higher Education Research and Development, 37(1), 44-57. https://doi.org/10.1080/07294360.2017.1339183

Labbe, J. and Blair, A. (2020, July 1). What is a Lecture? And other mysteries. QAA.
https://www.qaa.ac.uk/news-events/blog/what-is-a-lecture-and-other-mysteries

NUS. (2020). Covid-19 and Student Survey Report. https://www.nusconnect.org.uk/resources/covid-19-and-students-survey-report

 Production Guild. (2020). https://productionguild.com/member-resources/knowledge-base/british-film-commission-working-safely-during-covid-19-in-film-and-high-end-tv-drama-production/

Williams, P., Wray, J., Farrall, H., & Aspland, J. (2014). Fit for purpose: traditional assessment is failing undergraduates with learning difficulties. Might eAssessment help? International Journal of Inclusive Education, 18(6), 614-625. https://doi.org/10.1080/13603116.2013.802029