DMU eAssessment Guidelines

 
Print Friendly, PDF & Email
document Important information for the academic session

next_sectionStaff guide to the use of DMU core technologies for anonymous marking in eAssessment

next_sectionTurnitin Guidance for Students for Handbooks

Guidelines for e-Assessment @ DMU

1. Overview

These guidelines are designed to support staff in the process of developing on-line assessments, or e-assessments, and should be implemented with the guidance of Enhancing Learning through Technology (ELT) Project Officers, as appropriate to the curriculum and programme context.

e-Assessment “is defined as the use of electronic systems for the development, operation and delivery of accredited qualification assessment or the collection of performance evidence, which contributes to the awarding of a unit or an accredited qualification” (QCA, 2007, p. 5). These systems may include: Blackboard tests; Blackboard discussion boards, blogs or wikis; multimedia tools; Turnitin; and external, web-based tools, such as blogs or wikis, used to prepare portfolios. In embedding these systems, the use of e-assessment has technical, administrative and operational implications.

This document sets out guidelines for the use of e-assessment at DMU. All staff involved in the design, presentation, setup and invigilation of diagnostic, formative and summative e-assessment are advised to adopt them. Please note that these guidelines should be read alongside the University Assessment Policy and Procedures, which highlight that assessment should be: transparent; equitable; valid; reliable; just; and sustainable.

2. Validity and Reliability

  • Programme teams should ensure that e-assessment produces a valid and reliable measure of a candidate’s achievement of specific learning outcomes. The choice of assessment method should be independent of the technology on which it may be based.
  • Questions should be carefully designed with due regard for the material being assessed and the level-of-study.
  • Where on-line, summative assessment is planned, there should be formative opportunities for students to understand both the process and content of this type of assessment, and gain the appropriate feedback.
  • Students should be made aware of: the number and type of questions to be used; the scoring logic for individual items and the overall assessment; the contribution of this assessment to the overall module; logistical issues like timings, restraints on navigation, and permitted resources; and, any personal identification they will require.

3. Technology/Security

  • Programme teams should review the security of e-assessment systems with Faculty ELT POs and ITMS, to ensure authentic test outcomes. Procedures must be in place to assure the security of hardware and software and the integrity of assessment data, in particular where non-DMU systems are being used.
  • Where students are sitting phase tests either on- or off-campus, programme teams should ensure: 
    • appropriate security against plagiarism, collusion or copying;
    • enough lab capacity to cope with hardware or software failures (10% spare capacity is recommended). Timetabling enough time and space for assessments in labs is crucial; and
    • that on-line, assessed work can be properly attributed to individual students.
  • These same issues apply where programme teams are assessing students based upon outcomes produced through non-DMU, web-based technologies.
  • Where students are sitting phase tests either on- or off-campus, programme teams should consult with Faculty ELT POs and ITMS, to assure themselves of the robustness of the local technology that supports an assessment.
  • On the use of Turnitin on mobile technologies: 
    • Some universities have reported marking being lost when staff use the Turnitin mobile application for iPad;
    • ITMS are working on a policy for bring your own device/using your own device for work purposes, and will be creating procedures for this challenging area;
    • the iPad application for Turnitin GradeMark was enabled by iParadigms, who manage the Turnitin application. It cannot be switch-off by DMU. It is still subject to testing and where they are using the tool on an iPad, users should report any issues to their Faculty ELT Project Officer;
    • Given that sensitive data related to student assessment/records are stored on or accessible from devices like iPads, we advise that it is not used for Turnitin/GradeMark at this stage.
  • Teams must address the secure transfer of work to external examiners. Access to DMU systems, like Blackboard and Turnitin, can be arranged to give secure access for external examiners to module-level work for moderation purposes. However, as an alternative, teams may wish to copy and email, or print-off, examples of student work for moderation by external examiners.
  • Where scores or grades are communicated to students, the Blackboard Grade Center provides secure, personalised access to indicative marks. MyDMU provides secure, personalised access to validated marks. These system provide secure attribution of scores to students.
  • Where programme teams are assessing students based upon outcomes produced through non-DMU, web-based technologies (e.g. Web 2.0 tools), they should: 
    • assess any impact of the terms of use for a specific application (e.g. Blogger, wikispaces or Facebook) on their curriculum and assessment strategy;
    • implement safeguards against external threats like viruses and hacking;
    • induct students into the mechanics of the specific tool, and how it will be used for assessment;
    • clarify issues around plagiarism, collusion, IPR and copyright with their students; and
    • recognise that the security of, and access to, the hardware and software used to hold candidates evidence and assessment outcomes, cannot be guaranteed. Therefore, teams are advised to implement backups of such evidence on DMU systems. Individual archiving of assessment-related activity is critical.

4. Operation of Assessment

  • Anonymous Marking (AM) for all summatively-assessed coursework is the norm across DMU. This can be managed via a paper-based cover-sheet or curriculum teams may wish to opt-out of AM after talking to the Chair of their PMB. There is no requirement for staff to manage AM through e-assessment. However, there are three on-line solutions available for staff, which are detailed at: http://celt.our.dmu.ac.uk/anonymous-marking-for-eassessment-the-staff-guide
  • Both Blackboard and Web 2.0 tools, like blogs and wikis, can engage students in the assessment process though personal ownership of feedback, and self-and peer-assessment. However, clear links should be made between assessment modes (diagnostic, formative and summative), the technologies that support them and expectations of students.
  • Where staff use Blackboard or a non-DMU, web-hosted application to deliver summative assessment, the following issues should be addressed for both on- and off-campus students. 
    • Students should be given opportunities to familiarise themselves with the interface to be used.
    • The format of the test should be clearly explained to students, with a specific focus upon how to save answers, restrictions surrounding the test (time, multiple attempts etc.), and how to submit.
    • Deployment of tests using assessment pools, and the randomisation of questions during tests, may reduce the risk of collusion, but this increases the need for staff to be assured that all questions, and the order in which they appear to students, are of a comparable level.
    • Where browser windows cannot be locked down or web-access restricted, students should be made aware of the penalties for searching for answers in closed-book exams. Appropriate strategies for invigilation and ensuring that exam conditions are maintained are critical. The invigilation team should include a senior academic with knowledge of both the assessment and the assessment software.
    • The time limit allowed for the examination may need to be flexible to compensate for any delays (technical, emergency, etc). Additional time may be permitted for some students, e.g. those using assistive technology. Therefore, IT suites should normally be booked for 30 minutes before an assessment and at least 15 minutes after the last assessment is scheduled to be finished.
  • An identical assessment may be run in sequence to make best use of facilities and staff availability. In this instance, candidates in the first session are not permitted to leave the examination room, until everyone has finished and the next group moves into the room while the initial group(s) leave(s).
  • Programme teams should develop a contingency in the event that either DMU or external, web-based e-assessment systems fail at the point of delivery. This may include, for instance, postponement or paper-based delivery. For summative assessment this should be agreed with the Faculty Head of Quality
  • Programme teams should explain to students the procedures for managing late e-submission of work, where e-submission is the sole format or is being used alongside paper submission. Teams will also need to have a clear procedure for managing submission where technical systems fail (see Appendix 1).
  • As soon as possible after a CAA assessment has finished (within 60 minutes), a copy of the results must be made and transferred to a second, secure hard-drive.
  • Where collaborative partners are involved in e-assessment, programme teams should provide their staff with clear guidance on format, operation and IT requirements.
  • Where distance learning students are in the process of applying and enrolling on DMU programmes, they should be made aware of the minimum hardware, software and connectivity requirements for their assessment.
  • Where e-portfolios are being developed, programme teams should ensure that students can save evidence in a variety of media, in a secure, personalised system. For instance, this might include the Blackboard-based wiki or blog. Note that where evidence is stored on Blackboard, it will be backed-up and maintained for a maximum of 6 years, in-line with the University’s regulation on programme enrolment.
  • Online summative assessments should be managed via secure hardware, rather than mobile devices. In particular, taking summative assessments on tablets over a wireless connection is high-risk and should be avoided. Where staff wish to make formative assessments available to students using their own technology the following apply.
  • Mobile tests are available to students using iOS 4.0+ and Android 2.2+ devices only.
    • Mobile tests can be taken with both the Blackboard Mobile Learn app and via the standard Blackboard Learn environment on a desktop computer.
    • While Mobile tests are designed to be taken by students on mobile devices, mobile tests can only be created using the standard Blackboard Learn desktop environment.
    • Timed tests are not available via mobile devices
    • Mobile tests are not available via mobile web browsers, e.g. Safari on the iPad
    • Standard tests created with only mobile-compatible question types will automatically be available for your students to take via Blackboard Mobile Learn. Any test that contains non-mobile-compatible question types will not be available for students to take via mobile devices.
    • Students will need to upgrade their Blackboard Mobile Learn app on their mobile device to the latest version in order to access Mobile Tests. There is no cost for them to do so.
    • Tests that are not compatible with mobile devices will appear in the course map with a message directing students to take the test through a web browser.

5. Electronic submission and digital receipting

  • Where staff use Blackboard-Turnitin for electronic submission the following options are available: 
    • Assignment: where users “view/complete” an assignment, an electronic receipt is presented and the submission is stamped as “completed” in the assignment window. This is not date/time stamped;
    • Email submission through Blackboard generates a date/time-stamped receipt, which can be printed; and
    • Turnitin: where users “view/complete” a TII assignment, an email receipt is presented and the submission is date -stamped in the relevant TII window.
  • Where teams use non-DMU, web-hosted applications to prepare for or deliver assessment, submission and receipting procedures should be clearly explained to students.

6. Anonymous submission

  • Anonymous Marking (AM) for all summatively-assessed coursework is the norm across DMU. This can be managed via a paper-based cover-sheet or curriculum teams may wish to opt-out of AM after talking to the Chair of their PMB. There is no requirement for staff to manage AM through e-assessment. However, there are three main on-line solutions available for staff: Turnitin, Blackboard Assignments or Blackboard Tests, which are detailed at: http://celt.our.dmu.ac.uk/anonymous-marking-for-eassessment-the-staff-guide 
  • Anonymity at the point of marking scripts can only be provided where e-submissions are made to a third party who is not involved in marking. It is not possible at present for Blackboard, TII or email to provide true anonymity although, submission using p-numbers alone is possible.

7. Accessibility

  • Programme teams must have policies and procedures in place to ensure that disabled learners are not treated less favourably than non-disabled learners when implementing e-assessment, in terms of its format, operation and IT requirements. Faculty Disability Co-ordinators and Student and Academic Services are key sources of information on this issue.
  • Where appropriate, and when requested, a non-computer based assessment may be made available to a disabled candidate.

8. Health and safety

  • PCs, chairs and the general assessment environment must conform to standard safe environmental guidelines (clear screen, adjustable screen, adjustable chair, good lighting and ventilation).
  • Candidates should be advised to take ‘micro-breaks’ to relax their muscles and eyes i.e. stretch, look up at the ceiling.
  • If the assessment lasts for longer than 90 minutes, then there must be a provision for a break away from the PC. This may require supervision of candidates during the break, although facilities such as the provision of separate tests or blocks of questions may be used.

9. Further guidance

Staff seeking further guidance or professional development should contact their Faculty ELTPO.

Options if Turnitin fails

The following options are available.

  1. Give extended time for resubmission – this is at the prerogative of the tutors
  2. Paper based submissions (also enables anonymous marking)
  3. Blackboard assignments (also enables anonymous marking)
  4. Email (does not enable anonymous marking)
  5. Campus Pack Individual wiki/blogs or Blackboard Journals could be utilised to provide a private dialog for submission/feedback (does not enable anonymous marking)
  6. Blackboard tests: this will only work if question settings are set to allow ‘Add images, files, and web links to answers’ (in theory this will enable anonymous marking)

Note: in each of the above, there will be an impact on students. There may be more impact to staff and students in routes 3, 5 & 6, if tutors are unfamiliar with how to use these features.

Information about each of these routes is available via Faculty ELT Project Officers. However, the contingency for the potential unavailability of an institutional e-submission route will be locally-implemented, rather than institutional.

Faculty Communication from CELT and ITMS

Once it has been recognised that there is a service disruption to Turnitin, the following DMU-wide communication will follow.

  • ITMS, as DMU’s System Administrator, inform CELT Team of disruption (NOTE: CELT monitor Turnitin social media as default).
  • ITMS to raise ‘support help ticket’ with Turnitin, if necessary.
  • Head of CELT or CELT nominee issues all staff Blackboard user email. This agreed form of words to staff will include: the issue; the impact on assignment submission (including anonymous marking).
  • Faculties will need to decide about alternative submission routes or if a deferral of submission times/dates is to be agreed (see 1 above). This might be taken globally or by Programme Management Boards. Academic Practice Officers will be notified about downtime by the Faculty ELT Project Officer.

The following message will be placed on Blackboard and MyDMU for students.

Intermittent Turnitin Submission Issue

If you are submitting assignments to Turnitin you may experience problems in uploading your work. Please make a note of, or print-off, any message you receive from Turnitin and re-attempt the upload at a later time. All academic staff users of Turnitin have been informed of this intermittent issue. Students should still submit work to Faculty Offices/Advice Centres if directed to do so by their module teams, and should upload their work to Turnitin at the earliest possible opportunity.

This will be managed by CELT and ITMS.

Module/programme teams are responsible advising their students if there are any assignment submission changes as a result of the service disruption (i.e. extended date submission for electronic submission, with hard copy submissions are expected through faculty offices).

The Faculty ELT Project Officer will discuss updates/system risks with APOs and Chairs of FLTCs, as well as Heads of Study. The Head of CELT will communicate updates to staff users of Blackboard/Turnitin. The CELT Project Officer will update Library and Learning Services staff. When the system is fully functional as defined by a communication from Turnitin, CELT Head or CELT nominee will send an all users email.

To agree procedures where the failure of systems impacts processes like anonymous marking;

FLTCs and APOs need to consider Alternative on-line submission routes as contingencies’ and their implications for anonymous marking, given that options beyond Turnitin are limited.

Programme and Module Leaders might usefully consider procedures or alternatives with the Faulty ELT Project Officer, including clear communication with students.

Faculty APOs need to understand alternative assignment submission procedures for anonymous marking, so that they are aware of circumstances surrounding any change.

NOTE: the failure of systems might impact staff workloads and agreed turnaround times.