ONLINE EXAMS WITH STANDARD SOFTWARE IN SOCIAL SCIENCE COURSES

The 2020-2021 Coronavirus pandemic has had a significant impact on teaching worldwide, prompting a mass conversion to online instruction. Universities have been greatly impacted by this. While the conversion of classic in-person, presence-based teaching formats, such as lectures and seminars, to online teaching went relatively smoothly, written exams still represent a particular challenge. Due to the nature of online instruction, it would be helpful to have online exams that can be taken from any location. The goal of the QLU-eQuiz-Project is to use standard software to create a testing environment in which university examinees can take online exams. This is not just a technical stopgap solution to the current, pressing problems of conducting examinations online during the pandemic; the project also aims to address future development toward hybrid forms, focusing on exams for social science courses – specifically, here, courses in Logistics and Business Administration. Furthermore, the project seeks to identify problems that can occur in the context of online exams and to discuss proposed solutions. For this, a series of mock and "real" online exams were conducted as QLU-eQuiz exams, both with smaller and larger groups. Part of the project compares grades on online exams with previous inperson exams. 114 5 TH INTERNATIONAL SCIENTIFIC CONFERENCE IS IT TIME FOR A TOTAL RESET?.


Introduction
At the Ludwigshafen University of Business and Society, it was decided not to hold any in-person exams during the 2020/21 winter semester due to the Coronavirus pandemic. This decision was in line with practices at most other German universities. The Senate of the Ludwigshafen University of Business and Society passed a resolution to this effect on December 16, 2020. Alternatives to in-person, written exams were specified, including oral exams, homework, open book exams, and online exams. Due to the high number of participants needing to take written exams, it was decided to develop a software solution for online exams, which was called "QLU-eQuiz".
Universities use a wide variety of exam types for assessment. For the purposes of this paper, two types of exams are differentiated. First, there are "paper and pencil exams" (PP-LR), which are usually taken in-person on university premises: these are considered the "classic" exam format. Second, there are online exams, which are taken in private, online settings (OL-AH): these represent a new method of taking written exams (Fischer & Dieterich, 2021;Ilgaz & Adanir, 2019;McLaren & Reilly & Uhomoibhi, 2019).
With online exams, the examinee finds a private space, usually at home, to complete the exam. The exam itself is made available via a computer application. Since the Internet is usually used for this purpose, it is generally referred to as "online exams." Additional video conferencing software is required in order to supervise examinees and to carry out identity checks online. Due to particularities of the online exam format, it is especially important to prevent cheating, e.g. through the unauthorized use of telephones or documents such as books or course materials (Fask & Englander & Wang, 2014). The process of monitoring of online exams is henceforth referred to as "proctoring" (Schlüter, 2021). There are numerous academic discussions on proctoring, focusing mainly on legal aspects (Albrecht & McGrath & Uphues, 2021;Bredow & Fokken & Haug & Himmelrath & Preker, 2021;Deutscher Hochschul Verband, 2021;Schwartmann, R. 2021;Yousef, A. & Chen & Liu & Hsu & Liu, 2015). These discussions, however, are beyond the scope of this paper.
The software programs were selected for various reasons, including the availability of appropriate university licenses and the authors' solid experience with and knowledge of software application and programming. Furthermore, there were already preliminary experimental tests that had provided students with support in completing practice exercises with self-programmed applications. The software programs used included Microsoft Word (word processing), Microsoft Excel (spreadsheet), Microsoft Outlook (email), Qualtrics (online surveys) and Zoom (video conferencing).

The QLU-eQuiz Project
Within the framework of the QLU-eQuiz-Project, numerous tasks had to be completed, which can be categorized in the stages of preparation, implementation, and evaluation.

Preparation
The starting point for the use of the QLU-eQuiz in the 2020/21 winter semester were exams that were "classically" designed as PP-LR and were already printed out and ready to be copied for the upcoming examination period. After in-person exams were cancelled, the challenge facing university staff was to convert these PP-LR exams to OL-AH in such a way that all participants, especially examinees, would not be disadvantaged in any way by this form of exam. In the following, the steps taken to conduct the QLU-eQuiz are briefly described.
The work entailed in transferring multiple choice questions into the QLU-eQuiz was relatively straightforward, as a proven software-based solution was already available for this purpose, which is also used for PP-LR. Multiple-choice questions are stored in Excel tables, which are then transferred to Word for designing and printing the PP-LR. The copy & paste function can be used to import these questions into a Qualtrics script. Although Qualtrics has its own scoring model for multiple choice questions, it was not used. The Qualtrics feature of randomly displaying multiple choice questions was used, which allowed each examinee to be shown the questions in random order (selected by Qualtrics). Randomizing the order of questions makes it more difficult to cheat, as examinees cannot compare what answer they chose for Question 1 because Question 1 has different content for each examinee. An example of this would be examinees sharing "Q1,2,3" as an abbreviation for "Question 1, correct answers 2 and 3." Due to the randomized presentation of questions, cheating becomes much more complicated, if not downright impossible.
Excel tables were also used for designing calculation exercises, but a number of adaptations and extensions were necessary. Only the issue of "task variations" should be mentioned here. In order to prevent attempts at cheating by communicating solutions for the calculation exercises, the tasks in Excel were redesigned in such a way that a whole set of variations was created for each calculation question. This means that the specifications (numerical values) for a calculation task were automatically varied within a given range, thus resulting in different solutions. By default, ten different exam variations were created and randomly assigned to examinees during a later process. In this respect, QLU-eQuiz exams can be considered personalized exams.
The following graphic shows a screenshot of a QLU calculation task. In the upperleft corner of the screen is the selection menu with all the questions to be completed in the exam. To the right is the text of the exam question with the personalized information (numerical values). At the bottom of the screen are the input fields for calculating the solutions to the exercise. After importing the Excel data into Qualtrics, the task was to program a Qualtrics script for each exam. This does not require any programming knowledge, as Qualtrics is designed to be operated very intuitively, with a user-friendly ("no-code") approach. Due to time pressure, the types of Qualtrics questions used was limited to the variants "Multiple Choice" and "Text Input with Form Fields" as well as "Select, Group and Arrange" with the "Drag & Drop" function.

Implementation
The QLU-eQuiz-Project was based on various courses from the curriculum of the Bachelor's degree programs in Logistics and General Business Administration as well as the Master's degree program in Logistics in the Department of Services and Consulting at the Ludwigshafen University of Business and Society. Specifically, the QLU-eQuiz-Project was used to conduct six online exams with a total of 475 participants. The largest numbers of participants were in the exams for Cost Accounting, with 171 examinees, and General Business Administration (ABWL), with 153 examinees.
The implementation of a QLU-eQuiz exam is divided into two parts. The first part includes the completion of the online exam by examinees, i.e. accessing and solving the exam questions, as well as recording the solutions. The second part includes exam supervision by university staff (proctoring).
Mock exams were administered to practice both components of the QLU-eQuiz exam for respective course participants. These mock exams, which took place during lectures, were designed for a duration of 30 minutes and offered students not only the opportunity to familiarize themselves with the QLU-eQuiz and the other framework conditions of an OL-AH, but also provide examinees a chance to check their performance. Thanks to the largely automated grading of such exams, students receive their results by email in a timely manner. This message contains sample solutions in addition to the actual grading. Accordingly, if examinees get questions wrong, they can identify potential gaps in their understanding of course material and use the sample solutions in order to better prepare for the "real" exam. The QLU-eQuiz exam is initiated with an invitation link sent by email to exam participants in advance of the upcoming exam. After entering a password, participants receive access to the QLU-eQuiz exam.
Participants can choose the order in which they complete exam questions. In the case of multiple-choice questions, examinees simply select solutions by clicking on selection boxes. For calculation tasks, participants follow a procedure in which they transfer the information from the screen to so-called templates that have been printed out beforehand; alternatively, blank paper can be used. Examinees should concentrate on solving the questions and transfer the results into the corresponding response fields of the Qualtrics script.
In the Qualtrics selection menu there is a menu item that allows users to show all entered responses in an overview display. Corrections are possible at any time during the exam period. When the time period for the exam ends, the participants are asked to end the QLU-eQuiz exam. This is performed by the examinee, who selects the appropriate menu item and then confirms that he/she wants to end the exam. Afterwards, the participant receives an email with a message that they have successfully completed the exam.
The following table presents an overview of QLU-eQuiz exams conducted in the 2020/21 winter semester:

Evaluation
After the end of the exam, the data sets generated by Qualtrics can be exported as Excel files and used for evaluation purposes. In the first step, raw data sets are validated using Excel to check for the completeness of formal data and exam responses. In the next step, student responses to multiple-choice questions and the calculation-based questions are evaluated. In the final step, responses are scored in order to calculate point values for the exams. Already at the beginning of the project, it was clear that a separate grading process should not be programmed for each exam. The evaluation module of the QLU-eQuiz-Project currently grades a maximum of 20 multiple choice questions and ten solution fields, and an extension would be easy to program. An obstacle hindering the development of a fully automated process for exam correction is the fact that examinees' responses do not always exactly conform to exam instructions (e.g. inputting solutions "as a whole number without decimal places"). The variety of forms that responses can take is displayed in the follow table: the "Entry" column shows the original responses from QLU-eQuiz participants, which were entered as the solutions to calculation-based questions. Due to the "ingenuity" of the examinees and their unstandardized responses (which sometimes deviate from exam instructions), manual correction by exam graders may still be necessary even with "standardized" online exams. A simple solution to this problem would be to provide correct solutions for calculation-based questions, e.g. a percentage, in the form of a multiple-choice question with four possible answers in which three percentages are wrong and one is correct. The clear advantage to this approach is that exam correction can be easily automated, similar to a "normal" multiple-choice question. The disadvantage of this approach, however, is that the participants can also guess the correct solution and do not necessarily generate the answer on their own. Another issue is whether converting these responses to multiple-choice questions would exceed the proportion of points from multiple-choice questions permitted in written exams. Exam regulations at the Ludwigshafen University of Business and Society, for example, limit the proportion of multiple choice questions to 50 percent of the total points achievable in an exam.
Another possible solution to these issues is to highlight incorrect entries when the student formulates their response using a Qualtrics script that prompts them to edit their response with the proper format. This approach will soon be tested.

Initial evaluation of the QLU-eQuiz
Detailed exam results are not listed. In comparison to the exams from the previous semester, better grades were achieved in each case. It should be noted that the grades and comparisons of results are subject to the special circumstances in which the exams took place: due to the general burdens of the pandemic, lecturers tended to avoid giving very difficult exams in order to avoid further stressing students. Shortened exam times was another factor here.
Individual responses were compared to verify whether the online format disadvantaged students in any way (DeSouza & Fleming, 2003). In order to make a proper comparison, the respective questions in the PP-LR and QLU-eQuiz exams had to be identical. For the exam question "Crepe," which fulfills this prerequisite, the following picture emerges:

Figure 1: Comparison for exam question "Crepe"
Source: Author's illustration In the figure above, the number of participants (n=) and average grades in % for the exam question "Crepe" of the last three ABWL exams are plotted. The average grade is calculated as the ratio of the points achieved by all participants to the maximum number of points available. As the figure clearly illustrates, the average grade from the QLU-eQuiz exam during the 2020/21 winter semester is at the same level as the two previous exams, which were written as PP-LR in a shortened version (SoSe 2020, exam duration 60 minutes) and under normal conditions (WiSe2019/20, exam duration 120 minutes).
The easily accessible and widely available data from online exams can thus be used in the context of "exam controlling" to investigate, for instance, the extent to which the form or presentation of a question in an online exam can potentially disadvantage exam participants.
Overall, examinees gave positive feedback on the suitability of the digital QLU-eQuiz exam format. Of the responses (n=384) to this question collected immediately after completing the QLU-eQuiz exam, nearly two-thirds of examinees rated the exam format as "suitable" or "very suitable."

Conclusions
Implementing the exams planned for the 2020/21 semester took place under enormous time pressure: the focus of the QLU-eQuiz-Project was therefore to provide a viable, functioning online exam system -and this goal was achieved.
The potential of the QLU-eQuiz project is not limited to conducting online examinations at home (OL-AH). In the context of a multi-phased approach, this approach can also be applied to a future teaching and learning concept. The first phase consists of offering practice exercises in digital form using Qualtrics scripts during the lecture period (Schmeeckle, 2003). The second phase consists of conducting a mock exam during the lecture period. Thanks to the largely automated grading process, both students and the instructor can receive timely feedback on course performance (Enders & Gaschler & Kubik, 2021). Based on this information, instructors can carry out more targeted interventions to enhance student learning, which may also help to alleviate test anxiety (Dickhäuser & Dickhäuser, 2020). The third phase consists of conducting the actual QLU-eQuiz exam, as described in detail here.
It should be noted that this exam format can be used in a variety of contexts beyond OL-AH. It could also, for instance, be used on site at universities, whether in campus computer labs (with Internet access), or as a "Bring Your Own Device" exam (BYOD exam) where students bring their laptops or tablets and complete the exams online but in physically present in a university lecture hall (with Wifi). This also forecloses the possibility of conducting "hybrid" exams, which are written in parallel at the university, e.g. as PP-LR or BYOD exams, and as OL-AH.
In the fourth and final phase, the instructor offers a "digital" exam review. The procedure is in principle similar to a QLU-eQuiz exam, but here the examinee sees their responses to the exam questions on the screen alongside the corresponding sample solutions. Ideally, the exam review takes place in the following semester in the context of a lecture during which the lecturer can immediately answer questions from exam participants. For the topic of a digital exam review, it seems to be true that supply creates demand. The following table shows the registrations for the exam review of the exams written in the 2020/21 winter semester and one sees that nearly half of the students (48.8 %) participating a QLU-eQuiz registered for the exam review. However, the experience of the examinations carried out so far also shows that there are sometimes high no-show rates here. Looking forward, the first hypothesis here for future research is that by applying the multi-phase QLU-eQuiz concept, it will be possible to significantly increase the average grade in the ABWL lecture exam (which has been steadily decreasing over the last years) without actually lowering the demands of the lecture or exam.
Given the promise of online exams to reduce the time and effort required for exam corrections with automatic grading (Becker & Weidt, 2017), a second research hypothesis can be formulated as such: using QLU-eQuiz exams should make it possible to significantly increase the efficiency of conducting exams compared to "classic" PP-LR. This profitability analysis refers to the stakeholders, including the organization (university), lecturers, and students, as well as environmental considerations (CO2 footprint).
Furthermore, we will also investigate how the evaluation of the practice questions (Phase 1) and mock exams (Phase 2) helps the instructor to formulate effective interventions during the current semester to improve student learning in advance of the final exam.
Due to the current pandemic situation (as of April 2021) and the expected development, it is foreseeable that a series of examinations in the QLU-eQuiz format will also take place at Ludwigshafen University in the summer semester of 2021. This second exam cycle will be used to make further progress on the "learning curve". This concerns on the one hand the workflow for the organization of the four phases of the QLU-eQuiz concept and on the other hand software improvements, e.g. the interception of input errors. Furthermore, it is considered to extend the field of application to other departments or universities and to offer exams as "Exam as a Service (EaaS)". In a first step, exams would be offered that are very similar to the exams described here and consist of multiple choice and calculation tasks. The conception of the QLU-eQuiz allows a location-independent use, since the conditions for participation consist only in the receipt of emails and a (as stable as possible) Internet connection. For the implementation of multilingual solutions different options can be considered. At present, the offer of multilingual (German/English) examinations is already being tested using the "Translate survey" feature from Qualtrics. With this feature it is possible to add multiple languages to a single survey and therefore participants can view the survey in their personal preferred language. Against the backdrop of increasing internationalization of science and research (BMBF, 2019), offering multilingual online surveys could make another contribution.