V ALUE -I NFORMED I NNOVATION : I NTEGRATING V ALUE -S ENSITIVE D ESIGN AND E VIDENCE -I NFORMED P RACTICE IN E DUCATION

Digital innovation in education – as in any other sector – is not only about developing and implementing novel ideas, but also about having these ideas effectively used as well as widely accepted and adopted, so that many students can benefit from innovations improving education. Effectiveness, transferability and scalability cannot be added afterwards; it must be integrated from the start in the design, development and implementation processes, as is proposed in the movement towards evid ence-informed practice (EIP). The impact an educational innovation has on the values of various stakeholders is often overlooked. Value Sensitive Design (VSD) is an approach to integrate values in technological design. In this paper we discuss how EIP and VSD may be combined into an integrated approach to digital innovation in education, which we call value - informed innovation . This approach not only considers educational effectiveness, but also incorporates the innovation’s impact on human values, its scalability and transferability to other contexts. We illustrate the integrated approach with an example case of an educational innovation involving digital peer feedback.


Introduction
Digital innovation in education -as in any other sector -is not only about developing and implementing novel ideas, but also about having these ideas effectively used as well as widely accepted and adopted so that many students can benefit from innovations improving education.However, especially in pressing circumstances such as a lockdown, there is a considerable risk of introducing ad hoc innovations that do not deliver the expected effect or, even if they do, are not scalable or transferable to other contexts.
Effectiveness, transferability and scalability cannot be added after an innovation has been designed, developed or implemented in educational practice.Instead, it must be integrated in the design, development and implementation processes and from the start be an integral part of innovation development.Evidence-informed practice (EIP) has the potential to facilitate this integration.EIP aims to achieve greater effectiveness as well as scalability and transferability of educational innovations by making use of three types of evidence when generating educational innovations: 1) evidence from scientific research, 2) evidence from practical expertise and experience, and 3) evidence from local (system) data (Brown & Malin, 2022).
Engagement of others during the design process is also promoted by Froyd et al. (2017), who argue that a propagation approach, encompassing the early engagement of stakeholders and potential future adopters, has more chance of successfully transferring effective educational innovations to other contexts than a dissemination approach.
A very relevant, but often overlooked, aspect in the adoption of educational innovations is the impact an innovation has on the personal values of the various stakeholders.Values in educational innovation have received increasing attention in society with the emergence of various examples of unrest generated by digital innovations.An example is the ongoing discussion about the use of online proctoring software (Appelman et al., 2021;Ebbinghaus & Bös, 2020;Harwell, 2020;Scienceguide, 2022;Singer & Krolik, 2021).The importance of values is subscribed by Cukurova et al. (2019) who propose to include the perspectives and values of users as a fourth type of evidence in educational innovation.However, in what way the value perspective can be incorporated in EIP is not explicitly discussed by Cukurova et al. (2019), and this is lacking from other EIP literature as well.
Value Sensitive Design (VSD) is an approach to integrate values in technological design.VSD originates from the fields of information systems design and humancomputer interaction (Friedman et al., 2006).It is characterized by engaging both direct and indirect stakeholders in assessing the impact a new design may have on their values, such as autonomy, trust, responsibility, safety or wellbeing, and what design choices will generate the most positive impact.Applying VSD to the topic of online proctoring software, for instance, generated 21 additional implementation criteria to the criteria that were derived from a functional investigation including test runs with the software (van Steenbergen & van der Spoel, 2021).
In this paper we discuss how EIP and VSD may be combined into a integrated approach to digital educational innovations, which we call value-informed innovation, that not only considers educational effectiveness, but also takes into account the innovation's impact on human values, its scalability and its transferability to other contexts.
In the next section we discuss the theoretical background of VSD and EIP separately.In Section 3 we combine the two by incorporating values in EIP, using the ADDIE model to structure activities taken from VSD.We illustrate the integrated approach in Section 4 with the example of an educational innovation involving peer feedback and conclude with discussion and conclusions in Section 5.

Value Sensitive Design
Value Sensitive Design (VSD) is "a theoretically grounded approach to technology design that takes human values into account in a principled and comprehensive way throughout the design process" (Friedman et al., 2006, p. 349).Human value is defined in VSD as "what is important for people in their lives, with a focus on ethics and morality" (Friedman & Hendry, 2019, p. 4).VSD goes beyond instrumental aspects such as functionality, reliability and ease of use, integrating potential impact on moral values of individuals, groups and societies in the design process.VSD distinguishes four types of stakeholders whose values must be taken into account: the sponsor of the new design, the project team doing the design, the envisioned users of the design, called direct stakeholders,  (Oosterlaken, 2015), browsers (Friedman et al., 2002), educational apps (van der Stappen & van Steenbergen, 2020) and social robots (Smakman et al., 2021).

Evidence-Informed Practice in Education
Inspired by a trend in the health and social work professions, the attention for using evidence to inform educational practice has attracted increasing attention in the last decade or so (Nelson & Campbell, 2017;Nevo & Slonim-Nevo, 2011).Moving away from a (deterministic) step-wise approach usually denoted by evidence-based practice, the term evidence-informed practice (EIP) has become common in the field to describe the utilization of knowledge (evidence) by educational professionals to improve their practice (Brown & Malin, 2022;Nevo & Slonim-Nevo, 2011).
According to Brown and Malin (2022, p. 2), EIP can be described as 'fostering situations in which teaching practice is deliberately informed by knowledge such as: (1) formal research; (2) evidence produced by practitioners' inquiries; and/or (3) evidence derived from schoolor system-level data (e.g., student assessment data)'.Another definition of EIP is posed by Nelson and Campbell (2017, p. 129): 'EIP must be seen as the integration of professional judgement, system-level data, classroom data and research evidence.'In both definitions, the same three types of evidence -on which we elaborate in the next section -are mentioned that should be incorporated in the improvement of teaching practice.
Educational professionals may have different goals to work in an evidence-informed way.On the most basic level, using evidence to inform the process of implementing an educational innovation will increase the chance of being successful because the decisions made will be sound and grounded in knowledge.If in addition a validated process model such as the ADDIE approach (Branch, 2009) is used, the probability of making the right decisions in the right order will increase further.On a higher level of ambition, working in an evidence-informed way will help to extend the knowledge base by allowing for and facilitation of the vertical spread of the innovation (scaling up) and/or horizontal spread of the innovation (transfer to different contexts).

Types of Evidence in EIP
As mentioned above, according to Brown and Malin (2022), three types of evidence should be utilized by educational professionals: 1) formal research, or scientific evidence, 2) practical expertise, or practice-based evidence, and 3) (school) system data, or local evidence.

Scientific evidence
Scientific

Practice-based evidence
Practice-based evidence has been developed through practice-based research, describes 'what works where and for whom' and indicates success factors for implementation and contextual barriers and facilitators.Ideally , this type of evidence is transferable and generic in order to be informative in a different context, and for the outcomes to be scaled up to support a broader field of application (Andriessen, 2016).Examples are co-design-based studies, (didactical) usability studies, prototyping, and good or best practice descriptions (Prinsen & van der Stappen, 2021).Practice-based evidence is sourced from implementation contexts other than the one in which the innovation is developed.

Local evidence
Local evidence is knowledge obtained by systematically analysing multiple existing data sources within the school to describe 'what happens in our school'.These data sources can be both quantitative and qualitative, e.g., student characteristics, achievement data, classroom observations and system log files of learning management systems (LMSs) (Brown et al., 2017).Applying this evidence to innovate educational practice as well as to evaluate such innovations is called databased decision making (Schildkamp & Kuiper, 2010).Local evidence is sourced from the specific implementation context in which the innovation is developed.

Propagation of Educational Innovations
EIP is the combined usage of local, practice-based and scientific evidence (Brown et al., 2017;Nelson & Campbell, 2017;Nevo & Slonim-Nevo, 2011).This combination of perspectives can be considered as 'triangulation of evidence' to improve practice and has the potential to contribute to the propagation of innovations.We consider the propagation of innovations as the stimulation and facilitation of both vertical spread (scalability) and horizontal spread (transferability) of improvements to teaching and learning (Brown & Malin, 2022).Froyd et al. (2017) argue that a propagation paradigm towards educational innovation is paramount to eventually achieve system-wide adoption of an innovation.This propagation paradigm has an equal emphasis on both fit and efficacy of the educational innovation being developed.Early stakeholder engagement including stakeholders from a diverse set of contexts is central to the approach, as is learning how the innovation should be implemented best through engaging with potential users and adopters.
To fit an innovation effectively to the implementation context -and to ultimately reach adoption of the innovation -it is crucial that the innovation in question does not harm the values of direct and indirect stakeholders related to that context.If negative consequences of the innovation have impact on (intended) users and adopters, the remedy might be worse than the ailment.Indeed, Davies (1999, p. 115) already stated more than 30 years ago that consideration of values is paramount: »Evidence is also required about ethical issues of educational [or care] practice, such as whether or not it is right or warrantable to undertake a particular educational activity [or health care intervention].»Since then, many perspectives and approaches towards the utilization of evidence in professional practice have been proposed, but ethical issues have largely been underexposed.Recently, Cukurova et al. (2019) emphasized the use of four sources of information, in which the first three types of evidence are the ones mentioned above, whereas the fourth type of evidence is »the perspectives and values of those people who are directly or indirectly affected« (Cukurova et al., 2019, p. 5).What is currently missing however, are insights into how to incorporate values of direct and indirect stakeholders into the implementation of educational innovations.

Relation to other approaches combining research and practice
Several approaches exist that also combine research and practice and share similar underlying principles with EIP, such as Design Science Research (Hevner et al., 2004), Action Research (Babüroglu & Ravn, 1992;Baskerville & Wood-Harper, 1998) and Action Design Research (Sein et al., 2011).However, these research approaches all combine rigor and relevance towards the goal of theory advancement by e.g., incorporating practice and real-world usage into the research process.EIP, and our proposed approach, have the aim of incorporating existing knowledge into the work process of (educational) professionals, but not the goal of generating new knowledge.

Incorporating Values in Evidence-Informed Practice
A widely adopted systematic approach to instructional design and educational innovation is ADDIE (Branch, 2009), which comprises five phases from where the acronym originates: Analyze, Design, Develop, Implement and Evaluate.In all phases of this methodic approach, we can utilize the different types of evidence as described in Section 2.3.In Figure 1, we illustrate how we can incoporate practicebased evidence sourced from other implementation contexts (left), local evidence sourced from the specific implementation context in which we are running through the five ADDIE phases (center), and scientific evidence sourced from theory and literature (right).
Since the ADDIE approach was introduced, more agile and iterative approaches have been adopted in many professional domains concerned with design -starting from software engineering -as well as in education.It is possible to incorporate such short-cycled iterations within the ADDIE approach, as is depicted by the smaller cycle positioned behind the three phases Design, Develop and Implement.
The process starts with Analyze, subsequently running through Design, Develop and Implement in several iteration cycles, and rounding off with Evaluate.Partial, focused evaluations will be conducted within each of these iterations to guide the planning of the next iteration.In the final Evaluation phase, the innovation process is evaluated in an integral manner.VSD adds the perspective of moral impact to the aims of effectiveness, transferability and scalability.As VSD is strongly based on stakeholder engagement, it also aligns well with the propagation paradigm as proposed by Froyd et al. (2017).
It enriches conversations with practitioners by introducing the topic of personal values and widens the scope of conversational partners by explicitly including persons or groups that do not directly engage with the educational innovation but are nevertheless impacted by it (e.g., housemates of students when using online proctoring software).In Figure 2, we illustrate which VSD-activities can be undertaken to elicit value-based evidence from all three sources of evidence: other implementation contexts, the specific implementation context in question, as well as from literature.

Worked Example of the Integrated Approach
To illustrate how the integration of VSD in EIP might work, we walk through an hypothetical example process of introducing digitally supported peer feedback.This example case is one where for a specific course, a team of educators consider the introduction of peer feedback.The reasons for considering peer feedback are that the teachers are experiencing a high workload, many students are failing the course and students have consistently been asking for more and more timely feedback on their work.The hope is that allowing students to provide feedback on each other's work digitally will have a positive impact on all of these issues simultaneously.impact (VSD).To assess moral impact, an inventory is made of persons or groups that might be impacted by the introduction of peer feedback and what character that impact might have (e.g., positive impact on wellbeing of teachers because of less workload and on confidence of students because of additional learning opportunities, but maybe negative impact on assurance or privacy of students).In the Design phase, besides studying existing configurations and effectiveness of alternative configurations (EIP), stakeholders (teachers and students, but also examiners) are actively engaged and asked about how they experience alternative configurations (VSD), using techniques such as interviews, focus groups or experiments.In the Develop, Implement and Evaluate phases, evidence from data, practice and literature are used to assess alternatives and achieve not only an effective, transferable and scalable solution (EIP), but also a solution that respects human values (VSD).

Conclusion and Discussion
In this paper, we propose to integrate value-sensitive design with evidence-informed practice as a way to not only ensure effectiveness, transferability and scalability of educational innovations, but also a positive moral impact.Value-informed innovation extends evidence-informed practice with methods and techniques to actively take human values into account in the design process of educational innovations.Aligning well with a propagation approach, value-informed innovation extends the types of stakeholders to involve with relevant non-users and future users and it extends the topics to be discussed with human values.Value-based evidence originates both from academic science (conceptual investigation) and practice (empirical and technical investigation).It appears that, though values are rightly considered a fourth type of evidence, value-based evidence is of a different order than the three others.Unlike scientific, practice-based and local evidence, value-based evidence can be sourced from all evidence sources: it can be derived from scientific literature, by elicitation from stakeholders in other contexts, and derived from local data and experience.Moreover, value-based evidence can (and should) be gathered from contexts in which implementation is not intended (yet) by incorporating the time and pervasiveness perspectives.
Our work is a conceptual and theoretical contribution on the integration of EIP and VSD to incorporate human values in the innovation process in educational practice.
For future work, it would be interesting to test this proposed approach in real-life innovation projects, with the aim of concretizing the approach, to develop practical insights and guidelines and to evaluate and improve the ideas presented in this paper.Finally, we expect VSD can be integrated in a similar way in professional domains other than education with approaches that resemble EIP.

Figure 1 :
Figure 1: Utilizing different types of evidence in EIP within the ADDIE approach

Figure 2 :
Figure 2: Combining VSD with EIP At a conceptual level, the relevant stakeholders and values are identified and defined based on existing literature and knowledge.At an empirical level, the perception of these values by the different types of stakeholders is studied by means of methods such as interviews, focus groups or experiments, leading to the elaboration of the values into norms.At a technical level, values and norms are translated into technical design.VSD has been applied to a variety of technological designs, such as wind parks

Table 1 : Examples of the utilization of various types of evidence in the process of implementing an educational innovation
In Table1, we illustrate for this example case how the four types of evidence are iteratively collected and VSD is integrated in the process.During the Analyze phase, the envisioned problem is analysed by using local data, experience of practitioners with peer feedback and academic literature on peer feedback, not only to gain insight on workload, feedback quality and student performance (EIP), but also on moral Cukurova et al. (2019) in EIP operationalizes whatCukurova et al. (2019)propose as a fourth type of evidence, besides scientific evidence ('what works why'), practicebased evidence ('what works where and for whom') and local evidence ('what happens in our school'): the perspectives and values of those people who are directly or indirectly affected.This, what we call value-based evidence, describes 'what is considered valuable by whom', relating implementations to human values of direct and indirect stakeholders.35 TH BLED ECONFERENCE DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION