What makes IS Implementation Successful? A Study on Implementation Effectiveness

It has been noted that implementation climate is positively associated with implementation effectiveness. However, the recipe for a successful implementation of IS/IT systems still doesn’t exist. Specifically, it is unclear what a "good" implementation climate requires, what it should be, and to what extent the acceptance and success of the implementation of a new IS/IT system is affected. Despite success and opportunities for organizations that innovate with information systems (IS) and information technology (IT) in general there are also many failures of IS/IT implementations caused by both technical and non-technical problems. This study, based on the Klein-Sorra model of implementation effectiveness, shows that skills and innovation-values fit do significantly influence intention to use in the context of our questionnaire-based survey, the implementation of a new document management system (DMS) at the Dutch Police. Survey data was collected from 41 end-users. For practitioners, this research offers practices to be considered during implementation of a new system. 364 33RD BLED ECONFERENCE ENABLING TECHNOLOGY FOR A SUSTAINABLE SOCIETY


Introduction
While the topic of information systems (IS) implementation success has been the focal point of considerable research, the literature varies regarding how to study an implementation project and what variables determine its success or failure (Larsen, 2003). Despite success and opportunities for organizations that innovate with IS and information technology (IT) (Markus & Loebbecke 2013;Van Veldhoven & Vanthienen 2019) there are also many failures of IS/IT implementations caused by both technical and non-technical problems (Mclean, Antony & Dahlgaard 2017;Pan 2005). Reasons for a successful or failed IS implementation are complex and contested, as different stakeholders and perspectives are involved (Dwivedi et al. 2015). There is relatively little literature available on how these factors can now be governed during the implementation process and how in that light factors interrelate (Jacobs, Weiner & Bunger 2014;Muntslag 2001).
Police organizations have a long history in which they embraced new technologies to improve the efficiency and effectiveness (Koper, Lum & Willis 2014) and it is still an important driver to innovate and improve (Byrne & Marx 2011). However, in essence within the Dutch police the work is not changed very much over the past ten years. Technological developments have shown no changes in existing routines, processes and concepts (Terpstra et al. 2013). New technological innovations have been developed to prevent crime and to improve the performance of the police, but we know remarkably little about how and why (or not) certain innovations are adopted (Byrne & Hummer 2017). The main objective of this study is to offer new dimensions in research into possible interventions in IS-implementation and thereby provide insights into factors that can be managed to improve implementation effectiveness in digitization projects.
In the next section the concepts of this research are discussed and operationalized.
After that, the research methodology is described followed by the results of this study. Finally, the conclusions and recommendations for further research are provided.

Theoretical Foundation
Implementation is a critical gateway between the decision to adopt innovation and the routine use of the innovation within an organization (Klein & Sorra 1996). For implementation to be a success, the application should be no longer perceived as something new, and the "targeted employees use a given innovation consistently and well" (Klein & Sorra 1996). Markus & Mao (2004) define system implementation success as a high-quality process of preparing the target user community for use of the system (often called "change management") and/or a high quality "change" outcome, namely that the intended users (regardless of whether they participated in development) adopt the system, use it as expected, and/or use it with the desired effects. Change creates a sense of uncertainty and lost control, and employees' resistance and lack of support are some of the most cited causes for failures associated with organizational change (Blut, Wang & Schoefer 2016). This resistance represents a major barrier to changing the behaviors of organizational members to use the innovation and for the organization to reap its benefits (Hwang, Chung, Shin, & Lee 2017). To solve this issue an area of IS research consists of theories and models that are oriented towards the acceptance and use of IS (cf. Venkatesh, Morris, Davis & Davis 2003). These models provide an important theoretical foundation for studying how various users, and technological and environmental influences, can predict, explain and determine the use of IS. From these studies it is also clear that IS research still needs to better address and identify organizational mechanisms and means through which management can influence users' beliefs and attitudes towards adopting new information systems, hence reducing the possibility of failure (Venkatesh et al. 2003).

The Klein-Sorra-model
A predictive model which pays close attention to the specific factors that have an influence on the effectiveness of the implementation is the Klein-Sorra model (Klein & Sorra 1996). Implementation is "the process of gaining targeted organizational members' appropriate and committed use of an innovation'' (Klein & Sorra 1996, p. 1055. They define implementation effectiveness as "quality and consistency of targeted organizational members' use of an adopted innovation" (p. 1056), and is determined by implementation climate and innovation-values fit. Implementation climate is described as "employees' shared perceptions of the events, practices, and behaviors that are rewarded, supported, and expected in a setting" (p.1060). It is formed by (a) users' skills to use the system, (b) incentives for using the system and disincentives for avoiding system use, and (c) removing obstacles to system use. ENABLING TECHNOLOGY FOR A SUSTAINABLE SOCIETY Innovation-values fit is described as "the extent to which targeted users perceive that use of the innovation will foster the fulfilment of their values" (p.1063).

IS System success
Regardless of whether the economy is booming or busting, organizations want to ensure that their investments in information systems (IS) are successful (DeLone & McLean 2016). As information systems have become more complex, so has the evaluation of the effectiveness or success of the system (Petter, DeLone & McLean 2012). From an IS perspective, acceptance and system use have been the variables of choice for measuring success (Delone & McLean 2003, 1992 Seddon et al. 1999). In fact, both the original version of D&M of 1992 and its extension in 2003, appear among the most cited articles of the discipline in the past decade (Stein & Galliers 2014). In our study System implementation success (i.e., one of the dependent variables) was measured using the concepts of intention to use and user satisfaction. Some papers have shown that the changes of the nature of technology could influence the interaction between technology and its users, thus changing the model that represents that relationship (Mardiana, Tjakraatmadja & Aprianingsih 2015). Because actual use was mandatory and this study was on the deployment of a new Digital Police Dossier (DPD) system, actual use was not included as dependent variable. To many other practical factors such as technical implementation problems and other delays made this measurement less valid. Also the variable Net Impact is not included in the conceptual model. As this was a study of the deployment of a new DPD system during the time of data collection, the measurement of actual Net Benefits as such becomes an irrelevant factor. The quality variables were not included because the focus of this research was on the evaluation of the deployment of a specific application (the DPD system).

Attitude toward the system
When an organization mandates use, attitudes will likely take on heightened importance and thus warrant consideration (Brown et al. 2002). Mandatory use in the context of our study is that the decision to proceed with the development and use has been made by the management. In mandated environments attitudes might not align with actual behavior, that is, an employee might hold a negative attitude toward adopting the new system, but will ultimately use the system because he/she has to and no other options exist (Hwang, Al-Arabiat & Shin 2016). Users who do not wholeheartedly accept the innovation can delay or obstruct the implementation, and resent, underutilized or sabotage the new system (Markus & Mao 2004). Brown et al. (2002) believe these reactions are a result of the positive or negative attitudes employees form toward the technology. Attitude has been extensively studied as a predictor to Intention to Use, Use, and to lesser extents, User Satisfaction and Individual Impact (Petter et al. 2013) These findings are consistent with well-known past reviews of the literature that found moderate support for the relationship between attitudes toward technology and overall IS success, with most studies supporting this relationship (Petter et al. 2013

Conceptual model
Now that we have explained the variables of implementation climate and IS system success, and clarified the importance of attitude towards the system, the integrative conceptual research model of this study is illustrated in Figure 1. The model demonstrates that skills, absence of obstacles, incentives and innovation value fit can be considered as independent variables, while intention to use, user satisfaction and attitude towards the system can be considered as a dependent variable. As such the following hypotheses were postulated and tested.
Skills De Waal & Batenburg (2012a) appoint that studies show a positive effect with respect to users training and implementation success, at the same time they appoint that surveys show that the relationship with users training and implementation success is far more complex than the results suggest. Providing training influences System Usage because it contributes to trusting the new system and contributes to User Satisfaction (Guimaraes, Staples & Mckeen 2003). Knowing users' attitude towards computers and innovations might be a key to successful implementations as it would help recognize how users feel about the new system (Hwang et al. 2017). Building on these explanations the following hypotheses were tested: H1: Skills is positive associated to Intention to Use H2: Skills is positive associated to User Satisfaction H3: Skills is positive associated to Attitude Absence of Obstacles. Venkatesh & Brown (2001) have uncovered that offering resources and necessary technical support reduces knowledge and resource barriers.
Other researchers have found that users can also be motivated through the use of rewards and the provision of training (Igbaria, Guimaraes & Davis 1995). The value of technology appears to differ depending on the tasks of the user and whether the system is perceived to assist or hinder them in the performance of their tasks (De Waal & Batenburg 2012b). Hence: H4: Absence of Obstacles is positive associated to Intention to Use H5: Absence of Obstacles positive associated to User Satisfaction H6: Absence of Obstacles is positive associated to Attitude Incentives. Klein & Sorra (1996) use the shorthand phrase 'implementation policies and practices' to refer to the array of strategies that organizations put into place to promote innovation use. Engaging expected users and supporters in decision making about innovation design and implementation, providing incentives for innovation use, and providing feedback on innovation use, all enhance motivation. Also, organizations should make the innovation easily accessible or easy to use, give expected users time to learn how to use the innovation, and redesigning work processes to fit innovation use (De Waal & Batenburg 2014). Based on these findings: H7: Incentives is positive associated to Intention to Use H8: Incentives is positive associated to User Satisfaction H9: Incentives is positive associated to Attitude

Innovation Value Fit.
When an individual employee experiences innovation-values fit, they will be drawn to internalize and embrace the system (Klein & Sorra 1996) A study on ERP systems by (Osei-Bryson, Dong & Ngwenyama 2008) suggest that high innovation-values fit influences users to obtain better skills, perceive less obstacles, and feel more motivated in using the new system. A possible explanation is that if users perceive that the new system will help them solve their work related problems; they internalize the benefits of the new system and consequently, they are more open to ENABLING TECHNOLOGY FOR A SUSTAINABLE SOCIETY learning and mastering the system, thus becoming intrinsically motivated (Osei-Bryson et al. 2008). Some studies have found that when system usage is not intrinsically driven, it suffers underutilization or users' intentional sabotage (Markus & Keil 1994). When 'higher level of intrinsic motivation typically leads to willingness to more time on the task' (Venkatesh 2000), it is more likely that users experience a higher values fit, they are more skilled and motivated to use the new IS. Therefore, the following hypotheses were tested: H10: Innovation Value Fit is positive associated to Intention to Use H11: Innovation Value Fit is positive associated to User Satisfaction H12: Innovation Value Fit is positive associated to Attitude

Data collection
We applied a survey to measure all elements of our conceptual model to gain an understanding of the implementation context and practice of the questionnairebased survey. The survey was carried out during the implementation of the DPDsystem in May -June 2019. Participation in the study was anonymous and therefore data was anonymized so it is not traceable to the individual participant.

End-user survey
The survey was conducted using a web-based tool provided by the Police Academy. All 90 employees who finished their training and joined the pilots were personally asked to take part of the survey. In total 41 respondents completed the questionnaire indicating a response rate of 45%. The participants were located in five units: South-West, West, East, South-East and Midlands. Their function categories were reporter, file-owner or another function, such as implementation supervisor or command duty officer. Most respondents, 54% work in unit South-West. In this unit 47% worked as reporter, 41% as file-owner and 12% have another function. In the unit West the percentage is divided exactly between reporter (50%) and file owner (50%).
In unit East and South-East respondents only work in the function 'other'. In unit Midlands 13% of the respondents work as file owner, 6% as reporter and 81% had another function.

Instrument validation
The questionnaire was designed to measure all elements of the conceptual model. The purpose of the questionnaire was pre-tested for clarity and comprehension by using a small test panel of 5 implementation managers, and 1 communication officer.
Comments and corrections were discussed if necessary and incorporated. The items in the questionnaire include scales that are proved to be reliable and valid in previous studies. The items however were adapted to the context of the organization so that they were suitable for testing the hypotheses from the conceptual model. The four constructs Skills, Absence of Obstacles, Incentives, and Innovation-Values Fit, were measured by 26 items from a previous study by Osei-Bryson, Dong, & Ngwenyama (2008). The 'intention to use' construct was measured by one item only: 'If the system was not mandatory, I would still use it' as suggested by Seddon & Kiew (1996). To measure user satisfaction, the information satisfaction and service satisfaction items developed for the different constructs by Shaw, DeLone, & Niederman (2002) were used. Attitude toward the system is measured with four items of the survey instrument developed by . All items of the questionnaire could be answered on a 5-point rating Likert scale (1= fully disagree, 5= fully agree). In order to validate the measurement of the constructs, factor analysis was performed to analyze the construct validity of the items. For all constructs, principal component analysis (PCA) with Varimax rotation and Kaizer normalization was used. The results are presented in Table 1. As can be seen in Table 1, the own values were between 5.2 and 1.1, accounting for 91% to 65% of the explained variance. The reliability of the scales was confirmed by Cronbach's alpha value of 0.887 to 0.820 (cf. Nunnally & Bernstien 1994).

Results
In this and the next section we will describe the results of the user survey and the interviews. The main results of the survey are represented in Figure 2 and 3. The regression analysis provided the standardized path coefficients, p-values, and variance explained. The results from Figure 2 show that: − Skills holds a significant correlation (r = .69; p < .,01; N = 41), with intention to use, which confirms hypotheses 1. − Skills holds a significant correlation with user satisfaction (r = .43; p < .05; N = 41), which conforms hypotheses 2. − No significant correlations were found between skills and attitude. − Absence of Obstacles holds a significant correlation with Intention to use (r = -0.53; p < .001; N = 41), which conforms hypotheses 4. An explanation for the negative value is that a high value of the variable, thus experiencing a low level of Absence of Obstacles by the user, occurs with a high-value of Intention to use. − Incentives holds a significant correlation with attitude (r = .39; p < 0.,05; N = 41) and incentives holds a significant correlation with intention to use (r = 0.41; p < .01; N = 41), which conforms hypotheses 9. − Innovation value fit holds a significant correlation with user satisfaction (r = .51; p < 0.01), which conforms hypotheses 11. − Innovation value fit holds a significant correlation with attitude towards the system (r = .40; p < .05; N = 41) and also innovation value fit holds a significant correlation with intention to use (r = .72; p < .01; N = 41), which conforms hypotheses 12. Three multiple regression analyses (method Stepwise) were conducted to examine to what extent the influence of the independent factors was on one dependent factor. Figure 3 shows the relationship between the independent variables Skills, Absence of Obstacles, Incentives and Innovation Value Fit and the three dependent variables of Implementation success. The significant (standardized) regression (beta) coefficients are represented by the one-way-directed arrows in the figure. As the OLS regression model was applied, the potential problem of multicollinearity was investigated by computing VIF factors for each predictor in the regression model. Although in some cases correlations between independent variables were relatively high, VIF factors in none of the models exceeded 5 -a commonly applied rule of thumb (Hair et al. 1998). The results from Figure 3 show that:

Discussion and Conclusion
This study aimed to offer insights into factors that can be managed to improve implementation effectiveness in digitization projects. Based on a literature study a conceptual model was developed from which 12 hypotheses were derived. However, not all of the hypothesized relationships were confirmed. The study revealed that Innovation Value Fit does have a strongest relationship with Intention to Use and User Satisfaction and Attitude towards the System. The main contribution of this study is that these findings are used to complement the implementation plan for the upcoming releases. The results do offer insights that can be managed to improve implementations effectiveness in digitization projects at the Dutch police, which ultimately drive success.
Although the research was designed carefully, there are some limitations to this study. First of all, this research is limited one case organisation. The question is whether there is an organization comparable to the Dutch police. Second, the sample size is relatively small. This had to do with the progress of the program in which delay arose (cannot remove Absence of Obstacles) which meant postponement for further implementation of the new DPD. Another limitation is that the study is a specific implementation of DPD. IS implementation research show that IS implementation studies are context-sensitive, making it a topic of concern when it comes to the generalizability of results. A suggestion for further research is a study which consists of multiple cases and could provide additional insight in the causal relationships between the variables of the conceptual model. Furthermore, research could be executed by extending the conceptual model with additional critical success factors, such as management factors, user characteristics, the degree of user participation, etc. The last suggestion for further research relates to the interpretation and use of the results of the study. Further research could answer the question on how the results of this study can be brought into practice and what impact it will have.