Transformation of Cognitive Assistance Systems into Augmented Reality

: The field of Augmented Reality (AR) has received increasing attention in recent years, as AR can be applied to a wide range of problems. T he use of AR offers great potential, especially in the industrial sector. Among many applications in this area, one promising application is the augmentation of cognitive assistance systems. Much research has already been done on the development of augmented support systems but it is still lacking on how to transform existing assistance systems into an AR application. This paper focuses on this transformation and presents a process model that intends to support the migration or digitalization of existing support systems into an AR application. An experimental study validates the proposed approach and derives recommendations for action.


INTRODUCTION
The support of employees has become increasingly important in production operations in recent years. Due to the increasing number of possible product variants, the complexity in production has steadily increased [1]. This increasing complexity represents an additional challenge for those employees who have to assemble this range of variants [2]. In order to handle this challenge, different assistance systems are used, which support the worker during the assembly process [3].
Due to the digitalization of industrial processes, such assistance systems have become more and more relevant in recent years and have thus become an often discussed research topic in the context of the fourth industrial revolution and its challenges [3]. The topic of worker assistance is also becoming increasingly important due to the demographic change and the resulting ageing society in industrial nations [4]. As one category of assistance systems, cognitive assistance systems focus on supporting the employee on the perception and decision level and thus do not aim on relieving the physical strain on a worker [5].
With the ongoing improvements of technologies and the accelerated digitalization of industrial environments, new opportunities for companies to support their employees cognitively are constantly emerging. One of these possibilities is the use of Augmented Reality for information preparation, which has already been validated in numerous studies [6,7,8]. However, if companies decide to introduce an augmented assistance system in their production environment, they have to overcome the challenge of transformation. In many cases, companies already have work instructions that guide employees through the production. However, these instructions are often paper-based or, at best, digitally work processes that are displayed on screens. The migration and digitalization of the existing work instructions into an AR system is one of the challenges that must be overcome.
Research still lacks of approaches on how to transform an assistance system which consists of no or basic technology into an AR based assistance system. This paper aims to fill this gap and to present a process model on how to port an existing system into an AR system. An experimental study will validate and substantiate the proposed approach.
The structure of this work is as follows: First, existing literature and related research topics are reviewed. The focus is placed on assistance systems and their categorization, augmented reality for information support and the differences on migration and digitalization of assistance systems. Chapter 3 then presents and discusses the proposed process model for transforming a cognitive assistance system. In Chapter 4, research environment and the concept for the experimental study are presented, before discussing the implementation of the system using the proposed process model.

RELATED RESEARCH
This section focuses on summarizing the basics of and existing literature in this research field. First, the categorization of assistance systems is presented. Then related research on augmented reality in production processes is presented and principles for migrating and digitizing systems are covered.

Assistance Systems
Assistance systems support people preventively and operatively, adapted to their individual needs and specific activities [9]. This represents a promising approach in order to be able to fulfil job requirements for a longer time period. Additionally, they also enable the participation of aging or physically restricted persons in all areas of life [10]. In terms of digitalization, they represent a design principle, which is characterized by the term Industry 4.0 [11]. By using different technologies and adapting them to the individual employee, they contribute a significant added value to digitalization.
In order to understand the variety of assistance systems, a categorization has to be used that distinguishes different systems which assist the worker during the production process. In [12], Sanders presents the basic principles of the information handling process which is followed by the worker when fulfilling a certain task. He categorizes this process into four main steps: • Discover: Acquiring information from the environment using the human senses, • Recognize: Receiving the information and perceiving the contents, • Decision: Using the perceived information to make decisions, • Act: Executing the decision made by motor coordination.
Stockinger [13] uses the four main steps of a worker's information handling process as basis for the categorization of assistance systems. By assigning them to one of the steps through identification of their primary purpose, they can also be assigned to one of the types presented from Reinhart [14]. According to Reinhart, three types of assistance systems can be identified and used for classification: perceptual assistance, decision assistance and physical assistance. While physical assistance aims at reducing the physical strain on an employee, perception and decision assistance is concerned with the cognitive support of employees [15]. This is also referred to as a cognitive assistance system. Fig. 1 gives a visual overview of the information handling process and the classification of assistance systems.

Figure 1
The classification of assistance systems according to Stockinger [13], based on the information handling process of Sanders [12] and the classification of Reinhart [14] This paper focuses on cognitive assistance systems which are in terms of worker support also called worker guidance systems. Worker guidance systems are defined as cognitive assistance systems that are integrated into the work environment and provide work-related information [16]. This information is provided in a context-related way and, if possible, without additional effort for the employees. Such systems are used in production for various tasks such as assembly of parts or maintenance of machinery. The equipment used to guide the worker through his tasks can be different. Starting from paper-based instructions up to digital instructions on the screen or instructions embedded in the environment, using AR [17].

Augmented Reality for Worker Guidance
Due to the wide range of possible applications of AR in industrial environments, augmented reality has received growing attention in the research community in the recent years.
One of the possible applications which are researched in industry is the use of AR in the guidance of workers through their work tasks, by embedding the instructions together with virtual objects into the field of sight of the employee [18]. The advantage of AR compared to other forms of work instructions is that it makes it easier for the employee to obtain real time targeted situation-specific information which is directly projected into the working area of the employee, allowing them to see the desired information in the correct place of the workstation [19]. Various studies have already proven an improvement of effectiveness or quality when using AR assistance systems compared to traditional worker guidance systems [6,7,8].
When integrating an AR worker guidance system, various technologies can be used. The most common technologies are head mounted displays (HMD), in-situ projection, and handheld devices [20]. Typically, each technology has its strengths and weaknesses which need to be examined when deciding on the best fit for a desired application. HMDs (e.g. Google Glass or Microsoft HoloLens) offer hands-free work and enable good computational performance, but they are also often heavy and not usable over a longer period of time. The in-situ projection has the advantage that no device has to be worn by the worker and the computational performance is open for practically no limit. However, such projections are often hard to setup and can only be used at a specific location. Handheld devices, such as smartphones and tablets, get rid of the disadvantages of the technologies above, but are restricted in terms of hands-free operation.
To evaluate AR applications, studies have distinguished three types of measurements: effectiveness evaluation, usability evaluation and quality evaluation [21]. For these types of evaluation there is a huge diversity in the used methods to assess an AR application and thus various factors that influence the evaluation of a system are still unknown [22]. This leads to different results in studies, depending on the level of complexity of the assembled product, the user interaction methods and the rendering of the instructions that are evaluated. Although various methods on evaluating an AR guidance system exist, the results of the studies cannot be taken as a baseline for comparison. Moreover, when integrating an AR guidance system, the evaluation has to be done individually and cannot be directly assumed by the results of previous studies. By comparing these studies, one can only select the best suited evaluation methods in terms of used hardware and field of application. After selection of the desired methods, the evaluation can be done by comparing the previous worker guidance system with the new system. Evaluation methods for AR systems can be found in various studies such as [22,23].

System Transformation
Migration of systems has been a major challenge since the early steps of software development [24]. The task of developing systems in such a way that they are easily extendable, maintainable and changeable is already a focus of various software development methodologies and there is also a separate area of software maintenance that deals with this topic [25,26]. By definition, migration refers to the transfer of a software system to another target environment. Migrations are purely technical transformations with a clear definition of requirements [27]. Various workflow processes are known for migrating a technical system which is based on software. However, in industry it often occurs that a system which is currently using basic or non-technical solutions should be digitized. This challenge is discussed in various research fields concerning the digitalization process such as [28,29], but the approach of the projects strongly depends on the object to be digitized and can thus not be generalized. To the knowledge of the authors, there is no such model for digitizing non-technical work instructions into AR. There are several reasons why an existing system should be transformed. The main drivers to migrate or digitize a system are to increase flexibility, efficiency, effectiveness and thus to gain a cost advantage against competitors [30]. Concerning software systems, the need for migration can arise due to the replacement or update of hardware, system software (e.g. operating systems), architecture (e.g. change of programming design for better feature integration or change of programming languages), or because of better features in new systems [27]. Both the digitalization of a process and the process for migrating software are closely related, as presented in the following sections. Therefore, a typical software migration process, as well as a process for digitalization, show similarities in their process models.

Figure 2
Workflow for the migration of a software system as presented in [27]. Rows indicate the iterative workflow while the columns are different phases during the migration project with different intensities.

Workflow of Software Migration
Reference process models for software migration enable the planned, documented and controlled execution of migration projects [27]. The provision of reference processes is often based on best-practices and enables the targeted development, adaptation and optimization of individual migration processes [31]. Various process models for software migration and their application in practice are presented in [32][33][34]. These process models follow a common migration workflow which is executed iteratively and with different intensity in the different phases: conceptualization, design, construction and transition. Fig. 2 describes the iterative workflow and the different phases of a typical migration project. The first step is to define the migration strategy, which depends on the design and the quality of the current system. Then, the target environment is to be defined, focusing on hardware, used software, integrated development environment and architecture. Afterwards, the current system is evaluated and differences to the new system are to be set. After setting the scope of the migration project, the desired transformations are specified and implemented. Depending on the selected migration strategy, the handover of the system takes place at once or incrementally. Quality assurance measures for migration ensure that the functional equivalence of the old and target systems is guaranteed throughout the migration process. A more detailed description of the workflow process can be found in [27].

Digitalization Processes
When a cognitive assistance system is not migratable because the current solution has no sufficient technical basis, like for paper-based instructions, the workflow as described in [27] might not be applicable. In this case, the digitalization of the assistance system can be conducted.
When it comes to digitalization of processes or products to solve existing problems or to extend the functionality, one also refers to the term digital transformation [35]. To digitalize a process, the availability of data has to be assured. If non-technical systems are to be digitalized, the analog data first has to be digitized. As a result, a digital transformation for an analog product consists of two different parts: the digitization of the analog data and the digitalization of the product [35][36][37]. To digitize analog data, a process is proposed by Note [37] which focuses on defining the target environment, selecting and preparing the material for digitization, the actual digitization process, and the delivery of the created collection via a system. During this process, quality assurance is conducted. For the digitalization of the product, a strategy for the digital transformation has to be derived first [38]. The development of the product then follows a typical product development process as described by Cooper in [39]. Cooper describes an improved process model for product development and names seven stages: idea, preliminary assessment, concept, development, testing, trial and launch. In the idea stage, the technological possibilities have to be identified and the idea has to be formulated. In the preliminary assessment, the feasibility and attractiveness of the product are evaluated before a concept is developed in stage III and developed (stage IV). Both testing and trial focus on quality assurance and evaluate the functionality of the product. In the last stage (stage VII), the product is launched and used. [39] gives a more detailed description of the product development process.

CONCEPT DESIGN
When it comes to the transformation of cognitive assistance systems into a new technology, such as AR, a process model for the development of the product needs to be prepared. As seen in the previous section, the preferred process model to use depends on the assistance system in use. If no or non-technical worker guidance systems are currently used in production, a digitalization process has to be followed, whereas a migration workflow model could be used to for existing technical solutions. Due to the similarity of both process models presented above, this section presents a unified process model for the transformation of cognitive assistance systems. Furthermore, the focus is placed on the transformation of a worker guidance system into an augmented reality application which supports the worker during his task completion.

Transformation Process Model for Cognitive Assistance Systems
In this section, a process model for the transformation of cognitive assistance systems into AR is proposed. Therefore, the process models for migrating a technical system [27] and the digitalization process which follows the product development process [39] are combined into one unified process model which ensures a transformability which is independent of the technical characteristics of the existing solution.

Figure 3
The proposed process model for transforming a cognitive assistance system into AR Fig. 3 shows the proposed process model for this task. In the beginning, the strategy for the transformation of the cognitive assistance system has to be specified together with the definition of the environment in which the application should be transformed. For the development of a suitable process model, the design fields and parameters of the old and new cognitive assistance system have to be determined. Then, following the requirements for industrial AR applications and analyzing the existing AR technologies, a suitable medium can be selected. In the concept phase, the application is planned based on the preselected parameters before it is developed. At the end of the development phase, a finalization phase is included; it tests the application before launching it in production. The workers who are going to use the worker guidance system should be involved during the specification of the parameters, the concept design, as well as for the final testing before the launch (steps 2-4,6,7).
Besides that, quality assurance should be conducted in all phases to ensure an optimal outcome. In the next sections, the phases of the process model are explained in more detail.

Strategy and Environment Assessment
The first step is based on the first two phases of the migration workflow, as well as the strategy derivation for the digitalization process. Hence, the desired quality of the new assistance system has to be determined, the budget has to be planned and the decision regarding the transformation mode has to be made: In this phase, it has to be decided how deep the worker who uses the new system should be involved. In principal, workers should be included in phases 2-4 and 6-7 to ensure an optimal outcome. However, the intensity on how much they should be integrated is to be specified firstly. The second part of this phase is to assess the target environment. Therefore, an ergonomic assessment of the work environment should be conducted to identify possible influences on an AR worker guidance system [40]. Possible influences are lighting of the workplace, glare, interactions with other workers/vehicles in the environment and possible safety risks arising from the use of AR devices (for example, due to the restricted field of vision).

Specification of Design Parameters
The focus in the development of the worker guidance system in this paper is on assembly and therefore assemblyrelated information, such as parts lists and assembly instructions, and is placed at the center of the consideration. According to Reinhart [14] there are three design fields for worker guidance systems that can be adapted when transforming a cognitive assistance system: Information degree, information design and information device. Information degree focuses on the amount of information which is to be displayed by the system. This design field can be adapted with different parameters such as the definition of the amount of information to be displayed or what type of information to show. Information design focuses on how to present the information to the worker and how it should be displayed. This field focuses on parameters such as the perception of information and the style, mode and intensity of information. The third design field focuses on the information device which is used to assist the worker in their assembly task. This field consists of technical parameters such as the hardware to be used, the communication protocols and the hardware for feedback to the worker. Fig.  4 gives an overview of the design fields and parameters for worker guidance systems. The design parameters can be consciously designed and thus allow the differentiation between different types of worker guidance systems.
When it comes to transforming a cognitive assistance system, the design parameters of the currently used system as well as the new AR system have to be evaluated and determined. Since the assembly-related work information is not fundamentally changed by the transformation of the system, the evaluation of the existing system leads to the determination of the first design field, the level of information. In the next step, the desired information design has to be specified and the desired information device has to be selected. The parameters for defining the information design are strongly related to the selection of the information device. For example, the selection of an AR device can result in restrictions in the embedding of objects in the environment. The selection of a suitable information device depends not only on the selected design parameters, but also on industrial requirements.

Figure 4
Design fields and design parameters of worker guidance systems according to [14].

Selection of AR device
When integrating an AR application into an industrial environment, certain requirements have to be met. In [41], Lorenz, Knopp and Klimant present four categories of requirements which have to be met when integrating an AR worker support system in industrial environments: user, technical, environmental and regulative requirements. In terms of the user, the most significant requirements are to provide an overview about the required tools and materials, to present the workflow with the help of images, videos or 3D animations, to record statistical data for future planning and hands-free operation. The user requirements for transforming an existing assistance system also depend on the information degree as described in Section 3.1. The most important technical requirements are a stable data connection and the connectivity to a central data storage system. In terms of environment, it should be usable with a hard hat, safety glasses and should sustain a fall from a 1.2 m of height. Since regulative requirements have to be considered differently due to different dependencies, such as country, industry sector and assembled product, it is not discussed any further in this work. [41] provides a full detailed list of the requirements of AR guidance systems.
By considering the conditions in the given industry sector and the defined requirements for an AR device, a suitable medium into which an existing cognitive assistance system can be transformed can be chosen.

Concept and Development Phase
In the concept phase the derived specifications and identified parameters are used to create a roadmap for the implementation of the assistance system. Therefore, the scope of the transformation has to be determined and relevant features have to be formulated. Software objects for the system being transformed are to be formulated and their workload has to be estimated. Examples are user interfaces, masks, programs, jobs, files and databases which are necessary for the transformation of the assistance system. Following the digitalization process, non-technical data, such as paper based instructions, have to be digitized and elaborated.
In the implementation phase, the cognitive assistance system is transformed into an AR application, based on the identified requirements and specified features. During this phase, the developers can follow various processes such as SCRUM or Extreme programming.

Finalization Phase and Launch
The finalization phase is closely related to the implementation phase. Depending on the software development process, the testing is done continuously. Nevertheless, a final test has to be conducted, including the worker who has to use the system in the assembly line. Their feedback and the release of the system is essential for the completion and success of the project. If employees are not involved enough, there is a risk that the new system will be rejected and thus cannot be used successfully. This phase is related to the handover and employee migration phase in the migration work flow, as well as the test and trial phase in the product development process. When the system is finalized, the product can be launched in production and used in the assembly lines to assist the workers.

Quality Assurance
Quality assurance measures of the transformation ensure that the functional equivalence of the old and new assistance system is guaranteed. If partially automated transformations are used for conversion, they must be tested in detail. In general, it is recommended to test the transformations on suitable programs, data and masks. The presence of the old system allows the validation of the AR application and ensures the quality during the process.

CONCEPT VALIDATION
To validate the proposed unified process model, a project has been carried out in an industry-oriented environment. This section presents the prototype implementation in a learning factory at Graz University of Technology. First, the strategy and environment are assessed. Then, the application is specified and a concept is chosen. At the end, the application is implemented, finalized and launched.

Figure 5
The LEAD Factory at Graz University of technology. It is used to train participants "hands-on" in an industrial setting.

Strategy Development and Implementation Environment
The start of the project contained the discussion of a suitable strategy which is to be followed throughout the project. Therefore, the goal was set to transform the existing paper-based work instructions into an AR cognitive assistance system. The new system should be low cost and represent a demonstrator which only has to support basic functionality. Since the prospective workers are constantly changing students, it is hard to integrate the same personas throughout the project, but it is still tried to gain feedback from various interest groups, such as the personnel of the institute.
Next, the environment for the implementation is assessed. Therefore, the so-called LEAD factory has been selected as implementation environment since it offers an industrial setting within a teaching environment as it is usually used for training students. The LEAD factory is a learning factory, which represents an industrial assembly line. In this environment, participants get to know industrial processes and optimization methods by directly applying them in the learning factory. During workshops, participants assemble a scooter on five different workstations. Each work station is equipped with a paper-based assembly instruction, whereas in the scope of this project, the instructions at one specific workstation will be transformed. For this project, workstation 1 has been selected. Fig. 5 shows the assembly line of the learning factory. The working process starts at workstation 1 (right) and follows a U-shape through the assembly line to workstation 5 (left).

Design and Concept Phase
In the next phase, the existing cognitive assistance system design fields are evaluated and the target application, including the selection of the hardware is conducted. The information degree on the paper-based instruction consisted of step by step instructions for which part to assemble and an image showing details on the assembled product. This information is kept as well as the design of the instruction set is defined to stay the same. No embedded 3D objects were integrated as the prototype should only support basic functionality. The digitization of the paper based instructions did not pose a problem, as they were already available in digital form.
Google Glass is selected as information device since it enables assembly using both hands and offers the required hardware to project step by step instructions and images into the field of sight of the worker.
The concept of the application is planned to integrate a server which provides assembly information and communicates with the glasses via Wi-Fi. This way, Google Glass can retrieve the necessary assembly information and can send the progress status via the AR server back to the production system. This allows a real time monitoring of the assembled product and enhances the maintenance of assembly instructions as they can be changed directly on the server. For the control through the instruction steps, a smart watch is connected via Bluetooth to the glasses and waits for the worker input. Hence, the worker does not have to touch their face all the time to start and switch instruction steps, but can control it in the immediate vicinity of his hands. Fig. 6 shows the rough concept for the implementation.

Figure 6
Implementation concept for an AR cognitive assistance system.

Development and Finalization Phase
Having defined the concept, the prototype is developed and tested. Therefore, the communication interface, the user interface and the control interface are implemented according to the defined concept. When the basic functionalities were implemented, the assistance system was tested with selected personas and their feedback was integrated in several loops. The final product was then released and integrated into the workshop concept of the LEAD factory. A view through the prototype is shown in Fig. 7. The image shows the workplace as perceived by the worker. In the top right corner, an instruction view is projected, including the assembly information and images from the transformed paper based instructions. Additionally, the remaining time for the assembly is projected as well, as this information is available from the Manufacturing Execution System.
The control unit (smart watch) of the prototype consists of three main buttons: The "start" button to initiate the assembly instruction and the "next" and "back" buttons to control the sequence of the assembly instructions.

CONCLUSION
In this paper a unified process model for the transformation of a cognitive assistance system has been proposed. Therefore, a migration workflow and digitalization process have been combined and adapted to the needs of a worker guidance system. By transforming an existing paperbased cognitive assistance system into an AR system, the process model was validated in an industry-oriented environment, the LEAD Factory. The developed process model represents a guideline for the transformation of a system, which can be used as a basis for the implementation of such a project. Nevertheless, the model still offers potential for improvement, as not all steps have been defined in detail. In the future, it is planned to specify the steps more precisely; for example, to better define the selection of a suitable AR device. Furthermore, it can also be checked whether the presented model can be adapted for the transformation of other AR use cases. In terms of the prototype implementation, additional studies can be conducted, such as the quantitative comparison of the old assistance system to the new one or conducting tests in real industrial companies.

Notice
The paper will be presented at MOTSP 2020 -International Conference Management of Technology -Step to Sustainable Production, which will take place from 30 th September -2 nd October 2020 in Bol, island Brač (Croatia). The paper will not be published anywhere else.