Development of a Generic Framework to Assess Asset Management Maturity within Organisations

: With the comprehensive Lean Smart Maintenance philosophy and its associated maturity model, organisations were given a tool to reach asset and maintenance excellence. This paper discusses the approach used to transfer the scientifically based methods and concepts of the Lean Smart Maintenance Maturity Model into an assessment structure to generate a generic tool to collect the complete and correct information necessary to determine an organisation ' s maturity level. Research results show that a standardised assessment process combined with continuous improvement cycles, a more accurate assessment of the company's maturity is possible. A well - structured MM assessment supports less experienced assessors whereby experienced assessors will not need a full questionnaire but only a well - structured list of items and their maturity levels.


INTRODUCTION
The last decades, especially the last years with the challenges that arose from the global COVID-19 pandemic, have shown that the ability of companies to change and adapt to volatile market conditions is more important than ever. A central part of ensuring competitiveness in such environments is supporting production with a lean (efficient) and smart (effective) organisation. This is especially true for asset-intensive industries, where the asset management organisation is seen as a key enabler in this and smart factories contexts. Against this background, the Lean Smart Maintenance (LSM) concept, which takes a value-added perspective, focuses on dynamics, organisational learning, and risk orientation. [1][2][3] Based on these works and past maintenance maturity models (MM), a MM was developed as a tool to guide organisations towards LSM, with a strong focus on current developments in the fields of digitalisation. [4][5][6] For the LSM MM's trouble-free utilisation, a standardised assessment based on semi-structured interviews combined with a document review is used. [5] This scientific contribution intends to answer the question that repeatedly arises in reorganisation projects, which is, 'How must a maturity assessment be structured to obtain the greatest possible density of information (complete & correct) for the maturity rating of the organisation being surveyed with the most efficient use of human resources? ' The publication is structured in 8 different sections: after the introduction in section 1, a theoretical overview of the LSM MM and maturity assessment are given in section 2 of this paper, which includes an explanation of the assessment process in detail, the applied question types and the improvement process applied during the organisational analysis. Section 3 introduces the selected methodology, section 4 the expert and interview mode selection process, section 5 an in-detail case description. Section 6 results are presented and discussed, and section 7 concludes the paper with a summary and an outlook into possible future research.

LSM MATURITY ASSESSMENT
MMs are defined as artefacts with elements arranged in an evolutionary scale with measurable transitions from one level to the next and are used for benchmarking, selfassessment and continuous improvement. [5,7,8] Their basic architecture can differentiate MMs: capability maturity models, which focus on the organisation's capabilities and processes, progression models, i.e. evolution of elements and hybrid models, which use the architecture attributes, features, patterns, etc. of a progression model, with transitions between maturity levels defined by a capability maturity hierarchy. [7] MMs can be designed as descriptive to describe the as-is state of a system, prescriptive as a base for improvement or comparative for benchmarking purposes. [9] The LSM MM is classified as a primarily prescriptive, hybrid MM. It uses the Capability Maturity Model Integration (CMMI) base architecture and progressive elements to better understand the model's elements.  [5,6] It encompasses nine main categories arranged from normative, via strategic to operative management: philosophy & target system, corporate culture, business model & service strategy, asset strategy, maintenance budget & controlling, organisational structure, process organisation, knowledge management and data & technology, with the last two being designed as cross-sectional categories. [5] The basic structure of the model can be viewed in Fig. 1. In the hierarchical model, structure sub-categories can be found below the category level, which are comprised of items, which in turn form the basis for the assessment questions. The questions are an aid to find the right manifestation of an item over the different maturity levels. The items themselves are directly linked to the maturity levels of sub-categories, meaning that all item descriptions have to be fulfilled until a certain maturity level for an organisation reaches a certain maturity level in a sub-category. The minimum maturity level in of sub-category within a category then sets the maturity level for the overall category. [5,6] The logic behind the maturity evaluation can be viewed in Fig. 2. Although the main goal of the model was to create a prescriptive and not comparative model, similar asset organisations can be compared with the resulting maturity profile.
The LSM assessment targets to identify the maturity of an asset management organisation within each mentioned management category. For the completeness of an organisational assessment, a structured assessment process, assessment questionnaires and an appropriate team of assessors [10] are necessary alongside the corresponding maturity model. The defined assessment process is based on the ISO/IEC 33004:2015, consisting of seven main, a start and a stop phase. Continuously improving the assessment quality, the defined process was extended by the aspects of the DMAIC-cycle (Define, Measure, Act, Improve, Control). [11] This advancement aims to optimise date respectively information collection and assimilate gathered information alongside the assessment procedure. Figure 3 LSM assessment method, adapted after [12] As shown in Fig. 3, the assessment method includes two independent improvement loops, enabling the organisation that conducts the assessment to perform organisational learning. Organisational learning can be described as identifying gaps and errors and then deriving appropriate measures from them. [13] The first improvement loop is placed parallel to the M-A-I phases. After conducting the first interviews, the gathered information is analysed, and gaps resulting from elements of the maturity questionnaire not being asked are identified. These shortcomings are used as new input parameters for the data collection phase to complete the assessment information. Data collection and the defined assessment questions are scrutinised through this first learning cycle can be categorised as 'double loop learning'. [14] The authors propose a second loop, the General Assessment Improvement Cycle, and this represents the approach of deutero learning adapted to the assessment process. The highest characterisation of learning types, the deutero learning, seeks to enhance the learning process, and in this case, the assessment process itself. [13] Typical questions within this improvement step are, e.g., 'Do we have chosen the right types of questions?', 'Is the phraseology of the questions appropriate for the target group of the assessment?' and 'Is the assessment timeframe correct?' In addition to the assessment process, the assessment questionary is of utmost importance. The employees to be questioned have several tasks to fulfil, e.g. interpret and understand the meanings of the questions asked to provide the right information to the assessors. [15] For the assessment to be carried out properly, the questionary setup was derived by the MM structure. As described before in this section, the model is divided into different items per sub-category, and each item has a certain number of questions to be asked to identify the correct maturity level. Since the LSM MM represents aspects of the normative, strategic and operative management level, the question types must also be comprehensive, reflecting different views, opinions, attitudes, knowledge, and certain organisational behaviours such as value orientation and culture. [16] Therefore the questionary was developed by utilisation of two basic question types. The first one is the closed question with a defined number of possible answers related to the item's content, and the second the open question whereby this type was used to identify processual and organisational details, which are always different between different organisations.

METHODOLOGY
The stimulus for this scientific work was provided by several 'lessons learned' from various reorganisation projects. After previous organisational analysis had been completed, the question arose about whether it is possible to adapt the assessment system to acquire more information in the same time frame of an assessment than was previously possible. The LSM MM describes, in its simplest form, current states of, e.g., processes, structures, characteristics of maintenance organisations' and indicates potential development paths to increase efficiency and effectivity by reaching a higher level of maturity. The LSM MM was further developed using Hevner's Design Science Research approach (DSR) [6], and this is also used in this publication as a methodological guide. DSR consists of three research cycles: the so-called Relevance Cycle, the Design Cycle (DC), and the Rigor Cycle. [17] The methodological structure of this contribution focuses on the DSRs' Design Cycle. The LSM MM was validated with existing data from previous projects. [6] To further meet the DC requirements, the derived questionnaire was validated against an actual organisational development project, part of the validation assessment structure and mode described in this publication. The DC is structured as an iterative process that includes the phases of creation and evaluation of artefacts. Therefore, it is represented with the general assessment improvement cycle, shown in Fig. 3 and discussed in section 2. With the requirement for continuous improvement in the highest maturity level of the CMMI architecture, the iterative process transfers the MM requirements to the assessment and maturity model's design.
The general Assessment improvement cycle starts at the end of the assessment process (AP). Here, a survey was created to assess the efficiency and effectiveness of the AP. This survey was sent to the different assessors of the conducted project, further described in section 5. Questions asked in the survey included general information about the assessors, namely the name and role of the assessor during the interviewing process. Besides the general information, five specified questions were asked to identify the usability and quality of the assessment. The first question (Q1) focuses on the data collection process carried out during the interviews. Question two (Q2) deals with the phraseology and clarity of the assessment questions, and thus whether the interview participants, in the opinion of the experts, have understood them correctly, and the answers given are consistent with the context of the questions. Question three (Q3) is focused on the gap analysis between the target data for a holistic picture of the organisation to the gathered data from the interviews (e.g. identification of not used items or sub-categories of the maturity assessment), and question four (Q4) ties in with the result of Q3, and the possibility to determine the maturity level through the collected information. The final question, Q5, evaluated the assessors' ability to produce reports and perform organisational analyses based on the data collected. Furthermore, the assessors were asked which interview mode they used (Q6) (see section 4) and if they could collect enough information with the interview mode chosen (Q7). The result discussion is seen in section 6.

EXPERT AND INTERVIEW MODE SELECTION
In this section, the systematic process of expert selection for conducting the interviews and the decision process of choosing the interview mode are described. Choosing the right assessors is one of the most critical aspects of the assessment preparation process. [10] The starting point for selecting experts for LSM assessments is the qualification matrix of the assessing organisation. A staff qualification matrix is used to identify individual knowledge gaps in projects and further training. The staff's ability is analysed, reviewed, and expanded at regular intervals. It thus provides the current qualification status of the employees. [18] In the case described, the organisation, a scientific organisation that conducts scientific projects with industrial partners, offers training for employees of mainly industrial companies and lectures to students, differentiates in four different qualification levels: • Theory, acquisition of basic literature, training completed (x) • Project support, first application, first lecture performed (xx) • Repeated lecture, repeated project application (xxx) • Expert, several years of experience in further education and projects (xxxx).
The lead assessors in a project should have a level 2, ideally a level 3 qualification in the relevant qualification categories. General assessors should have at least an intermediate level in the different qualification categories. Assistant assessors need to complete at least the first qualification level most relevant to the assessments. The tasks of the lead assessor include interviewing and the aggregation of information gathered by the different interview teams. The assessor's main task is interviewing. The assistant assessors record the information gathered and lead some first interviews to prepare for possible future roles as assessors or lead assessors. The relevant qualifications for LSM assessments include the following, are however not limited to: • Expertise based on maintenance-and/or asset management related projects • Expertise based on conducted maturity assessments • Scientific expertise gathered by being part of the LSM MM development team.
The qualifications of the different assessors in the assessing organisation are shown in Tab. 1. If further special knowledge is required for certain parts of the organisation, this is further considered when assigning the assessors. Besides the mentioned main qualifications of the assessors, each assessor has a substantial background in moderation and skills in conducting interviews, and the lead assessors have several years of workshop experience.
Due to the LSM MM description's detail with its high granularity, two possible paths arise when interviewing: The first starts with the interviewer filtering the question depending on the interviewees' horizontal and vertical position in the company. For the maintenance manager of a factory, this would, e.g. mean filtering for 'middle management' and 'maintenance'. The interviewer would now only see the questions relevant for this interviewee and question him or her accordingly. The second possibility for the interviewer would be to take the item descriptions in the different maturity levels and think of questions that would generate answers to sufficiently describe the organisations' status regarding the used model. Before starting the interviews, the different interviewers need to be assigned or chose an interview mode.

CASE DESCRIPTION
The case selected for this paper is an LSM assessment carried out in a large, central European company from an asset-intensive industry with slightly less than 2000 employees in maintenance. The company sells its products globally and faces strong international competition. Due to its size, the maintenance organisation is designed with a large central workshop, supporting the two decentralised, assetspecialised departments. The central workshop is further split into specialised units, four of which were relevant for the assessment. Reorganisation projects are handled in a standardised process consisting of 6 stages to ensure the sustainable development of an organisation.

Figure 4 Hierarchical interview distribution
Besides the education and training of the employees on the LSM components, the companies' 'as-is' state has to be analysed. The analysis, in turn, consists of several steps, as shown in Fig. 2. The 'define' phase of the assessment was decided to conduct at least 160 interviews within the organisation. This number was defined to cover all relevant functions and hierarchical levels of the surveyed area with at least two candidates. Based on the defined number of participants, the company and its divisions agreed to make 164 employees available for the interviews. The hierarchical distribution of the different interviewees is depicted in Fig. 4. This approach pursues identifying potential gaps between the management's defined maintenance processes and the operational level's applied to the process and getting a clear picture of all the different relevant organisational interfaces and their potential shortcomings. A further step in the Define phase was to select the lead-assessors, the assessors and assistant assessors, according to the process described in section 4. In the list of possible assessors depicted in Tab. 2, persons 2, 3 and four were chosen as leads, 1, 5, 6, 7 and 8 as assessors and 9, 10, 11, 12 as assistant assessors. From this list, three teams were formed according to the organisational structure: One team for decentralised unit one (DU1), one for decentralised unit two (DU2) and one team for the central workshop (CW) with the size of the teams depending on the number of interviewees per organisation. Furthermore, a flexible team (Flex) was created to support the three other teams. In the next step, the interview plan was created together with the company's management. This resulted in three weeks of parallel interviews, with interview durations ranging from 45 to 120 minutes, depending on the interviewee's hierarchical and functional position in the organisational structure. Furthermore, two of the leadassessors (3 & 4) decided to use the MM items as a base for thinking of questions that would generate answers to sufficiently describe the organisations' status regarding the used model, as described in section 4. All other interviewers used the established process by directly asking questions from the questionnaire.
During the actual interview phase, the three phases measure (data collection), act (data validation), and improve (query gaps) phases were cycled through each interview day. The collection of information was conducted through interviews. At the end of each interview day, the information for each unit was gathered, aggregated, different answers in items compared, and gaps in the information base for the maturity evaluation visualised and analysed to address them during the interviews the following day directly. The check phase included first defining the maturity levels by the assessment team and then validating them in consensus workshops, where experts from the relevant units discussed the results with the assessment team. The workshop participants' years of experience and expertise were contrasted with the assessment results, and small adjustments in the maturity rating were done. However, these workshops pursue two objectives, first, as already mentioned, the validation of the assessment results, and second, employee participation, regardless of organisational level, is increased through participation and voice or voting rights in the maturity level results.

RESULTS AND DISCUSSION
The compilation of the first assessment questionnaires was based on the experience of the lead assessors mentioned above. The initial questionnaire consisted of about 80 core questions selected from a total question catalogue consisting of about 580 questions and sub-questions. These questions were selected to gather first insights into the organisation by posing basic questions about each MM item. The first interview day also represented the first M-phase (data collection) and set the assessment-improvement loop (M-A-I) in motion. A total of seven assessment teams were deployed and conducted 32 interviews on the first day. After the first consolidation phase of the collected information, the ratio of 'essential queries' to 'existing queries' was carried out by the assessment team. Based on this evaluation, which was supported by an Excel tool, it was possible to identify blind spots or contradictory statements from the different areas. Contradictory statements were cross-checked using two different approaches: by comparing them with company documents and repeatedly asking different interview participants for the same information. As the interviews themselves were time-limited (0.75-1.5 hours), those teams that worked with the questionnaire and the predefined closed questions types had a higher response rate than those teams that oriented themselves to the item structure with the open questions types.
The first impression of these differences was examined in detail by the entire team of assessors. These differences in response rates were attributed to three identified factors. Firstly, it was found that the questions were asked 1:1 by the partly less experienced interviewers, which led to misunderstandings and contradictions to the same questions in different areas. The phraseology problem was identified as significant in this context, and it was more prominent at the operative level than at the middle or top management level. Secondly, it was found that there were technical problems in the handling of the assessment tool, which led to limited usability and thus to time that was not usefully utilised. The third and final issue identified was the size of the questionnaire itself. Due to the lack of detailed information about the interviewees' job descriptions, it wasn't possible to adequately predefine the questionnaire regarding the interviewees' knowledge concerning certain parts of the organisation. These three points show the necessity of the authors' improvement cycle, which repatriates the gaps in information identified within the I-phase to the M-phase, enabling the assessment setup's continuous improvement. The M-A-I phases were run through each interview day, which resulted in a significant improvement in the quality of the information received as the interview phase progresses After finishing the interviews, to close the complete DMAIC cycle, the assessors were subjected to a survey (see section 3) to assess the continuous improvement approach's effectiveness, efficiency, and questionnaire design. Survey responses at Q1-Q5 were captured by a typical 5-point Likert scale (1 = very poor, 2 = poor, 3 = neutral, 4 = good, 5 = very good). First glance at Fig. 5 shows four out of five questions (Q1, Q2, Q4, Q5), a low rating with 1 and a rating of 2 on Q3 was given.
This response was based on the assessor that assessed the maintenance logistics area. Although this project was a reorganisation project of maintenance departments, the logistics area was also assessed, as it is an interface area that closely cooperates with decentralised and central maintenance. However, the results reflect that only parts of the LSM assessment are suitable for logistics organisations. The Q1 and Q2 responses resulted in an average value of 3.73 and 3.90, respectively, whereby focusing on the approach and usability concerning 'data collection' and 'data validation' of the LSM-assessment. The best rating was given at Q3 and Q4 with an average value of 4.27 and 4.09, which confirmed that the new questionnaire design is very suitable for identifying blind spots that may arise during the assessment. The last question was ranked with an average value of 3.55, reflecting a good result regarding creating a comprehensive report with the interviews' gathered information. The interview results were also based on the type of interview that was chosen. Nearly 20% of the assessors chose the second type of interview (see section 4), and 80% oriented their conducted interview on the detailed questionnaire catalogue. The last question was asked to evaluate the overall satisfaction with the comprehensiveness of information gathered through the chosen interview method to determine the organisation's correct maturity level. More than 90% of the assessors agreed that they could gather all necessary information with the interview mode chosen. The assessment results were validated with the subsequent consensus workshops held with the individual departments and the correspondingly selected representatives from the operational and management levels, which only resulted in slight adaptions of the identified maturity levels in the subcategories. The general improvement cycle results are the lessons learned from the described case, used to improve the whole assessment. These include that filtering by core questions is less practical than filtering by position and role of the interviewee, that both detailed item descriptions, as well as a complete questionnaire, are necessary for an assessment due to the different qualifications of the assessors and that the assessment has to be further optimised for departments that are positioned at organisational interfaces, e.g. maintenance logistics service organisations.

CONCLUSION
This paper presents the methods employed to improve the LSM assessment during the interview phase (double-loop learning) and after the interview phase (deutero learning).
The authors showed that the presented model for an assessment process and improvement cycle allows effective assessments and continuously improves the interview guide with the underlying MM structure. Limitations include that the assessment method was validated using the LSM MM in one Austrian asset-intensive organisation with six different units. The assessment team consisted of team members from one scientific organisation, whereas seven out of twelve assessors already had experience implementing maintenance reorganisation projects. Finally, this CIP was applied to the specific LSM MM assessment. Further possible applications include testing the presented process on other assessment types (e.g. Total Productive Maintenance, Total Quality Management or European Foundation for Quality Management (EFQM) model assessments). Further research is needed to effectively implement a systematic change process based on identified MM development steps to reach the ideal LSM maturity level. Other necessary research includes integrating the LSM maturity assessment into a holistic excellence framework and evaluating the value added by moving an organisation towards LSM.

Notice
The paper will be presented at MOTSP 2021 -12 th International Conference Management of Technology -Step to Sustainable Production, which will take place in Poreč/Porenzo, Istria (Croatia), on September 8-10, 2021. The paper will not be published anywhere else.