Implementation of Augmented Reality Application and Computer Graphics: The Case of the Stolen Paintings

This paper addresses the issue of reconstructing digital paintings and making them visible with the use of an augmented reality application. It contributes through raising awareness of stolen heritage due to a necessity to ensure digital copies and provide better safety for works of art, paintings, historical objects etc. The paper presents a novel method for bringing the lost museum artifacts back to public eyes by using AR in a simple and intuitive way, as well as model for archiving existence of 2D or 3D form. The implementation of the whole process has a methodological approach which is introduced in three phases. The first one consists of painting reconstruction, second relies on marker selection while the third one is based on creating the android-based AR app itself. The results of this implementation show that AR should be used extensively, not only for audio guidance or for additional information, but moreover for raising awareness of the need for digitalization of the arts and heritage in general.


INTRODUCTION
The illegal trade of numerous works of art, paintings, historical objects, antiquities, books etc., with the goal of making profit has been part of human history throughout the ages.Criminal activities associated with them have not diminished, and this in part has led to the impoverishment of the existing cultural treasury fund.Interpol's report states that illegal trade in art and antiquities represents the third largest branch of international organized crime and the most common type of illegal activity.This problem has been addressed to many institutions, such as museums, state ministries, Interpol's national committees, the United Nations Educational, Scientific and Cultural Organization (UNESCO) etc.As for today, the problem is still not covered in the manner it could be.In addition to finding lost or stolen heritage, it is necessary to maintain detailed written and photographic records of each item and digitalize inventory cards and books [1].
In addition to this, mobile digital devices, such as tablets and smartphones, are rapidly gaining popularity.Mobile applications are easy to access and simple to create from scratch, which leads us to today's ICT which allows simple and fast digitalization, visualization and interaction with previously created analogue paintings.Augmented reality (AR), is a technology with great potential [2] that allows a user to access information that would not physically exist.Augmented reality represents an interactive experience of a real-world environment whose elements are "augmented" by computer-generated perceptual information [3].Augmented features that a user can experience, apart from visual, could also be auditory, haptic, somatosensory (creates a sense of touch), and olfactory (creates a sense of smell) [4].The information displayed to a user can be constructive or deconstructive depending on whether the information is added to the natural environment or generated information is locked/blocked by a part of the real environment.It is fair to say that AR partially or completely changes the perception of a user's real world environment.
This technology can reconstruct an augmented experience as it once was or what it might be like in the future.This is extremely important for the earlier stated problem of stolen heritage, in our case mainly paintings, which represents the major part of this work.

RELATED WORK
Augmented reality is a technology that allows, in this instance, museums to create and manage augmented exhibitions based on 3D models and computer generated images (CGI) [5].Augmented reality in civil usage is not a new technology.Research dates back to the past century where researchers and later the industry began to use sound recordings as an augmented reality for tourist and educational purposes [6].Other authors approached application of this technology by extending existing application solutions for museum guides with visual and auditory extensions [7].AR represents the next logical step after use of classic museum applications for mobile devices which were used to collect, edit and share certain (educational or entertainment) content.It is well known that AR has the potential to be a successful alternative to navigation systems and a pathway for user interaction and behavior in closed/open museums and other areas.AR technology allows museum visitors to interact with the content in an intuitive and exciting way [5].In addition to the mobile technology mentioned above, AR glasses could play an important role in the intuition of use of this technology [8].The question is: what kind of impact does a mobile device that a user is holding have on the perception of movement and its orientation?Researchers are still looking for answers that could have a major impact on the widespread use of this technology.A study was conducted where 135 students participated [9], which shows that respondents who used AR had longer retention on art images, learned more and stayed longer in the flow state.A comparison was made between the AR group, the group that only used audio guidance, and a group that did not have any form of augmented reality [9].The research has been confirmed by other authors that it is still not known how all AR modes affect a user [10].The software solutions used to create augmented materials are broadly available, easy to use, and multiplatform.Once created, content can be used on any mobile device that possesses a marker recognition camera and an application with which the marker is connected.Such an application can also be part of a web interface that allows access to a museum and can enable remote interaction by a user with such a system through a website.Researchers in this area state that it is necessary to expand the augmented materials area and include information about artists themselves, the style of painting and other interesting things that a user could, but should not necessarily choose [11,12].This is about expanding information, but again in the form of the problem stated above.
Given this idea, the authors of this paper searched the GoogleScholar database of expert and scientific papers, as well as the Science Direct database, with the following query: "stolen heritage" AND "cg reconstruction" AND "AR".The results of Science Direct was just one paper that dates back to 2008, while Google Scholar provides three papers that were from 2010, 2011 and 2012 respectively.When a query was set only with the following keywords: "stolen heritages" AND "Augmented Reality", Science Direct gives two results, none of which is relevant to this subject, while Google Scholar does not give any result for this query.The author of this paper was unable to find any instance in which stolen arts were reconstructed using AR technology.For this reason, this work is significant from the aspects of graphic technology, information and communication technology, but also from the aspect of preserving art and raising awareness of the digitization of the same.

METHODOLOGY 3.1 Reconstruction of the Paintings
Initially the work examined all paintings and photographs that required reconstruction.Reconstruction is necessary because pictures were stolen and never found or returned to a museum.The problem arose when paintings were stolen and museums did not have a digital copy of such an item.There were only records or image names under which research was conducted to find the same on the Internet.Several images have been found in 20 or 30 year old analog video TV programs with exceptionally poor quality.However, on the basis of these videos, a static image was extracted that was then processed as much as possible.In instances where there was a monochromatic copy of the picture and the original was in color, a picture had to be reconstructed to match the original as close as possible.It is extremely difficult to achieve all the shades, the feel of the brushes, the canvas, and everything else that the artist leaves as an impression in one painting but the art itself no longer existed, so this approach was the only one available at that moment.After initial reconstruction the pictures were then hand painted with 2D drawing programs.

Creating and Selecting Markers
In this instance markers are images (which can also be 3D objects) that behave as a trigger of a particular action in the AR application.Markers can be any pattern, however there are rules that should be followed because if a marker is incorrectly made, it is more likely that the AR application will not work properly.The paper outlines the rules relating to the markers best practice [13].

EXPERIMENTAL AND PRACTICAL USAGE OF AR IN MUSEUMS 4.1 Main Problems and Painting Reconstruction
The first and fundamental problem was the lack of references to original images.The references authors received from museums were mostly black and white photographs of these artworks which, besides having no color data, were in many cases destroyed with a flash glare that could be seen on the photographs.Flash glare often covered a large part of pictures, as shown in Fig. 2.
A small sample of photographs was taken from old TV shows from the 1980s and partially from the Ministry of Police and Interpol archives used in international stoning for stolen heritage, so the reference basis ranged from bad to catastrophic.To make these references something similar to original artwork, authors had to digitally reconstruct them.This reconstruction was conducted by trying to obtain a better black and white photo as soon as possible.The authors achieved this by combining several black and white photographs into one that was relatively clean and without flash glare.In a couple of instances the authors even had to manually draw parts of the image to compensate for a lack of detail.Following this, authors combined a couple of photographs from TV clips and police databases that were in color, but with very small resolution, and used them to colorize those black and white photos that were of a slightly higher resolution and with more detail.From a total of 26 paintings for which the authors had some references (more than 60 paintings were stolen), authors were able to reconstruct 13 of them to the levels that were usable for this Project (this is not a 100% accurate and faithful reconstruction, but sufficient for its needs).
As Fig. 3 shows, two steps were made to get to the relative clean example of black and white photograph that was used in the next step for color coding.The first step aimed to correct the photograph's perspective since it was not properly taken in the first place.The second step removed the first layer of the flash glare that had altered and destroyed one part of the original painting.For this purpose, the authors of this paper had to manually reconstruct the left shoulder and background.Details were damaged but not so badly that they could not be easily fixed.This is the basic principle of all of 13 photographs that were reconstructed.It is interesting to keep in mind that the authors did not have any professional or academic qualifications in art or painting.

Second Pass of Camera Flash Glare Removal and Highlight Correction
Removal of scratches and Leftover Flash Glare   4 shows some finer techniques that were used for removal of scratches and leftover flash glare from the earlier photograph.It is necessary to complete this with as much detail as possible, but on the other hand it is good to think about the medium on which the photograph will be shown.In this instance, 10-inch tablets were used with a (full HD) resolution of 1920×1080 pixel with 218 pixels per inch (PPI).With that said, it is not possible to notice details smaller than a single millimeter in size.The final touch was converting the newly created black and white photograph into low quality color to allow colorization of the reconstructed photograph.The transfer of the colors had to be done with precision since this was the last step in this reconstruction and the outlines of the paintings were strong enough for a viewer to notice the separate elements that coexist next to each other on the object (e.g.badges, different pieces of clothing etc.).Fig. 5 shows the final artwork completed by the authors of this work.

Application Development and Testing
An augmented reality application was created using the Vuforia system.Testing demonstrated that Vuforia is the most reliable, most accurate and most stable AR system.The authors tested free alternatives such as NyARToolkit, ARToolKit and EasyAR, but they all gave poorer results.The biggest problem that arose was the time required to detect markers and the stability of the displayed AR image.With free AR alternatives, displayed objects often "vibrate" on the screen or sometimes even disappear for a moment.This is especially true for low-end mobile devices and in the case of more complex scenes (e.g. more detailed 3D models).However, in this Project the authors did not have any complex 3D models (the artwork projection itself was on a 3D plane), but authors still decided to use Vuforia because it is a platform which authors intend to continue to use in the future work in this field.

Figure 6 Reconstructed Image Sample and AR Application
Since the idea of the Project was to have an empty picture frame for each of the paintings from across the museum where paintings were stolen, the authors had to create markers that the AR application could use for 3D tracking.To this end the authors used a free online application to generate AR markers, and have also added some short information on each painting.Since these generated markers are unique to each image, and also had very good sample and contrast ratio, they provided highquality and stable tracking regardless of whether the application recognized an entire marker or just one part of it.While testing the application, the essential factor was the ratio of the virtual artwork and the size of the actual picture frame.The authors wanted to get the effect that every virtual painting was perfectly "seated" in its original frame.To achieve this, the authors had to create these virtual paintings in the exact dimensions and ratio as their real-world counterparts, and then test it all over the actual frame replicas.In the end, this effect succeeded, and all the paintings that were shown through the AR were exactly in their actual frames.
This Project and the exhibition showed an interesting use of technology to display something that will probably never be seen in reality again.Also, it clearly illustrated the need to create a detailed and extensive database of all heritage, so that, in instances of theft, good quality references exist for verification and validation of any found artwork.With this in mind authors would like to introduce a model for archiving of cultural heritage which would be useful for Augmented and Virtual reality as well as for all forms of digital backup.The model is presented in Fig. 7.

CONCLUSION AND FUTURE WORK
The purpose of this paper is to illustrate that a serious problem exists with regard to stolen, destroyed or lost heritage, but also the fact that this problem is still not being addressed in an adequate manner.Museums are still missing fully digitalized artwork both for the form of images (2D objects) or sculptures (3D objects).It is necessary to create a unique digitization approach that would ensure the preservation of the originals in cases of physical loss.The paper shows an example of an augmented reality that goes beyond the easy use of audio material that is visible in various exhibitions around the world.Augmented reality has found usage in almost all spheres, and so the museum examples are not merely exceptions.The advantage of this approach is that a user is the center of attention and his own movements and decisions affect the information displayed on-screen.These capabilities are extremely effective and we need to explore the boundaries that delineate the point between intuitive and enjoyable work and the moment when technology begins to distract attention and create a negative counter effect for users.The paper also shows a type of marker that is highly detailed and provides tracking even in conditions of low lighting that are often present within the museum environment.In addition, this marker works well with the Vuforia system, which was used for creating augmented reality, even when only one part of the marker is visible.It is important to note that this created AR application can be used outside of the museum if a user has previously installed the application locally on his or her own mobile device.In this case, the AR exhibition can be used with brochures in any space and serve as an augmented gift or souvenir.
For this manner, we have created this example as well as Archiving model.The result of our work should be greater involvement to ensure best practice for securing existing art and any kind of cultural form.
Future work will focus on marker quality testing and defining minimum requirements for quality tracking.It is necessary for a rendered model to be as stable as possible and remain on the mobile device's screen.Future work will also aim to extend similar projects and provide additional ways in which an audience may be in touch with an exhibition.

•
Avoid organic shapes, • Avoid low sample contrast, • Avoid Feature distribution, • Avoid organic shapes, • Avoid repetitive patterns, • Use local contrast enhancement, • Use Thick Glossless Paper (200-220g/m), • Use flat paper with clean or no background, and • Use a large marker of at least 5cm to 12 cm.

Figure 1
Figure 1 Generation of a Used Marker

Figure 2
Figure 2 Initial Low Quality Picture Sample Perspective Correction

Figure 3
Figure 3 Perspective Correction and First Pass of Glare Removal

Figure 4
Figure 4 Scratch Removal and Coloring the Photo

Fig.
Fig.4shows some finer techniques that were used for removal of scratches and leftover flash glare from the earlier photograph.It is necessary to complete this with as much detail as possible, but on the other hand it is good to think about the medium on which the photograph will be shown.In this instance, 10-inch tablets were used with a (full HD) resolution of 1920×1080 pixel with 218 pixels per inch (PPI).With that said, it is not possible to notice details smaller than a single millimeter in size.The final touch was converting the newly created black and white photograph into low quality color to allow colorization of

Figure 5
Figure 5 Final Reconstructed Image Ready for AR Application

Figure 7
Figure 7 Archiving model for AR and VR reconstruction