A Novel Computer Vision-Based Advertisement System for Individual Marketing

To improve the performance of marketing efforts, marketers seek new ways of attracting customers. Individual marketing is an efficient way of communicating with individuals. Meeting the individual customers’ expectations and needs is becoming increasingly important in modern marketing activities. Marketers have started to concentrate on specific customers’ demands. The paradigm shift from the masses to individuals has required changes to firms’ marketing strategies. In this paper, a new system called the Personalized Smart Billboard System (PSBS) is introduced, which personalizes promotional messages by developing customer profiles using customers’ accessories and some of their facial features. Once any of these clues are detected, related advertisements are shown to the customers. The system was tested with various portrait photos with different visual appearances and promising results were obtained. We anticipate that this system will present a helpful means of establishing more effective communication between businesses and their customers.


INTRODUCTION
In recent years, the marketing activities of businesses have started to change in order to satisfy the expectations of individual customers.In this context, companies have begun to especially concentrate on individuals and local customers.This means that micromarketing efforts have gained more importance.Micromarketing, which involves individual marketing toward individuals, embraces the personalization of marketing mix elements, such as product, price, place and promotion.The success of personalization primarily depends on the seller's ability to obtain and utilize customer-specific information [1].
In this study a smart advertising system, termed the Personalized Smart Billboard System (PSBS), is developed, which enables the personalization of promotional messages.The system detects faces from waist-up photos and segments the facial region into subregions, such as forehead, eye, mouth and neck.Within these sub-regions, PSBS attempts to find predetermined accessories, including hats, glasses, sunglasses, necklaces and facial hair such as a beard or moustache.Once any of these clues are found, related advertisements are shown to the customer.The contribution of this study is threefold.First, a new system is proposed in which a smart billboard shows the advertisements concurrent with customers' accessories and some of their facial features.In this respect, the system can be regarded as novel and complementary to common smart advertisement systems, which are typically based on the identification of gender and age profiles.The second contribution is the low cost of the proposed system.In PSBS, feature extraction can be performed using a single image, instead of employing a video sequence or a motion sensor, which reasonably reduces the information system resources needed to implement the model in practice.Final contribution is the proposed computational pipeline that integrates relevant computer vision techniques to detect domain-specific objects.Experimental results determine that this pipeline works effectively.
The remainder of the paper is organized as follows.Section 2 provides the theoretical background about market targeting strategies.Section 3 presents related work in the area.Section 4 demonstrates the system workflow.
Section 5 shows the experimental results of the study.Section 6 explains the implications of PSBS, and Section 7 concludes the study and presents possibilities for future research.

THEORETICAL BACKGROUND 2.1 Market Targeting Strategies
Target market selection is one of the important components of the marketing activities of businesses.According to [2], the target market "consists of a set of buyers who share common needs or characteristics that the company decides to serve".Basic strategies for selecting target markets are: undifferentiated marketing, differentiated marketing, concentrated marketing and micromarketing [2].
Undifferentiated marketing, also known as mass marketing, is one of the market targeting strategies that focuses on developing a single marketing program for the whole market.A fundamental issue with this strategy is that it ignores the differences between market segments and treats the whole market in the same manner.Designing the same sweater for male and female consumers is an example of undifferentiated marketing.The second market targeting strategy, differentiated marketing, is the opposite of the undifferentiated marketing strategy.It considers the differences between market segments and therefore targets different marketing programs to related market segments.Producing different sweater designs for men and women is a representative example of differentiated marketing.Concentrated marketing is the third alternative in selecting target markets.As a business tends to concentrate on a small part of the whole market, this strategy is mostly useful to firms with limited resources.The main philosophy in this strategy is to survive in the small market segment rather than die out in the larger market.Manufacturing sweaters only for women is an example of concentrated marketing.Finally, micromarketing is the last strategic option in the selection of a target market.In this strategy, local customer segments and specific individuals' expectations gain more importance.Therefore, businesses start to consider local customer groups (local marketing) and individual customer (individual marketing) preferences.Designing different sweaters for customer A, customer B and customer C is an example of micromarketing [2,3].

Individual Marketing
Individual marketing is the adaptation of the marketing program according to the wants and needs of individual customers.In this context, the data collected about individual customers become more important in the formulation of marketing mix elements, such as product, price, place and promotion [4].This formulation can be developed by learning the consumers' wishes, matching offerings to the true consumers and/or assessing the learning-matching process [5].
Operations and objects are the important parts of the personalization process [5].Operations are the activities that take place in different stages of the process while objects are the important elements in the execution of the operations.The basic elements, customer, customer data, customer profile, marketing output, are classified as objects.On the other hand, interactions, processing, customization and delivery refer to operations.The relationships between the elements in the process of personalization are shown in Fig. 1.

Figure 1 Process of personalization
In individual marketing, customers are at the center of the personalization process.Demanding customers have started to expect products and services that match their individual preferences.Interaction is the way of realistically satisfying the customer demands.Customer interactions can be undertaken using interviews, surveys, web pages, etc. Briefly, the data obtained from these interactions and gathered from external sources enables generation of customer data, such as past buying behavior and demographic characteristics.In the processing stage, the customer data are converted into customer profiles.In this manner, customer segmentation can be conveniently performed using techniques such as fuzzy logic or datamining.The generated customer profiles enable the marketing outputs to be customized and related with marketing mix elements.The delivery stage correlates with the channels that will be used to access customers using personalized marketing.After the delivery stage, a new interaction process begins using the responses obtained from customers [5,6].
It is emphasized that "Customers, whether consumers or businesses, do not want more choices [7].They want exactly what they want-when, where, and how they want it-and technology now makes it possible for companies to give it to them".This sentiment should be adopted as a motto for today's businesses.
Recently, most firms have concentrated on individual marketing efforts and have strived to predict customers' preferences.Numerous examples of this trend exist, among which Amazon.com, Inc. (Seattle, WA) is a well-known case.This website presents special offers and services, such as music and book recommendations, that might attract individual consumers' attention [4].Additionally, related electronic commerce sites use dynamic pricing strategies, which are based on personalized price arrangements [8].Similarly, the electronic commerce company eBay Inc. (San Jose, CA) is an impressive example, which presents personalized offerings matching the probable demands of a consumer, according to past searches and buying behavior [4].Another more general example of the personalization of services is offering a smoke-free hotel room to non-smoker customers [9].

RELATED WORK
There have been various attempts to develop smart advertisement billboards that employ computer vision techniques.A large body of work is focused on the identification of gender and age.For instance, Adidas AG has used in-store billboards and kiosks based on facial recognition technology to recommend shoes for women according to their age range [10].Similarly, Kraft suggests different macaroni and cheese recipes to women appearing to be young mothers who come close to their street-level billboards [11].NEC Corporation has developed a facerecognition-based ad signage system to identify the gender, ethnicity and age range of Japanese shoppers.In this way, their system can suggest appropriate ads for shoppers, such as perfumes for women [12].Further, a consortium of eleven railway companies in Japan launched the 'Digital Signage Promotion' project to determine the gender and age of passersby to collect data on the attributes of people who look at certain ads at certain times of day [13].The German beer maker Astra installed an external digital billboard that employs a built-in camera and facial recognition technology to detect female customers.The billboard tells the person to try Astra beer if it detects a female customer [14].An interactive billboard at a London bus stop estimates the gender of passersby by measuring the distance between their facial features.The billboard displays ads to women and plays a message about a campaign in London [15].
Apart from the practical applications explained above, there have been some academic efforts related to the design and implementation of various smart advertisement systems.Schönböck et al. proposed an interactive billboard called 'MirrorBoard' that aims to capture the attention of passersby [16].It was aimed at enhancing the memorization of a product by providing an interaction between people and the billboard.Moreover, the observer was integrated into MirrorBoard.This integration was carried out using a camera and image processing algorithms, such as background subtraction.The observations suggested that customers are more likely to remember the message given by the advertisement when they can interact with the billboard.In a more recent study, a marketing system was proposed that engages the audience with the advertisement using attentiondemanding advertisements [17].The system enables the necessary virtual environment to attract a larger audience and computes the passerby's count and time spent in front of the display.Furthermore, 3D tracking and face recognition technologies were employed to extract the audience profile and an augmented virtual environment was created to draw attention.In a different study, a digital interactive advertisement system was proposed [18].The system is based on facial recognition and the content of the digital advertisement is changed when people are at a certain proximity in relation to the billboard.Lee et al. presented an intelligent image-based customer analysis system [19], which utilizes 2D and 3D image technologies.The 2D part enables eye detection, light compensation, feature extraction and classification to recognize the gender, age and attention time of people.The 3D part extracts depth information to acquire the viewing distance, height and total number of visitors.The results of the analysis can yield substantial feedback to advertising companies or store owners.

THE SYSTEM WORKFLOW
In this study, a new system, PSBS, is proposed, which personalizes promotional messages by developing customer profiles using their accessories and some of their facial features.The overall framework of PSBS is outlined in Fig. 2. The system is initiated using a waist-up photo of a person.Next, the face region is detected by computer vision algorithms.After this, the detected face is segmented into four regions: forehead, eye, mouth and neck.To localize these regions, a specialized object detector function with varying parameters and certain geometrical dependencies of the human face are used.Next, a hat is searched for around the forehead region.Similarly, glasses and sunglasses are searched for within the eye region.Facial hair, such as a moustache or beard, and necklaces are searched for within the mouth and neck regions, respectively.These searches are performed using object detection algorithms based on color and local image features.According to the detection results, corresponding advertisements are randomly selected from an advertisement pool and are suggested to the customer.Further, the detection accuracy is also computed.The proposed methodology is implemented using the MATLAB programming environment [20].

Face Detection
Face detection is the fundamental step to extract the customer profile.Here, detection is performed using the 'CascadeObjectDetector' function in MATLAB [21].This function is based on Viola-Jones algorithm, which is the first ever real-time face detection system.It starts with the Haar feature selection and integral image creation.Then, Adaboost training stage is performed to select features.Finally, the faces are detected with an Adaboost trained cascade classifier [22].An example face detected by the 'CascadeObjectDetector' function is illustrated in Fig. 3, where the yellow frame encloses the face region.

Face Region Categorization
After detecting the face region, four sub-regions (eye, mouth, forehead and neck) were determined individually.To localize the eyes, the same function used in the face detection was employed, using parameters more suitable for close range photos and for detecting both eyes.For cases where the person wears sunglasses, PSBS will most likely fail to locate the eye region.In these cases, the eye region is estimated using known geometrical dependencies for the human face [23].The localization of the mouth region is carried out in the same way as that of the eye region, using suitable parameters.To include the moustache and beard regions, the mouth region was enlarged upward and downward to a certain extent.Finally, the forehead and neck regions are estimated based on the assumptions defined for the human face.To identify whether a hat is worn, the forehead region was enlarged and shifted upward to a certain extent.To locate the neck region, the mouth region was enlarged and shifted downward, so that it can encompass a potential necklace worn by the individual.
(a) (b) Figure 4 An example of sunglasses detection using image histograms

Glasses Detection
The person is analysed according to three cases within the eye region: wearing glasses, sunglasses or neither.To detect whether a person is wearing sunglasses, the grayscale intensity histogram of the region is analysed.If the mean of the histogram is lower than the grey scale value of 127, which is the centre value between 0-255 for a grayscale image, the system infers that the person is wearing sunglasses.This is the case where darker image pixels dominate the eye region.The other two cases are specified with respect to the density of the corner points in the eye region.Corner points are extracted using a Harris corner detector, which is a widely accepted method for point detection [24].The person is specified as 'wearing glasses' if there are a number of corner points above a predefined threshold within the eye region.After conducting several experiments, this threshold was accepted to be 20.However, this value should not be considered global.If the number of corner points is less than the threshold value, PSBS infers that the person is not wearing glasses.Sample images of these cases are illustrated in Fig. 4 and Fig. 5.In Fig. 4(a), the histogram is skewed to the right for the person wearing sunglasses, and it is skewed to the left otherwise (Fig. 4(b)).Similarly, in Fig. 5, there is a certain amount of detected corner points exceeding the predefined threshold value (red dots) for the person wearing glasses (up) compared with the person not wearing glasses (down).This is because of the fact that the glasses frames produce several corner points.

Facial Hair Detection
In this analysis, the face image is investigated according to three cases in the mouth region: whether a person has a beard, a moustache or neither.To this end, the mouth region is segmented into black and white using a grey-level threshold value.This value is automatically computed by Otsu thresholding method.This method assumes that the image contains foreground and background pixels.It then calculates the optimum threshold value separating the two classes so that their intra-class variance is minimal, or equivalently their interclass variance is maximal [25].Then, the region is separated geometrically into two sub-regions by centring the lip line.The upper and lower regions of the mouth are used to investigate the presence of a moustache and beard, respectively.Next, two image windows are selected from the lower and upper portions and the black/white pixel percentages are computed within these windows.The system infers that the person has a moustache if the black pixel percentage is greater than the white pixel percentage inside the upper window.Similarly, the person is found to have a beard if similar percentages are computed for the lower window.In cases where the white pixel percentages are greater than the black pixel percentages for both windows, this means that no facial hair is detected by the system.
Figure 6 An example of facial hair detection using binary image analysis within image windows Fig. 6 exemplifies the detection of facial hair on a sample image.Note that the black pixel percentages are greater than the white pixel percentages for the upper and lower windows.Therefore, the system infers that the person has a moustache and a beard.

Hat Detection
The existence of a hat is determined by analysing the corner points in the forehead region.As in the analysis of glasses, the Harris method is employed to find corner points.It is observed that a person wearing a plain hat produces fewer corner points compared with a hatless person.By applying a similar strategy used in the analysis of glasses, a threshold value is determined.In this case, 15 corner points are assumed to be the optimum value.Sample images of these cases are illustrated in Fig. 7.It is apparent from this figure that too few corner points (red dots) are detected for the person wearing a hat (up).

Necklace Detection
In this analysis, a necklace is searched for within the predetermined neck region using an edge-based method.First, the edge points are detected using the Sobel operator, which is widely used in image processing and computer vision applications [26].Then, the edge points are counted.If there is a necklace, the number of edge points dramatically increases.The number of edge points decreases, or nearly vanishes, if the person is not wearing a necklace.By defining an edge point threshold (about 200 edge points after many experiments), PSBS decides whether a person is wearing a necklace.An example case is demonstrated in Fig. 8.Note that the edge detector reveals many edge points (white pixels) within the region of interest (upper row), while only a few edge points are generated when there is no necklace (lower row).

PRELIMINARY RESULTS
After detecting the specified features, the PSBS outputs are evaluated.The evaluation is carried out by comparing the results of the system with the reference data, which are extracted manually.In this respect, the 27 portrait photos, presented in Fig. 9, are analysed and the overall detection accuracy is summarized in Tab. 1.
Each photo is analysed with regard to six features: glasses, sunglasses, beard, moustache, hat and necklace.To prepare the reference data, these features are labelled as 1 or 0 manually for each photo. 1 means that the feature is in the photo and 0 means it is not.The results show that the detection ratio ranges between 50% and 100%, where 50% means three out of six labelled features are successfully detected by the system.In Photo #16, all features are successfully detected yielding a 100% success rate.The average success rate is 66%, which can be interpreted as four out of six features being correctly identified by PSBS.The graphical user interface of the system is shown in Fig. 10, where a sample photo (Photo #27) is analyzed.The text in green means that the system detects these features.In other words, in this figure, PSBS claims that the person has a beard, hat and necklace.The text in red refers to undetected features, which, in this case, are: moustache, glasses and sunglasses.According to the reference data, the person does not have a beard or moustache.Additionally, she is not wearing glasses or sunglasses.She is, however, wearing a necklace and a hat.The system detects five features correctly and misses one feature, yielding an overall accuracy of 83% for this specific example.Undetected features are mainly resulted from problem specific (fixed) parameter values used in edge detection, point detection and histogram analysis.With the use of dynamic threshold values, the overall accuracy is expected to increase.According to the analysis results, PSBS suggests pictorial advertisements for detected features, which are randomly extracted from an advertisement pool.The advertisement pool includes various shop banners for opticians, jewellers, milliners and barbers.
The results obtained are believed to be promising.The proposed system has a great potential to be a bridge between the businesses and target markets.By this system, a modern communication tool can be established between businesses and their customers.The system will also bring profit both to firms and customers in terms of reaching target markets in a low-cost and satisfying customers' needs in a true manner.

IMPLICATIONS OF PSBS
The proposed system has important contributions and implications for businesses and consumers.The major potential implications are as follows: • Delivering value to consumers • Establishing true and effective communication with consumers • Following up the benefits of interactive marketing • Enabling promotional messages to be personalized • Presenting true advertisements to true consumers • Providing cost savings to businesses • Advertising firms' products to a broad area

CONCLUSION AND FUTURE RESEARCH
In this study, a computer vision-based advertisement system (PSBS) that personalizes promotional messages by developing the customer profiles is proposed.PSBS extracts features from a waist up-photo of individuals.In this system, regions of interest, such as the forehead, eye, mouth and neck regions are automatically segmented.A MATLAB interface was developed to acquire the feature detection results.The system was tested on 27 sample images and approximately 66% of the predefined features were detected successfully.
We believe the proposed PSBS system has several advantages for consumers and businesses, including that it enables effective communication between firms and customers and will result in cost savings for businesses.
PSBS is a novel system in that it requires low-cost image acquisition equipment and still images.The system is quite different from common smart advertisement systems in that it takes different features into account (i.e., glasses, sunglasses, hats, facial hair and necklaces).It is worth noting that PSBS can be regarded as a good complement to other smart advertisement systems, which mainly determine gender and age profiles of customers.
For future research, detection accuracy can be improved by employing more sophisticated object detection methods and global threshold values.Additionally, more customer features, such as dresses and other accessories, may be considered to expand the advertisement pool.

Figure 2
Figure 2 General workflow of the proposed system

Figure 3
Figure 3 An example of face detection

Figure 5
Figure 5 An example of glasses detection using corner point analysis

Figure 7
Figure 7 An example of hat detection using corner point analysis

Table 1
Evaluation of PSBS outputs Figure 10 A screenshot of the PSBS graphical user interface