Extraction of Eyes for Facial Expression
Identification of Students
G.Sofia M.C.A,M.Phil.,(Ph.D) 1 , 1
Lecturer in Computer Science, Lady Doak College (Autonomous), Madurai.
joesofi@ymail.com
Dr. M. Mohamed Sathik
M.Sc.,M.Phil.,M.Tech.,M.S,M.B.A,Ph.D 2 2
Associate Professor in Computer Science, Sadakathullah Appa College, Tirunelveli.
mmdsadiq@gmail.com
Research and Development Centre, Bharathiar University, Coimbatore.
Abstract – Facial expressions play an essential role in communications in social interactions with other human beings which deliver rich information about their emotions. Facial expression analysis has wide range of applications in the areas such as Psychology, Animations, Interactive games, Image retrieval and Image understanding. Selecting the relevant feature and ignoring the unimportant feature is the key step in facial expression recognition system. Here, we propose an efficient method for identifying the expressions of the students to recognize their comprehension from the facial expressions in static images containing the frontal view of the human face. Our goal is to categorize the facial expressions of the students in the given image into two basic emotional expression states – comprehensible, incomprehensible. One of the key action units in the face to expose expression is eye. In this paper, Facial expressions are identified from the expressions of the eyes. Our method consists of three steps, Edge detection, Eye extraction and Emotion recognition. Edge detection is performed through Prewitt operator. Extraction of eyes is performed using iterative search algorithm on the edge image. All the extracted information are combined together to form the feature vector. Finally, the features are given as an input for a BPN classifier and thus the facial expressions are being identified. The proposed method is tested on the Yale Face database.
Keywords: Facial Expression, Edge Detection, BPN Classifier, Emotion Recognition
I. INTRODUCTION
With the ubiquity of new information technology and media, more effective methods for Human Computer Interface are being developed which rely on higher level image analysis techniques which has its wide applications in automatic interactive tutoring, multimedia and virtual environments. For these tasks, the required information about the identity, state and intent of the user can be extracted from images and make the computers to react accordingly, ie. by observing a person’s facial expressions. Faces are rich in information about individual identity, and also about mood and mental state, being accessible windows into the mechanisms governing our emotions. The most expressive way humans display emotions is through facial expressions. Facial expressions are the primary source of information, next to words, in determining an individual's internal feelings. In the virtual environments, for the computer to interact with humans, it needs to have the ability to understand the emotional state of the person which is also applicable for the virtual classrooms too.
good at observing student’s facial expression, manner and every action and movement. This helps the teachers to understand their own weakness, discover it and change it.
[10]Students use smiles, frowns, nodding heads as shown in figure1 and other cues to tell teachers to slow down, speed up or in some other way modify the delivery of instructional material and to express whether they have understood or not. Momentary expressions that signal emotions include muscle movements such as raising the eyebrows, wrinkling the brow (the forehead or eyebrow) , rolling the eyes or curling the lip. [12] When students are feeling uncomfortable, they may squirm, blush, bite their lip, pick at their fingernails, and have a hard time maintaining eye contact as shown in figure1. To be a good receiver of student messages, a teacher must be attuned to many of the subtle nonverbal cues that their students send. It is just as important for teachers to be good nonverbal communication senders as it is for them to be good receivers.
Figure.1 Facial Expressions
Recognition of emotions from facial expressions involves the task of categorizing active and spontaneous facial expressions so as to extract information about the underlying emotional states. Detecting facial landmarks (such as position of eyes, nose, mouth, etc.) play an important role in face recognition systems [1]. In practical face recognition system, these subsystems determine the denotation behind the expression of the recorded images.[2] This paper, however focuses on the robust and accurate detection of landmark points on the face and classifying the expressions accordingly. The presented approach uses iterative search algorithm to effectively represent the landmark points.
Approaches for the recognition of emotions from facial expressions can be are divided into two main categories [3]- target oriented and gesture oriented. In the former, recognition of a facial expression is per formed using a single image of a face at the apex of the expression. Gesture-oriented approaches extract facial temporal information from a sequence of images. Transitional approaches were also developed that use two images, representing a face in its neutral condition and at the apex of the expression. We employ a target-oriented approach and our proposed work consists of three steps, namely edge detection, facial feature extraction and emotion recognition.
facial features namely the eye block by an iterative search algorithm making use of the edge information of the cropped face region in gray scale. Finally, we perform the recognition of two basic emotions of students in the classroom – comprehensible, incomprehensible, by giving the extracted blocks of eye to a feed forward neural network trained by back-propagation [4] which identifies the emotion expressed by the facial expression.
The remainder of this paper is organized as follows. The methods adopted in this paper are presented in section II. The experimental results on the Yale face database are discussed in section III. Finally, a conclusion and directions for future work are briefly covered in the last section.
II. METHODS
Initially the acquired original images are converted into gray scale images. Pointing the centers of two eyes on each face image, all images are properly rotated, translated, scaled and cropped into 100×100 pixels. Images are then subjected to some image pre-processing operations. The image pre-processing phase includes contrast enhancement, illumination Normalization, filtering and Face Localization [5]. In our work, such a preprocessed image is taken and its corresponding edge image is obtained by applying the Prewitt edge operator. Then the facial features that correspond to a facial expression, namely the eye block is extracted from the face image using this edge information of the face. Then recognize the emotion from the extracted facial features. Though various methods exist for emotion recognition, neural networks hold its position due to its robustness. So, we apply a neural network based approach for recognizing emotions. The actual process is described in the following sections.
A. Edge detection
Edge detection method finds the edges in the given image and returns a binary matrix where the edges have value 1 and all other pixels are zero. The given image is a non-sparse numeric array but the output image is of class logical which means the matrix of the output image will be with 0’s and 1’s. The output of edge detection should be an edge image or edge map [8], in which the value of each pixel reflects how strong the corresponding pixel in the original image meets the requirements of being an edge pixel.
Here we use the gradient-magnitude method Prewitt, which seeks out areas of rapid change in pixel value; threshold is used to calculate gradient magnitude. The Prewitt method finds edges using the Prewitt approximation to the derivative. It returns edges at those points where the gradient of the image is maximum. Edge detected image is given in Figure2.
Fig. 2. Edge detected image
B. Extraction of Eyes
Based on our knowledge the eyes are present in the upper portion of the face region. So we need to search for eye only in the upper half portion of the edge image. The method employs an iterative search algorithm which traverses in the vertical direction and counts the number of white pixels in horizontal overlapping blocks. Now, the block which contains maximum number of white pixels is the required block which contains the eyes. This block is extracted from the grayscale image is shown in Figure3.
Figure 3. Eye Block
C. Classification
The extracted features are fed as inputs to the Back Propagation Neural network, by resizing the eye block to 20 x 32, and the 2-D matrix is converted into 1-D vectors such that each row follows one another sequentially to form a single column. Thus, 640 x 1 column vector is obtained from the eye block, which is given as the input to the neural network.
TABLE 1: NEURONS IN EACH LAYER FOR OUR THREE LAYER NEURAL NETWORK
S. No
Layer Number of Neurons
1. Hidden layer 1
64 2. Hidden
layer 2
16 3. Output layer 2
The number of neurons that has to be in the output layer is fixed as we know the number of emotions that we are going to consider is 2. Therefore, in our case, the number of neurons in the output layer is chosen as 2. The number of hidden layers in the network and the number of neurons in each layer is chosen by trial and error method based on the performance function until it reaches the specified goal. By trying various combinations, we have chosen the efficient architecture, which is a three layer feed forward network where there are two hidden layers and an output layer. The neurons in each layer are shown in Table 1. We have chosen Back-propagation training algorithm for training the network because of its simplicity and efficiency. For training the network to recognize various emotions, we used different face images from Yale databases [9]. The faces from which samples for training were extracted are shown in Figure 3.
The study included survey from hundred (`100) students of an autonomous college both from postgraduate and under graduate level. Considering the survey along with the results of previous researches, the following expressions of eye block are considered to be the expressions which measure the comprehension level. The positive expressions include - No Expresssion, Eyes are bright and Raise eyebrows slightly and the negative expressions include Eyebrows raised more, Eyes shrink, Eyes Closed(sleepy), Angry,
Looking out, Looking down and looking sad.
a. Expressions for the state comprehensible
b. Expressions for the state incomprehensible
Figure 3. Sample Eye blocks for training
III EXPERIMENTAL RESULTS
This algorithm is applied on various face images containing the frontal view of the human face using Matlab7.0. The images were obtained from the Yale databases available in the World Wide Web. For emotion recognition, the neural network is simulated. Before simulation, the network has been trained with samples, corresponding to various emotions as targets. The training performances of various emotions are given in Table 2. For the final step, i.e. Network Simulation, the image vector, which is of length 640X 1, is given as the input to the trained neural network and the network is simulated to obtain the recognized emotional category. When the network was simulated with the test input vectors corresponding to various emotions, as mentioned above, the obtained recognition rates are mentioned in Table2.
TABLE 2. RECOGNITION RATE
S.No. Emotion Recognition Rate
1. Comprehensible 78
2. InComprehensible 86
IV. CONCLUSION
Recent research documents tell that the understanding and reorganization of emotional expressions plays an important role in the development and maintenance of social relationships. By accurately interpreting another’s emotions, one can obtain valuable information about social situations. Facial expressions offer important clues to the meaning of social conversations, and those who are able to accurately interpret facial expressions benefit from the information given. This paper presents a neural network based face recognition system for recognizing the comprehension level of the students from their facial expressions. A face recognition system should identify a face from a new image despite the variations between images of the same face. From now on, the effectiveness of the method will be justified by testing it with face images of more persons and of some common databases. Therefore, our next step will be to improve the algorithm and including more features which would be able to employ more complex classifiers which reduces the error value and improves the recognition rate.
REFERENCES
[1] Resmana Lim, M.J.T. Reinders, Thiang ,” Facial Landmark Detection using a Gabor Filter Representation and a Genetic Search Algorithm”, Proceeding, Seminar of Intelligent Technology and Its Applications (SITIA’2000), Graha Institut Teknologi Sepuluh Nopember, Surabaya, 19 April 2000.
[2] Jain, L.C. et al. (eds.), Intelligent Biometric Techniques in Fingerprint and Face Recognition, CRC Press, NJ, 1999.
[3] E.D.Cowie, R.Cowie, W.Fellenz, S.Kollias, J.G.Taylor, N.Tsapatsoulis, G.Votsis, “Emotion Recognition in Human Computer Interaction”, IEEE Signal Processing Magazine, Jan 2001.
[4] G.Cottrell, C.Padgett, “Identifying Emotion in Static face image”, University of California, San Diego, Nov1995.
[6] Mohamed Roushdy, “Comparative Study of Edge Detection Algorithms Applying on the Grayscale Noisy Image Using Morphological Filter”, GVIP Journal, Volume 6, Issue 4, December, 2006.
[7] Ehsan Nadernejad, Sara Sharifzadeh, Hamid Hassanpour, “Edge Detection Techniques: Evaluations and Comparisons”, Applied Mathematical Sciences, Vol. 2, 2008, no. 31, 1507 – 1520.
[8] B. J. Lei, Emile A. Hendriks, M.J.T. Reinders, “On Feature Extraction from Images”, MCCWS project Information and Communication Theory Group, Tuesday, June 01, 1999
[9] YALE Face database – http://cvc.yale.edu
[10]Body language in the classroom-By Miller, Patrick W., Journal Academic English, Publisher: Association for career and technical education
[11] Building a Harmonious Classroom Atmosphere - By Huloria, ArticlesBase
[12] Can some people read minds? – by PERRY, BRUCE, Science World, Sep 4, 2000
[13] Recognition Of Facial Expressions Of Six Emotions By Children With Specific Language Impairment, Kristen D. Atwood, Brigham Young University, August 2006
AUTHORS
G. Sofia has been working as a Lecturer in the Department of Computer Science, Lady Doak College, Madurai. She has completed M.C.A in Manonmaniam Sundaranar University, Tirunelveli and M.Phil., Computer Science in Mother Teresa Women’s University, Kodaikanal. She is pursuing Ph.D in Bharathiar University, Coimbatore under the guidance of Dr. M. Mohamed Sathik. She has presented many papers in National and International conferences. Her areas of specialization are Image Processing and Virtual Reality.
