However, in order for this type of application to work properly, faces must be captured with high clarity and sharpness. Occlusions such as glasses, sunglasses, face masks, scarves, hands, and more cause serious damage to face images and weaken the identification performance of face-based applications. The difficulty of the task lies in the fact that, a reconstruction method must find a way to restore the closed face parts to a non-closed form, aiming to generate a clean face.
It is these details that further raise the level of complexity of the facial reconstruction process. The aim of the thesis is to restore occluded face images to a non-occluded form in order to facilitate their identification. To achieve this, we explore a number of imaging models and evaluate them on a face recognition task.
The first, supervised method, known as Generative Landmark Guided Face Inpainting (or LaFIn) [17], exploits some of the most v. To evaluate the proposed methods, we worked on a part of the popular CelebA dataset, which contains representations of the faces of many famous personalities.
PREFACE
INTRODUCTION
- Motivation
- Related Work
- Supervised Methods
- Unsuvervised Methods
- Objective
These details, such as the distance between the eyes or the shape of the chin, are then converted into a mathematical representation and compared with data on other faces collected in a facial recognition database. However, external conditions are not the only ones that can affect the quality of the face recognition process. However, most of the methods mentioned above can hardly preserve the structure of the original image, and the painted result is often blurred, especially in large occluded areas.
In the paper [19], a non-blind coloring method suggests a unified scheme to determine the filling order of the target region, using an exemplar-based texture synthesis technique. The first, supervised method known as Generative Landmark-Instructed Face Painting (or LaFIn) uses some of the latest. We will elaborate more on data structure and occlusion types in the next chapters of the thesis. Regarding the evaluation process, three different evaluation models were used that aim to reveal the dominant in.
LAFIN: GENERATIVE LANDMARK GUIDED FACE INPAINTING
- Why adopt landmarks?
- How to guarantee attribute consistency?
- Network Architecture
- Landmark Prediction Module
- Image Inpainting Module
- FANFace
Once the landmarks are acquired, they immediately determine the topology structure, pose and expression of the face, as shown in Figures 2.2 and 2.3. By the term consistency we mean the need to connect the clean and the painted parts of the face in a smooth manner so that there is no visual distinction between them. To meet the consistency requirement, the inpainting algorithm must take the information of the clean parts as a reference point to reconstruct the occluded parts.
In the context of Single Image Super Resolution (SISR), this increases the network's ability to reconstruct larger and more complex edge structures. The goal of the Landmark Prediction Module is to retrieve a set of 68 landmarks from a cor. For the purposes of face inpainting, we are more concerned with obtaining landmarks that can accurately identify the facial structure and its basic features. such as pose and expression, rather than finding the precise location of each unique facial landmark.
The reason for this simplification is that most of the benchmarks between them do not provide important information about the face painting task. LaFIn's Landmark Prediction module follows the same architecture as most preex. Specifically, it is built on the MobileNetV2 model, proposed in "Mobilenetv2: Inverted residuals and linear constrictions" [24] and fo. Figure 2.4: Evolution of remaining blocks. a) MobileNets: Residual Block, (b) MobileNetV2: Inverted Residual Block [24]. In fact, the improved MobileNetV2 architecture leads to a 75% reduction in the number of network parameters and increases the average accuracy achieved on the ImageNet dataset by 1.6%, compared to MobileNets.
The purpose of the Image Inpainting module is to restore faces by capturing obscured images and their predicted orientation points. It consists of repeatedly applying two 3 × 3 uncushioned convolutions, each followed by a ReLU activation function and a 2 × 2 max pooling operation with step 2 to downsample. Each step in the expansive path consists of upsampling the feature map followed by a 2×2 convolution.
Finally, in contrast to works such as [7], where two discriminators are used (i.e., the global discriminator evaluates the aggregate consistency of the image, while the local one tries to ensure the local consistency of the drawn region), LaFI's imaging module uses only one discriminator, which as the input requires only the image and its landmarks.
ROBUST PRINCIPAL COMPONENT ANALYSIS USING SIDE INFORMATIONINFORMATION
- Problem Definition
- Principal Component Analysis
- Robust Principal Component Analysis
- Problem Variation
- Problem Solution
In fact, gross errors are now ubiquitous in modern data-centric applications because some measurements can be AR. In the new problem, the goal is to recover a low-rank matrix L0 from highly corrupted measurements. The entries in S0 can be of arbitrarily large size and their support is assumed to be sparse but unknown.
35], that not only can this problem be solved, but it can be solved by tractable convex op. The authors of [35] finally showed that the Principal Component Pursuit (PCP) solves under rather weak assumptions. In fact, they proved that the above problem can be solved by efficient and scalable algorithms, at a cost not that much higher than the classical PCA one.
The authors chose to solve the convex PCP problem (3.4) using an Augmented Lagrange Multiplier (ALM) algorithm introduced in [36]. ALM has the ability to achieve high accuracy rates in a small number of iterations and it works stably over a wide range of problem settings, without requiring parameter adjustments. A generic Lagrange multiplier algorithm [37] will solve the PCP problem by repeatedly setting (Lk,Sk) = arg minL,Sl(L,S,Yk), and then it will update the Lagrange multiplier matrix viaYk+ 1=Yk+µ (M−Lk−Sk). For this particular PCP problem (3.4), it was not necessary to solve a series of convex programs, having realized that both minLl(L,S,Y) and minSl(L,S,Y) are very simple and efficient solutions.
The authors let Sτ : R →R denote the shrinkage operator Sτ(x) = sgn(x)max(|x| −τ,0), which naturally extends to matrices,Sτ(A) by applying it to matrixAelementwise.
- Robust Principal Component Analysis using Side Information, Features and Missing ValuesMissing Values
- Problem Upgrade
- Problem Solution
Many variations have been proposed over the years trying to deal with the convex PCP problem (3.4) in a more efficient way for many different applications, including the back. The main disadvantage of this model is that the functions must be precise and silent, which is not trivial in practical scenarios. Finally, the presented model is able to use side information to exploit prior knowledge of the column and row spaces of the low-rank component.
At first, the authors of [21] presented a PCPSM model, which uses side information with missing values. In order for (3.10) to be valid, they impose as a precondition that a noisy estimate of the low-level data component S ∈Rn1×n2 must be available. Then, they used their proposed PCPSM model to generalize PCPF (3.8), which led to the introduction of the new PCPSFM model, using side information, features and missing values.
The low rank matrix L is recovered from the optimal solution (H∗,E∗) to the objective (3.11) viaL=UH∗VT. Similar to solving the convex problem (3.4), the authors chose the alternating direction multiblocks of multipliers (ADMM) method to deal with the problem (3.11). However, for ADMM to be effective, it must be ensured that the U,V features correspond to orthogonal matrices.
In this specific problem (3.11), a small number of iterations is of great importance, due to the high computational costs incurred by oc. For example, the orthogonalization of the features U,V via the GramSchmidt process has a number of operations of O(n1d21) and O(n2d22) re. If the features U,V are also not present, PCP with missing values can be recovered by fixing both of them to identity.
Later we will refer to three basic categories of parameterizations that we used to conduct our experiments for the task of face inpainting.
EXPERIMENTAL EVALUATION AND DISCUSSION
- Data Preparation
- Dataset
- Occlusions
- Models Setup
- LaFIn
- PCPSFM
- Test Procedure
- Models Execution
- Inpainting Results
- Evaluation on Face Recognition
- KNearest Neighbors Classifier
- Linear SVM Classifier
- VGGFace2 Classifier
- Interpretation of Classification results
Last but not least, we have to take into account that we evaluated the results of inpainting according to the size of the occlusions. Our main purpose is to track the effect of occlusion size in a face painting task. The first, general observation is that as the size of the occlusions increases, the quality of the imaging results decreases.
On the other hand, each of the CRPCA models struggle to make the occlu. In particular, it applies a kind of blur in the occluded area and assumes the skin color of the examined id. This face structure generalization will cause a great deal of confusion in the evaluation methods of the face recognition task, as we will see in the next section.
To quantify and evaluate the quality of the in-painting results, we used a performance indicator in the form of a reconstruction error metric. In the previous chapter, we commented on the quality of the painting results produced by our models, based on the illustrations shown in Figures 4.8 – 4.10. Before going into each evaluator, we should first analyze the basic steps of the face recognition process.
Next, we calculated and stored the face encoding of each of the 800 face images, creating a new database consisting of face encodings. A schematic of the painted image and 8 clear face images of the database is shown in Figure 4.11. Due to the nature of the SVM Classifier, we cannot implement the K-ranked accuracy metric.
Starting from the case of small occlusions, we notice a clear supremacy of the LAFIN model. Although after a 1-1 examination of the models, based on the λ parameter, we realize that PCPFM models are inferior to the CRPCA models. Namely, as the size of the occlusions increases, not only the quality of the inpainting results decreases, but also the efficiency in solving the face recognition task.
CONCLUSION AND FUTURE WORK
This way, the comparison of the two methods will be fair, in contrast to the circumstances of this project, where we used the default version of LaFIn pre-trained on the initial inthewild, aligned CelebA images. In addition, key conclusions about the functionality of the models can be drawn if we cluster our data set by additional occlusion criteria besides size. For example, as analyzed in subsection (4.1.2), we could classify occlusions based on their shape, sparsity, or even the location where they lie in the face image.
Last but not least, there is always a chance to find a better evaluation method, capable of achieving higher accuracy scores for the face recognition task.