The KL divergence is always positive and zero only if the variational distribution Q(d) is equal to the true posterior distribution P(d|I). By substituting the equations (2.23) and (2.21) in the equation for KL divergence we get:
KL Q(d)
P(d|I)) = − X
x
X
dx∈L
Qx(dx)Vx(dx,I)
− X
x
X
y∈Nx
X
dx,dy∈L
Qx(dx)Qy(dy)Vx,y(dx, dy)
+ X
x
X
dx∈L
Qx(dx) logQx(dx) (2.25) Finding the marginalsQx(dx) from the above equation is not straight-forward and requires iterative re-substitution. As theQ(d) is fully factorized (2.23), the above equation can be optimized one marginal component at a time.
The update equations (also the fixed point equations) are derived by minimizing the KL divergence (2.25) with respect to each marginalQx(dx)inQ(d)while keeping the remaining marginals fixed. This is equivalent to taking the partial derivative ∂Q∂KL(·)
x(dx) and setting it to zero. It can be easily verified that, the minimization of (2.25) with respect marginals Qx(dx) leads to the following:
Qx(dx)← 1 Zexp
−
Vx(dx,I) + X
y∈Nx
X
dy∈L
Qy(dy)Vx,y(dx, dy)
(2.26) These update equations (collectively for all x) are also referred to asMean Field equations.
The normalizationZ ensures thatP
dx∈LQx(dx) = 1. Successive application of the updates correspond to iteratively enforcing different Mean Field equations. These equations indicate that the updates are made “locally” by averaging the neighbours with respect to the other component distributions. In other words, eachxonly sees the “mean effect” of the neighbours Nx.
Yuille et al. [1990] were among the first to use Mean Field Approximation in the context of stereo. They formulated their energy function on psycho-physical grounds, incorporating discontinuities and the matching of different primitives instead of just intensities. However, as mean-field approximates the true distribution with a much simpler form with marginals over a single variable, it does not provide a good approximation of the true posterior. More complicated forms for Q(·) can be used instead of the fully factorised version in (2.23) (Wainwright and Jordan [2005] and Jaakkola [2000]). Nevertheless, Strecha et al. [2006]
used this approximation within an Expectation-Maximization framework, estimating both disparities and occlusions, and showed results comparable to the state-of-the-art. Mean Field approximation can also be interpreted as parallel message-passing algorithm, in the spirit of message passing algorithms such as Belief Propagation. Here, each site x sends a message Qx(dx) to its neighbours, which is in turn based on the message it received from its neighbours in the previous time step.
2.3.2 Belief Propagation
Belief Propagation (BP) was first introduced by Pearl [1986], and is best understood in the context of probabilistic graphical models (See Section 2.2.2.1). BP is part of the most popular class of algorithms called the message passing algorithms, where information or
“message” is passed from one site to another along the edges of a graph until convergence.
The BP is exact on trees (graphs with no cycles), but in graphs with cycles, like the MRF shown in figure2.4, there are no guarantees on convergence.Surprisingly, despite these issues, BP provides good approximate results on such cyclic graphs. In particular good results are obtained when BP is applied to stereo matching, where the graph G (See section 2.2.2.1) with cycles represents the MRF model. When the BP is applied to graphs with cycles or loops, it is often referred to as Loopy Belief Propagation (LBP)3.
In order to explain the message passing in LBP algorithm we define the following nota- tions: bx(dx) is the belief at the site x, and is equivalent to the approximate distribution Qx(dx)in the previous sectionmx,y(dy)is the message sent by site xto yas to what state the variable dy should be in. Note that both bx(dx) and mx,y(dy) are vectors of length L corresponding to the number of states in the set L = d1, d2, . . . dk, . . . , dL. To improve readability, we re-write the posterior distribution in (2.21) as follows:
P(d|I)∝Y
x
ψx(dx,I)Y
x
Y
y∈Nx
φx,y(dx, dy) (2.27) whereψx(dx,I) = exp −Vx(dx,I)
andφx,y(dx, dy) = exp −Vx,y(dx, dy)
. The belief at every site x is proportional to the local evidence at that position, in our case the data or the likelihood termψx(dx,I) and to all the messages coming in to thex:
bx(dx) =k ψx(dx,I) Y
y∈Nx
my,x(dx) (2.28)
where k is the normalizing constant ensuring that P
dx∈Lbx(dx) = 1. The messages are updated iteratively as follows:
mx,y(dy)← max
dx∈L ψx(dx,I)φx,y(dx, dy) Y
z∈Nx\y
mz,x(x) (2.29)
The right-hand-side of the above equation takes into account the data term, the interaction term and all the messages that are coming into x, except the one from y. The updates for the two equations are diagrammatically shown in the the figure 2.5. These equations represent what is called as the max-productBP. If the max-operation in (2.29) is replaced by a sum it becomessum-productBP. While, the max-product BP algorithm finds the MAP estimate or minimum energy associated with the MRF, the sum-product algorithm can be used to approximate the posterior probabilityp(d|I)of each label (in our case disparity) for each pixel.
3. Note that in this thesis we use LBP and BP interchangeably.
x y
mxy =
(b) (a)
myx 1
my2x
myx 3
myx 4
ψx my1x
my3x
my2x x y
ψx
Figure 2.5: Update equations for Belief Propagation (a) shows diagrammatically the equa- tion (2.28) where belief at x(shown in red) is calculated taking into account the messages from the neighbours (myx) and the dataψxand (b) shows the equation (2.29) the message mxy passed fromxtoy(shown in red). mxy includes messages passed from the neighbours inNx to xand the evidenceψx.
The equations (2.28) and (2.29) merely provide the update equations for BP. The princi- ple of BP, however, can be explained using variational methods. Yedidia et al. [2003] showed that fixed points of the BP (update equations) actually, correspond to the minimum of the Bethe approximation, and to the global minimum if there are no loops. Unlike the Mean Field approximation, where only the single site distributions are considered (2.23), the Bethe approximation considers both pair-wise and single site distributions. A more detailed and in-depth comparison of the Mean Field and Belief Propagation is provided in the paper by Weiss [2001]. He found that despite its non-convergence, BP finds better local minima than Mean Field approaches.
In the context of stereo matching, LBP was first used by Sun et al. [2003] to com- pute disparities. However, if used directly this method is still computationally demanding.
Felzenszwalb and Huttenlocher [2006], therefore, provide methods for speeding up the LBP approach. In particular, Felzenszwalb and Huttenlocher suggested the use of a multi-grid approach where the computations are performed in a coarse to fine manner, thus providing substantial speed up in stereo matching application. A large number of variants of BP have also been developed over the years, among which the sequential-tree re-weighted BP (sequential-TRW) by Kolmogorov [2006] has been noticeable. This is because sequential- TRW achieves lower bound on energy, thus ensuring convergence. A comparison of sum- product, max-product and sequential-TRW in context of stereo and other vision applications such as photomontage and binary image segmentation is done by Szeliski et al. [2008]. This
comparative study showed that the sequential-TRW consistently provided better results than both forms of LPB.
2.3.3 Graph Cuts
Graph Cuts from combinatorial optimization can be used to minimize the MRF energy function (2.22). Graph Cuts include the max-flow/min-cut algorithms in combinatorial optimization. These techniques were first used in computer vision by Greig et al. [1989] in the context of binary image restoration. Greig et al. showed that global minimum of the energies of the form (2.22) could be achieved when the number of labels is limited to two (binary labelling). Graph Cuts, while widely used in computer vision applications, are in general limited to only binary labelling problems.
In order to apply Graph Cuts to multi-label problems (such as stereo correspondence), Ishikawa [2003] transformed the graph to an equivalent binary problem. However, in their framework the interaction functions were constrained to be convex. The problem with having convex interaction function is that the resulting method performs poorly at the disparity discontinuities. Alternatively, Boykov et al. [2001] proposed to apply the Graph Cuts repeatedly on pairs of labelling within their inner-loops. They suggested two graph- cut algorithms, namely expansion move and swap move algorithms. For a label α ∈ L, expansion move allows any set of sites to change their labels to α. The local minimum is found, if no expansion move for any labelαyields a lower energy. The swap move algorithm interchanges the labels of some subset of sites labelled α to β and vice versa. Similar to expansion move the swap move finds the local minimum such that no other swap move will produce a lower energy. These two algorithms produce very stable minima, and allow the interaction function to be metric in the case of expansion move and semi-metric in case of swap move algorithms. Kolmogorov and Zabih [2002b] showed further classes of interaction functions that could be considered within the Graph Cuts framework.
The expansion and swap move algorithms have been applied to stereo correspondence problem by Boykov et al. [2001], Kolmogorov and Zabih [2001] and Kolmogorov and Zabih [2002a] with some success. The comparative study done by Tappen and Freeman [2003] of BP and Graph Cuts showed that the two algorithms produced comparable results under identical parameter settings. Tappen and Freeman, however, used the simple Potts model for comparing the two algorithms. The extended comparison of BP, Graph Cuts and sequential- TRW by Szeliski et al. [2008] showed that sequential-TRW and expansion move algorithms performed the best for stereo matching. While they compared the algorithms based on the energy, they found that some energies calculated were lower than that of the ground-truth.
This illustrates the need for more accurate modelling of the problem itself. The figure 2.6 shows the output disparity map obtained by using Mean Field, standard-BP and Graph- cut on the map image from the Middlebury database. A coarse to fine strategy was used in case of both Mean Field and BP based on the method suggested by Felzenszwalb and Huttenlocher [2006]. As can be see the performance on this simple image is comparable in
all the three cases.
Figure 2.6: (a) Shows Left image of the stereo pair map. (b) the disparity ground-truth.
(c), (d) (e) and (f) show the disparity maps estimated using Mean Field, BP, Graph Cuts and Dynamic Programming respectively. (f) shows the “streaking” caused by the Dynamic Programming algorithm (discussed in section2.3.4).
2.3.4 Other methods
Among other discrete optimization methods to minimize (2.22, page 24) is Dynamic Programming (DP) (Birchfield and Tomasi [1999a], Ohta and Kanade [1985], Torr and Criminisi [2004]). While (2.22) is defined on a2D-grid, the DP algorithm finds the global minimum for independent scanlines (owing to the epipolar constraint). However, when used in stereo matching DP suffers from “streaking” effect (See figure2.6(f)). This is because of the difficulty in enforcing the inter-line consistency. Moreover, DP makes use of the ordering constraint, which may not always be true. Kolmogorov et al. [2006] suggested an extension calledlayeredDP to overcome the problem interline consistency.
Alternatively, the stereo problem can be expressed in continuous form with a continuous disparity range. In such cases variational methods like the one suggested by Alvarez et al.
[2000] can be used to minimize the energy functional. Alvarez et al. derive the Euler-
Lagrange equations, which are partial differential equations (PDE), whose solutions are the fixed points of the energy functional. They seek the solution of these underlying PDEs using a gradient descent method. In order to reduce the risk of being trapped in some irrelevant local minima during the iterations, the authors use a focusing strategy based on a linear scale-space. Faugeras and Keriven [1998] suggested the use of Level-Set methods to deform an initial set of surfaces obtained the Euler-Lagrange PDEs. Convex programming methods were also suggested by Miled and Pesquet [2006] and Pock et al. [2008]. Bhusnurmath and Taylor [2008] used the interior point method to solve the matching problem by approximat- ing the data term with a convex function and using Laplacian terms for interaction. While these methods allow computation of sub-pixel disparities directly, they suffer from a major drawback in that they require the energy functional to be convex and require the minimiza- tion to be carried out on a convex set. This means that the final disparities produced will be smooth across the boundaries. It is also more difficult to introduce occlusion handling, given the restrictions on the energy function. Furthermore, methods using PDEs require discretization which is not straightforward owing to numerical instability.
Another technique which allows the direct measurement of sub-pixel disparities is the phase-based technique (Sanger [1988], Fleet et al. [1991]). This method finds a solution to the correspondence problem by using the differences in the phase of local spatial frequency components. Odd and even one-dimensional spatial Gabor filters, at different spatial fre- quencies are convolved with the stereo pair. The difference between the phase at corre- sponding points in the two images is used to find the stereo disparity. Two main limitations of this method are: (1) the maximum disparity is limited by the filter width, therefore al- lowing only small disparity range in practice. (2) phase information is very sensitive to local image characteristics. This is mainly because of singularities in the phase-signal, which are often by caused textureless regions in the images.
Finally there are methods like the one suggested Zitnick and Kanade [2000] which do not fall into the category of either global or local methods. Zitnick and Kanade’s algorithm is variant of Marr and Poggio [1976]’s and is inspired by human stereo vision. This algo- rithm performs local computations by imposing constraints on uniqueness of matches and smoothness. Furthermore, it uses a3D support window to allow for slanting surfaces. While the computations are carried out locally, the optimization is done iteratively using nonlinear operations. This results in a behaviour similar to that of global optimization algorithms.
As stated by Szeliski et al., while there are number of methods to optimize, the main requirement is actually to have an energy which is representative of the scene. However, having an accurate function is not necessarily good as it may be too complex to optimize. As result there is trade-off on what a good energy is and how this function could be optimized.
We will now continue our discussion how the model can be improved by incorporating additional cues.