Graphical models, from graphs to trees (and back)

Graphical models, from graphs to trees (and back)

Andrea Montanari

Stanford University

April 14, 2008

Outline

1 What is this talk about, and why should one care

2 Uniform decorrelation

3 Non-Uniform decorrelation

4 Trees vs graphs: from reconstruction to pure states

Outline

1 What is this talk about, and why should one care

2 Uniform decorrelation

3 Non-Uniform decorrelation

4 Trees vs graphs: from reconstruction to pure states

Outline

1 What is this talk about, and why should one care

2 Uniform decorrelation

3 Non-Uniform decorrelation

4 Trees vs graphs: from reconstruction to pure states

Outline

1 What is this talk about, and why should one care

2 Uniform decorrelation

3 Non-Uniform decorrelation

4 Trees vs graphs: from reconstruction to pure states

What is this talk about, and why should one care

Graphical models

x ₁

x ₂ x ₃ x ₄

x 5

x 6

x ₇ x ₈

x 9

x ₁₀

x 11

x 12

G = (V , E ), V = [n], x = (x ₁ , . . . , x _n ), x _i ∈ X

1 Y

This talk

1. G has bounded degree (on average).

2. G has girth `(n) → ∞ (apart from o(n) vertices).

3. G is random.

This talk

1. G has bounded degree (on average).

2. G has girth `(n) → ∞ (apart from o(n) vertices).

3. G is random.

This talk

1. G has bounded degree (on average).

2. G has girth `(n) → ∞ (apart from o(n) vertices).

3. G is random.

Example 1: q-coloring

G = (V , E ) graph.

x = (x ₁ , x ₂ , . . . , x _n ), x _i ∈ {1, . . . , q} variables

Example 1: q-coloring

G = (V , E ) graph.

x = (x ₁ , x ₂ , . . . , x _n ), x _i ∈ {1, . . . , q} variables

Uniform measure over proper colorings

µ(x) = 1 Z

Y

(i,j)∈E

ψ(x _i , x _j ) , ψ(x , y) = I (x 6= y) .

Example 2: k -satisfiability

n variables: x = (x 1 , x 2 , . . . , x n ), x i ∈ {0, 1}

m k -clauses

(x 1 ∨ x 5 ∨ x 7 ) ∧ (x 5 ∨ x 8 ∨ x 9 ) ∧ · · · ∧ (x 66 ∨ x 21 ∨ x 32 )

Uniform measure over solutions

x ₃ x ₁

x ₆ x ₄

x 2

x ₅

x ₇ x

x

x

8

9

10

← variables x i ∈ {0, 1}

← clauses, e.g. (x ₅ ∨ x ₇ ∨ x ₉ ∨ x ₁₀ )

F = · · · ∧ (x

∨ x

∨ · · · ∨ x

)

| {z }

a-th clause

∧ · · ·

M

Many other examples

Communications/signal processing (technologically relevant) . . .

Probability, physics, computer science, information theory,. . .

Many other examples

Communications/signal processing (technologically relevant) . . .

Probability, physics, computer science, information theory,. . .

Is this motivating enough? (a personal view)

Emerging conceptual unity:

Approximation of sparse graph models by trees.

. . .

[cf. Aldous’ local weak convergence]

Is this motivating enough? (a personal view)

Emerging conceptual unity:

Approximation of sparse graph models by trees.

. . .

[cf. Aldous’ local weak convergence]

Is this motivating enough? (a personal view)

Emerging conceptual unity:

Approximation of sparse graph models by trees.

. . .

[cf. Aldous’ local weak convergence]

Is this motivating enough? (a personal view)

Emerging conceptual unity:

Approximation of sparse graph models by trees.

. . .

[cf. Aldous’ local weak convergence]

Uniform decorrelation

Random k -satisfiability

Each clause is uniformly random among the 2 ^{k n} _k

possible ones.

n → ∞, m = αn

Random k -satisfiability

Each clause is uniformly random among the 2 ^{k n} _k

possible ones.

n → ∞, m = αn

Number of ‘good’ truth assignments

Z n (β) = X

x

exp

− 2β #[clauses violated by x]

Theorem (Montanari, Shah, 2007)

If α < α u (k) = (2 log k)k ⁻¹ [1 + o k (1)] then 1

n log Z _n (β ) −→ ^a.s. φ(α, β) ,

where . . .

. . . where. . .

φ(α, β) = −kαElog[1 + tanhhtanhu] +αElog 8

<

: 1− 1

2^k(1−e^−β) k Y i=1

(1−tanhh_i) 9

=

; +

+Elog 8

><

>:

`+ Y i=1

(1 + tanhu_i⁺)

`− Y i=1

(1−tanhu_i⁻) +

`+ Y i=1

(1−tanhu_i⁺)

`− Y i=1

(1 + tanhu_i⁻) 9

>=

>; ,

and h, u are the unique solution of

h = ^d

l

X

a=1

u a −

l

X

b=1

u _b ⁰ , u = ^d f β (h 1 , . . . , h k−1 ) .

[Conjectured by Monasson, Zecchina 1999]

Proof: 1st preliminary remark

Sufficient to prove 1

n E log Z _n (β) −→ φ(α, β) ,

Proof: 2nd preliminary remark

The tree ensemble T(`), T(∞)

Poisson(kα/2) Poisson(k α/2)

Proof: 2nd preliminary remark

The tree ensemble T(`), T(∞)

Poisson(kα/2) Poisson(k α/2)

Proof: 2nd preliminary remark

The tree ensemble T(`), T(∞)

Poisson(kα/2) Poisson(k α/2)

Proof: 2nd preliminary remark

The tree ensemble T(`), T(∞)

Poisson(kα/2) Poisson(k α/2)

Proof: 2nd preliminary remark

The tree ensemble T(`), T(∞)

Poisson(kα/2) Poisson(k α/2)

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Write ^d _dβ log Z _n (β) in terms of local expectations. ????

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Write ^d _dβ log Z _n (β) in terms of local expectations. ????

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Write ^d _dβ log Z _n (β) in terms of local expectations. ????

Local expectations ????

x ₃ x ₁

x ₆ x ₄

x 2

x ₅

x ₇ x

x

x

8

9

10

← variables x i ∈ {0, 1}

← clauses, e.g. (x ₅ ∨ x ₇ ∨ x ₉ ∨ x ₁₀ )

µ(x) = 1 Z n (β)

m

Y

a=1

ψ _β,a (x _i

_(a) , . . . , x _i

_(a) ) ψ _β,a ( · · · ) =

1 if clause a is satisfied

e ^−2β otherwise

d

dβ log Z _n (β) = −2

M

X

a=1

µ(clause a is not satisfied)

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Express ^d _dβ log Z _n (β) in terms of local expectations.

3. Prove that local expectations on G converge to expectations on T(∞).

4. Show ^dφ(α,β) _dβ is equal to the same expectations on T(∞).

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Express ^d _dβ log Z _n (β) in terms of local expectations.

3. Prove that local expectations on G converge to expectations on T(∞).

4. Show ^dφ(α,β) _dβ is equal to the same expectations on T(∞).

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Express ^d _dβ log Z _n (β) in terms of local expectations.

3. Prove that local expectations on G converge to expectations on T(∞).

4. Show ^dφ(α,β) _dβ is equal to the same expectations on T(∞).

Proof strategy

1. Check it for β = 0: φ(α, 0) = log 2 = n ⁻¹ E log Z n (0).

2. Express ^d _dβ log Z _n (β) in terms of local expectations.

3. Prove that local expectations on G converge to expectations on T(∞).

4. Show ^dφ(α,β) _dβ is equal to the same expectations on T(∞).

Convergence to tree values

z ₁ z ₂ z ₃

` T(`)

r

T(∞) infinite k -SAT tree T(`) first ` generations

µ <sup>`,z</sup> ( · ) Boltzmann measure on T(`) boundary condition z

µ ^`,z r ( · ) root variable marginal

Convergence to tree values

z ₁ z ₂ z ₃

` T(`)

r

T(∞) infinite k -SAT tree T(`) first ` generations

µ <sup>`,z</sup> ( · ) Boltzmann measure on T(`) boundary condition z

µ ^`,z r ( · ) root variable marginal

Convergence to tree values

z ₁ z ₂ z ₃

` T(`)

r

T(∞) infinite k -SAT tree T(`) first ` generations

µ <sup>`,z</sup> ( · ) Boltzmann measure on T(`) boundary condition z

µ ^`,z r ( · ) root variable marginal

Uniform decorrelation (Gibbs measure uniqueness)

E

max

z(1),z(2)

||µ ^t,z(1) _r ( · ) − µ ^t,z(2) _r ( · )||

→ 0 .

‘Easy’ sufficient condition

True only at very small α

(α _u (k) ' (2 log k)/k , conjecture up to ' 2 ^k log 2)

Uniform decorrelation (Gibbs measure uniqueness)

E

max

z(1),z(2)

||µ ^t,z(1) _r ( · ) − µ ^t,z(2) _r ( · )||

→ 0 .

‘Easy’ sufficient condition

True only at very small α

(α _u (k) ' (2 log k)/k , conjecture up to ' 2 ^k log 2)

Uniform decorrelation (Gibbs measure uniqueness)

E

max

z(1),z(2)

||µ ^t,z(1) _r ( · ) − µ ^t,z(2) _r ( · )||

→ 0 .

‘Easy’ sufficient condition

True only at very small α

(α _u (k) ' (2 log k)/k , conjecture up to ' 2 ^k log 2)

Non-Uniform decorrelation

Ferromagnetic Ising model

Ferromagnetic Ising model

G n = (V n ≡ [n], E n ) x _i ∈ {+1, −1}

µ(x) = 1

Z n (β, B) exp







β X

(ij )∈E

x _i x _j + B X

i

x _i







[Johnston, Plech´ ac 1998, Leone et al 2004]

Ferromagnetic Ising model

G n = (V n ≡ [n], E n ) x _i ∈ {+1, −1}

µ(x) = 1

Z n (β, B) exp







β X

(ij )∈E

x _i x _j + B X

i

x _i







[Johnston, Plech´ ac 1998, Leone et al 2004]

Free energy density

Theorem (Dembo, Montanari 2008) If G n converges locally to T(P ), then

1

n log Z n (β, B) −→ ^a.s φ(P , β, B ) .

where. . .

Free energy density

Theorem (Dembo, Montanari 2008) If G n converges locally to T(P ), then

1

n log Z n (β, B) −→ ^a.s φ(P , β, B ) .

where. . .

φ(P , β, B)

For B ≥ 0, let θ ≡ tanh β, h ⁽⁰⁾ > 0, and h ^(`+1) = tanh ^d

B +

K−1

X

i=1

atanh(θ h _i ^(`) ) ,

Then h ^(`) →

h ^∗ and

φ(P, β, B) ≡ log cosh B + P

2 log cosh β − P

2 E log(1 + θh ₁ ^∗ h ₂ ^∗ )+

+ E log (

(1 + tanh B)

L

Y

i=1

(1 + θh _i ^∗ ) + (1 − tanh B)

L

Y

i=1

(1 − θh _i ^∗ ) )

.

φ(P , β, B)

For B ≥ 0, let θ ≡ tanh β, h ⁽⁰⁾ > 0, and h ^(`+1) = tanh ^d

B +

K−1

X

i=1

atanh(θ h _i ^(`) ) ,

Then h ^(`) →

h ^∗ and

φ(P, β, B) ≡ log cosh B + P

2 log cosh β − P

2 E log(1 + θh ₁ ^∗ h ₂ ^∗ )+

+ E log (

(1 + tanh B)

L

Y

i=1

(1 + θh _i ^∗ ) + (1 − tanh B)

L

Y

i=1

(1 − θh _i ^∗ ) )

.

T(P , `)

P _k

ρ _k

T(P , `)

P _k

ρ _k

T(P , `)

P _k

ρ _k

T(P , `)

P _k

ρ _k

T(P , `)

P _k

ρ _k

‘Converges locally’

P ≡ {P _k } _k≥0 Degree distribution

T(P , `) `-generations Galton-Watson tree

B i (`) Ball of radius ` around uniformly random node

Definition

G _n converges locally to T(P ) if uniform bound on the edge number distribution and, for any `,

B _i (`) converges in distribution to T(P , `).

Why non-uniform control? Phase transition. . .

For β > β _c ≡ atanh(1/ρ)

lim

B →0+ lim

n→∞ E i hx _i i = − lim

B→0− lim

n→∞ E i hx _i i > 0

. . . and its tree counterpart

z ₁ z ₂ z ₃

` T(`)

r

z = (+1, +1, . . . , +1) ⇒ lim

`→∞ hx <sub>r</sub> i <sub>`</sub> > 0 z = (−1, −1, . . . , −1) ⇒ lim

`→∞ hx <sub>r</sub> i <sub>`</sub> < 0

A different case: Ising spin glass

µ(x) = 1 Z exp







β X

(ij )∈E

J _ij x _i x _j + B X

i

x _i





 J _ij ∈ {+1, −1} uniformly random

[Viana, Bray 1985]

[Other approaches: Talagrand 2001, Guerra, Toninelli 2003]

[Approximation by trees: Montanari, Gerschenfeld, in preparation]

Trees vs graphs: from reconstruction to pure states

Alice and Bob

Alice, Bob and G

root

Exit Bob

root

Alice samples a proper coloring (uniformly). . .

root

. . . and hides a ball B(root, t)

root

Bob. . .

root

?

. . . guesses right!

root

!

The problem

Does Bob have a chance?

Formally

X = {X _i : i ∈ V } uniformly random proper coloring.

µ _U ( · |G ) distribution of X _U ≡ {X _i : i ∈ U ⊆ V } B(r, t ) = {i ∈ V : d (i, r ) ≥ t}

Definition

The reconstruction problem is solvable for the sequence of random rooted graphs G _n = (V _n = [n], E _n ) if for some ε > 0,

||µ _{r ,B(r,t)} ( · , · |G _n ) − µ r ( · |G _n )µ _{B(r ,t)} ( · |G _n )||

≥ ε ,

with positive probability (bounded away from 0 as n → ∞).

When G =Tree

→ Bleher, Ruiz, Zagrebenov (1995): Ising model on b-ary trees

→ Evans, Kenyon, Peres, Schulman (2000): Ising on general trees

→ Mossel, Peres (2003): Non binary variables

→Brightwell, Winkler (2004), Martin (2004): Independent sets.

→Chayes et al. (2006): Asymmetric Ising.

Pure states decomposition in q-COL

γ _d (q) γ c (q) γ s (q)

[Biroli, Monasson, Weigt 2001]

[M´ ezard, Parisi, Zecchina 2003]

[Achlioptas, Ricci 2007]

[Krzakala, Montanari, Ricci, Semerjian, Zdeborova 2007]

Pure states decomposition in q-COL

Conjecture (M´ ezard, Montanari, 05)

γ _d (q) =

= Multiple pure states

= Graph reconstruction threshold

= Tree reconstruction threshold

A general sufficient condition

Theorem (Gerschenfeld, Montanari, 2007) If µ( · |G ) is roughly spherical then

Graph solvable ⇔ Tree solvable.

If µ( · |G ) is not roughly spherical then

Graph reconstruction is solvable

A general sufficient condition

Theorem (Gerschenfeld, Montanari, 2007) If µ( · |G ) is roughly spherical then

Graph solvable ⇔ Tree solvable.

If µ( · |G ) is not roughly spherical then

Graph reconstruction is solvable

A general sufficient condition

Theorem (Gerschenfeld, Montanari, 2007) If µ( · |G ) is roughly spherical then

Graph solvable ⇔ Tree solvable.

If µ( · |G ) is not roughly spherical then

Graph reconstruction is solvable

Roughly spherical???

X i ∈ {0, 1}.

X ⁽¹⁾ = {X _i ⁽¹⁾ }, X ⁽²⁾ = {X _i ⁽²⁾ } independent with distribution µ( · |G _n )

X ⁽¹⁾

X ⁽²⁾

(1) (2)

Roughly spherical???

X i ∈ {0, 1}.

X ⁽¹⁾ = {X _i ⁽¹⁾ }, X ⁽²⁾ = {X _i ⁽²⁾ } independent with distribution µ( · |G _n )

X ⁽¹⁾

X ⁽²⁾

µ( · |G _n ) is roughly spherical if d (X ⁽¹⁾ , X ⁽²⁾ ) ≈ n/2 with high

probability.

Can you check this condition?

Theorem

1. q-coloring, γ < (q − 1) log(q − 1): roughly spherical.

2. Ising spin glass 2γ(tanh β) ² < 1: roughly spherical.

3. Ising ferromagnet: not roughly spherical

[Tree reconstruction threshold

1. Bhatayangar, Vera, Vigoda 2008, Sly 2008

2. Evans, Kenyon, Peres, Schulman 2000

3. Tree6=Graph ]

Can you check this condition?

Theorem

1. q-coloring, γ < (q − 1) log(q − 1): roughly spherical.

2. Ising spin glass 2γ(tanh β) ² < 1: roughly spherical.

3. Ising ferromagnet: not roughly spherical

[Tree reconstruction threshold

1. Bhatayangar, Vera, Vigoda 2008, Sly 2008

2. Evans, Kenyon, Peres, Schulman 2000

3. Tree6=Graph ]

Conclusion

Combinatorics/Probability problems on random sparse graphs.

Unifying approach: approximation by trees.

Naturally leads to analytc questons.

Conclusion

Combinatorics/Probability problems on random sparse graphs.

Unifying approach: approximation by trees.

Naturally leads to analytc questons.

Conclusion

Combinatorics/Probability problems on random sparse graphs.

Unifying approach: approximation by trees.

Naturally leads to analytc questons.