Wavelets and Images. Luiz Velho. IMPA Instituto de Matematica Pura e Aplicada Rio de Janeiro, Brazil

Wavelets

and

Images

Luiz Velho

IMPA

Instituto de Matematica Pura e Aplicada Rio de Janeiro, Brazil

Outline

 Image Processing – Image Compression – Digital Painting  Image Analysis – Multiresolution Textures – Image Querying

Image Processing

 Images

– Trivial Geometry

– Complex Attribute Function

 Wavelets – Representation / Operations

f

: [0

;

1]



[0

;

1]

! R  Advantages – Multiscale Model
Resolution – Compact Representation
Approximations – Hierarchical Structures
Fast Algorithms Relation with

Applications in Image Processing

 Image Compression

– Subband Image Coding with Three-tap Pyramids,

Edward H. Adelso n, Eero P. Simoncelli, Picture Coding Symposium, (1990)

– Compressing Still and Moving Images with Wavelets,

Hilton,L., Jawerth, D., Sengupta, N., Multimedia Systems (1994)

 Digital Painting

– Multiresolution Painting and Compositing,

Berman, D., Salesin, D., Siggraph (1994)

– B-Spline Wavelet Painting,

Perlin, K., Velho, L., Siggraph ( 1994)

– Live Paint: Painting with Procedural Multiscale Textures,

Image Analysis

 Problems: – Feature Extraction – Model Inference  Wavelet ! Model – Multiscale Structures – Detection of Singularities – Hierarchical Search  Relation with – Fractals – Edge Detection

Applications in Image Analysis

 Feature Detection

– Corner Detection using Spline Wavelets,

Chan, A, Chui, C., Zh a, J. Liu, Q. SPIE Conference (1991)

 Pattern Matching

– Fast Multiresolution Image Querying,

Jacobs, C., Filkelstein, A., Salesin, D. Siggraph (1995)

 Model Synthesis

– Pyramid-Based Texture Analysis/Synthesis,

Heeger, D., Bergen, J. Siggraph (1995)

IMAGE COMPRESSION

 Image Coherency – Spatial – Spectral  Visual Information – Shade Gradations – Textures – Edges  Compression Method – Eliminate Redundancy – Encode Information  Types of Compression – Lossless – Lossy

Wavelets

 Wavelet Basis

– Support Varies with Scale – Detect Transitions

Properties of Wavelets

for Compression

 Good Localization

– Spatial and Frequency Domain

 Fast Computation

– O

(

N

)

 Exact Decomposition / Reconstruction

– Lossless Compression



n

Vanishing Moments

– Higher Compression Rates

 Rational Coefficients

Basic Scheme

 Compression – FWT – Quantization – Encoder  Decompression – Decoder – Dequantizer – IWT Forward Wavelet

Transform Quantizer Encoder

Inverse Wavelet

Transform DeQuantizer Decoder

Wavelet Transform

 Entropy Coding – Orthogonal – Biorthogonal  Vector Quantization  Adaptive Transform – Wave Packets  Edge Based

Statistical Analysis

 Histogram 0 100 200 300 0 0.005 0.01 0.015 Pixel Value Frequency

Original Lena Image

0 5000 10000 10-6 10-4 10-2 100 Coefficient Value Frequency

Transformed Lena Image

 Information Packing

 Lossless Compression

– Lower Entropy

 Lossy Compression

Image Approximation

 Energy Invariance – Proportional Changes  Mechanisms – Thresholding Function

T

(

t;p

) =

(

0

ifj

p

j

< t

p

otherwise – Quantization Function

Q

(

p

)

Quantization Cells
Quantization Levels

Quantization

 Optimal Quantization

min

jj

I

(

u;v

)

e

I

(

u;v

)

jj  Error Metric – Euclidean (RMS) – Perceptual  Wavelets

– Separate Quantizer for each Scale

 Quantizer Design

– Properties of Visual System

Allocation of Bits

– Statistics of Coefficients

Encoding

 Dictionary of Symbols D  Types of Codes – Fixed Length – Variable Length
Huffman
Arithmetic  Code Structure – Run-Length – Zero-Tree

Example

 32:1

Comparison with JPEG

 Image: Lena 20 22 24 26 28 30 32 34 36 38 40 42 4 8 16 32 64 128 PSNR (dB) Compression Ratio

Comparison of Wavelet Compression Methods

Zerotree JPEG Biorthogonal + VLC Biorthogonal + FLC W6 + VLC W6 + FLC  JPEG - up to 25:1  Wavelet - 30:1 to 100:1

MULTIRESOLUTION PAINTING

 Recent Development

View and Modify the Image at Multiple Resolution Levels

 Advantages

– Quick Changes (large areas) – Unlimited Detail (small areas)

 Applications

– Interactive Paint – Color Correction

– Size Independent Compositing – Variable Resolution Exhibition

System Structure

 The Painting Process

modify image at level

x

# "

move up or down a level

– Operations

Paint: paint on the current view Push: magnify by a factor of two Pop: demagnify the view

 Computational Support

– Multiresolution Image Representation – Painting over Multiple Levels

Need for a Linear Decomposition

 Propagating changes in a MIP Pyramid

L

L - 1

 Propagating changes in a Wavelet Pyramid

L

=

Multiresolution Representation

 Low Pass Pyramid

– Redundant Representation – Used when painting at level j

 Band Pass Pyramid

– Linear Description:

I

=

P

j

d

j

j

+

c

0



0

Painting over Multiple Levels

 Painting

– While painting at level j

Composite

A

OVER

c

j

Accumulate



 Propagation

– When changing level

if moving UP

Bandpass Decomposition

if moving DOWN

Lazy Evaluation Propagate



Painting Process

Main

Paint_init(canvas_size); while not quit do

switch on paint_op: case PAINT: Drop_paint(draw_path, brush); case BRUSH: Set_Brush(attributes); case ZOOM_IN: Paint_change_level(down, cursor_xy); case ZOOM_OUT: Paint_change_level(up, cursor_xy); Drop Paint forall xy in region img[c+xy]=PAINT(img[c+xy],region[xy]); Paint p2.c = p2.c+(1.0-p1.a)*p1.c; p2.a = p2.a+(1.0-p1.a)*p1.a;

Multiresolution Compositing

A

+

c

o

(1



) +

X

j

d

j

P

j

(1



)

 Canonical Case: – Image

A

at level 0 – Mask



of

A

– Band Pass Pyramid

X

j

d

j

j

+

c

0



0

Changing Levels

Paint_change_level(delta, center) {

Put_image(curr_level)

if (delta > 0) /* magnify */ propagate mask to next level

Wavelet_sca_reconstruct(app_cur, t1); Wavelet_det_reconstruct(det_cur, t2); Composite(t1, t2, m_next, app_next); Put_image(next_level);

clear and store mask at curr_level

else /* demagnify */

Wavelet_decompose(app_cur, app_nxt, det_nxt); Put_image(next_level);

compute new window

Get_image(next_level); }

Memory Management

 Virtual Array

– Tree of Pages

 Page

– Pointers to Descendent Pages – Actual Data DATA DATA DATA

...

0 .. N-1 N...N-12

 Virtual 2D Canvas (N =

res



res

)

MULTIRESOLUTION TEXTURES

 Live Paint

– New Tool, designed for MPS – Complex Patterns

– Automatic Generation of Detail

 Procedural Bandpass Pyramid: f (s,u,v)

– Algorithmic Representation – Domain

Spatial
Scale

 Texture Alpha Channel:



f

Texture Refinement Mechanism

 Procedural Multiresolution Compositing

Procedural Band Pass Pyramid

 Components

– Base Procedure (s = 0)

initial appearance

called when the texture is first painted

– Detail Procedure (s ¿ 0)

add detail

Example

Stripe Texture Base(x) { if (x mod 2 = 0) 1 else 0 } Detail(x,l) { t = x mod 2l if (t = 0 or t = 2l-1-1) 1 else if (t = 2l-1 or t = 2l-1) -1 else 0 }

Texture Primitive

 Instance – Id – Type – Base Level  Representation – Procedure: f.type (s, u, v) – Texture Alpha Channel:



_id

Examples

 White

if dlevel = 0 then 1 else 0

 Rock pseudorandom(x,y,dlevel) / 2 dl evel =2  Stripes sqr wave(x) / (

dlevel

+0

:

4) 0:1  Sawtooth

saw wave(x) / (

dlevel

+1) 0:14

 Squares

sqr wave(x) * sqr wave(y) / (

dlevel

+0

:

1) 0:14 sqr wave(x) if i = 0 or i = n/2-1 then 1 else if i = n/2 or i = n-1 then -1 else 0 saw wave(x) if i = 0 then 1 else if i = n-1 then -1 else 0 n := 2 dlevel+1

x

Texture Combination

 Composite Texture

– Vector of Texture Alpha Channels

a

= (

a

1

;a

2

;:::;a

n

)

 Mixture Scheme

– Linear Blend

P

Examples

 Rock, Squares and Stripes

Future Work

 Improve the Framework

– Layers

– Interpreted Code

 User Interface

– Texture Generators

– Environment for Texture Design

 Images

– Texture from Samples – Extrapolation

IMAGE QUERYING

 Problem: Find an Image in a Database

 Types – Query by Content
Keywords – Query by Example
Similarities  Image Match

– Query Image:

Q

[

i;j

]

– Target Image:

T

[

i;j

]

* Distortions

Query Metrics



L

p

Metrics (p = 1, 2) jj

Q;T

jj

p

= (

X

i;j

j

Q

[

i;j

]

T

[

i;j

]

j

p

)

1

=p

– Expensive to Compute – Not Perceptual  Color Histogram – Not Spatial – Color Shifts

Advantages of Wavelets

 Perceptual Metric  Compression – Compact Signature  Edge Detection – Key Features  Multiscale

– Resolution Independence of

Q

and

T

 Fast,

O

(

N

)

Wavelet Metric

 Representation (

M

- wavelet transform)

M

(0

;

0)

Mean

M

[

i;j

]

,

(

i;j

)

6

= 0

: Wavelet Coefficients  Metric

jj

Q;T

jj

=

w

0

;

0

j

Q

[0

;

0]

T

[0

;

0]

j

+

X

i;j

w

i;j

j

Q

[

i;j

]

T

[

i;j

]

j  Simplifying the Metric

– Truncate coefficients:

T

2 R ! e

T

2 f

1

;

0

;

1

g

– Substitute: j

Q

[

i;j

]

T

[

i;j

]

j by

(

Q

[

i;j

]

6

=

T

[

i;j

])

 Rewriting for Fast Computation

P

i;j

(

Q

[

i;j

]

6

=

T

[

i;j

])

! P

i;j

w

k

P

i;j

(

Q

[

i;j

] =

T

[

i;j

])

Q;T

Tuning the Metric

w

_i;j

=

bin

(

i;j

)

 Adapt to a Class of Images : Training Sets

– Scanned – Painted

 Solutions

– Multidimensional Optimization – Logit Statistical Model

 Function

bin

(

i;j

) = min(max(

i;j

)

;

5)

 Weights Painted Scanned b w Y w I w Q w Y w I w Q 0 4:04 15:14 22:62 5:00 19:21 34:37 1 0:78 0:92 0:40 0:83 1:26 0:36 2 0:46 0:53 0:63 1:01 0:44 0:45 3 0:42 0:26 0:25 0:52 0:53 0:14 4 0:41 0:14 0:15 0:47 0:28 0:18

Wavelet Transform

 Haar Wavelet DecomposeImage(I) { for (row = 1 to r) DecompArray(I[row,v]); for (col= 1 to s) DecompArray(I[u,col]); } DecompArray(A[x]) { while (h>1) { h = h/2; for (i=0 to h-1) { B[i] = (A[2*i]+A[2*i+1])/sqrt(2); B[h+i] = (A[2*i]-A[2*i+])/sqrt(2); } } copy(B,A);

Database Structure

 Data

– Values of

T

[0

;

0]

– Signs of Largest

m

Coefs

T

[

i;j

]

 6 Search Arrays

(3 channels, pos/neg)

D

+

c

[

i;j

]

D

c

[

i;j

]

– Entry is a List of Pointers

 Image Info

–

T

[0

;

0]

Computing a Score

ScoreQuery(Q, m) { DecomposeImage(Q); InitScores(scores,0); foreach (color_channel c) { foreach (database_entry T) scores[index(T)]+=w[0]*|Q[0,0]-T[0,0]|; TruncateCoeffs(Q,m); foreach (Q[i,j] != 0) { if (Q[i,j] > 0) append(list, D_plus[i,j]); else append(list, D_minus[i,j]); foreach (element l of list)

scores[index(l)] -= w[bin(i,j)] }

}

Finding a Match

 Rank all Images

– Score Query  Sort Scores – Heap Select  Select Top 20 – Show Thumbnail (5x4)  Match – Visual Inspection

User Interface

 Query Type

– Scan

File

Results

 Comparison  Speed

Metric

Time

n

= 1093

n

= 20

;

558

Lq

0

:

19 0

:

44

L

1 (8 8) 0

:

66 7

:

46

L

2 (8 8) 0

:

68 6

:

39

L

1 (128 128) 47

:

46 892

:

60

L

2 (128 128) 42

:

04 790

:

80

Lc

0

:

47 5

:

03 * Database Size