Visualizing More Performance Data Than What Fits on Your Screen

(1)

Visualizing More Performance Data Than What Fits on Your Screen

Lucas Mello Schnorr (LIG – CNRS)

6^th International Parallel Tools Workshop Stuttgart, Germany

September 25th, 2012

1/ 30

(2)

Challenges and Motivation

Supercomputers today

– with very large applications

Sequoia (IBM BlueGene/Q) with1,572,864cores (#1 - Top500 - June/2012)

→Sequoia (4 threads per core):6 millionsthreads

Space/Time trace size explosion

Many entities in space + Very detailed behavior in time

Performance Visualization

→How to keep the representation useful on scale?

→Considering the limited screen space available

2/ 30

(3)

Challenges and Motivation

Supercomputers today – with very large applications

Space/Time trace size explosion

Performance Visualization

2/ 30

(4)

Challenges and Motivation

Space/Time trace size explosion

Performance Visualization

2/ 30

(5)

Challenges and Motivation

Space/Time trace size explosion

Performance Visualization

2/ 30

(6)

Challenges and Motivation

Space/Time trace size explosion

Performance Visualization

2/ 30

(7)

Trace visualization

Real BOINC availability trace file

→Availability is either true or false One volunteer client

GNUPlot to a vector file: 8-month and 12-day zoom

3/ 30

(8)

Trace visualization

Real BOINC availability trace file

→Availability is either true or false One volunteer client

GNUPlot to a vector file: 8-month and 12-day zoom

3/ 30

(9)

Trace visualization – trust the rendering?

Acroread

Evince

Same vector file, two different views

→Should we trust the rendering?

4/ 30

(10)

Trace visualization – trust the rendering?

Evince

Same vector file, two different views

4/ 30

(11)

Trace visualization – trust the rendering?

Evince

Same vector file, two different views

4/ 30

(12)

Space/Time views

Widespread, useful, intuitive, fast adoption All trace events represented, causal order

Paje

http://paje.sf.net Vite

http://vite.gforge.inria.fr

Vampir http://vampir.eu

However...

Also impacted by ever larger trace sizes Limitedvisualization scalability

5/ 30

(13)

Space/Time views

Widespread, useful, intuitive, fast adoption All trace events represented, causal order

Paje

http://paje.sf.net Vite

http://vite.gforge.inria.fr

Vampir http://vampir.eu

However...

Also impacted by ever larger trace sizes Limitedvisualization scalability

5/ 30

(14)

Space/Time views – limitations

MPI Sweep3D – 16 processes

Simulated with SMPI, traced with SimGrid

→pj_dump’ed to a csv file, loaded into R Gantt-charts by R, dumped to a vector file

6/ 30

(15)

Space/Time views – limitations

Evince

6/ 30

(16)

Space/Time views – limitations

Ghostscript

6/ 30

(17)

Space/Time views – limitations

MPI Sweep3D – 16 processes

Real execution on the Griffon cluster of Grid’5000

→traced with TAU, converted withtau2paje Once again, Gantt-charts by R→vector file

7/ 30

(18)

Space/Time views – limitations

Evince

7/ 30

(19)

Space/Time views – limitations

Acroread

7/ 30

(20)

Space/Time views – closer look (ViTe tool)

Trust the OpenGL rendering, no data aggregation

Source: http://vite.gforge.inria.fr

8/ 30

(21)

Space/Time views – closer look (new Pajé)

Trust the rendering^→withoutor with OpenGL

9/ 30

(22)

Space/Time views – closer look (new Pajé)

Trust the rendering^→without orwithOpenGL

9/ 30

(23)

Space/Time views – closer look (old Pajé)

Opaque aggregating filter (no user interaction)

→Slashed rectangles represent time-integrated states Self-configure depending on temporal zoom

Source: http://paje.sourceforge.net

10/ 30

(24)

Space/Time views – closer look (old Pajé)

Space dimension: one process per vertical pixel

→at best, 1000 process represented at the same time

11/ 30

(25)

Data aggregation for trace visualization

Always present (in different forms)

→with large traces

Three groups

Implicit Data Aggregation

→uncontrolled, no visual feedback Explicit Data Aggregation

→analyst control + visual feedback Forbidden Data Aggregation

→detect the amount of data, forbids overview

Tools might be classified in more than one group

→multiple approaches depending on data dimension Main objective→Explicit Data Aggregation

12/ 30

(26)

Data aggregation for trace visualization

Three groups

→uncontrolled, no visual feedback

Explicit Data Aggregation

→detect the amount of data, forbids overview Tools might be classified in more than one group

12/ 30

(27)

Data aggregation for trace visualization

Three groups

→analyst control + visual feedback

Forbidden Data Aggregation

12/ 30

(28)

Data aggregation for trace visualization

Three groups

→detect the amount of data, forbids overview

Tools might be classified in more than one group

12/ 30

(29)

Data aggregation for trace visualization

Three groups

→multiple approaches depending on data dimension

Main objective→Explicit Data Aggregation

12/ 30

(30)

Data aggregation for trace visualization

Three groups

12/ 30

(31)

So, why controlled data aggregation?

Technical need: too much data to fit on small screens

Data aggregation→keyfor large-scale visualization Semantic: aggregated data→moremeaningful

13/ 30

(32)

So, why controlled data aggregation?

Data aggregation→keyfor large-scale visualization

Semantic: aggregated data→moremeaningful

13/ 30

(33)

So, why controlled data aggregation?

13/ 30

(34)

So, why controlled data aggregation?

Goal: Visualization techniques for aggregated traces

13/ 30

(35)

Outline

1 Multi-Scale Data Aggregation

2 Visualization techniques Squarified Treemap View Hierarchical Graph View 3 Tools and framework

4 Conclusion

14/ 30

(36)

Multi-Scale Data Aggregation

15/ 30

(37)

Multi-Scale Data Aggregation

In short F_Γ,∆ :







R × T →R (r,t) 7→

Z Z

NΓ,∆(r,t)

ρ(r⁰,t⁰).dr⁰.dt⁰

time space

Method requires dimension ordering

→Time scale is trivial – what about space scale? Additional scales could be added

Some Examples

time space

16/ 30

(38)

Multi-Scale Data Aggregation

In short F_Γ,∆ :







R × T →R (r,t) 7→

Z Z

NΓ,∆(r,t)

time space

→Time scale is trivial – what about space scale?

Additional scales could be added Some Examples

time space

16/ 30

(39)

Multi-Scale Data Aggregation

In short F_Γ,∆ :







R × T →R (r,t) 7→

Z Z

NΓ,∆(r,t)

time space

Additional scales could be added

Some Examples

time space

16/ 30

(40)

Multi-Scale Data Aggregation

In short F_Γ,∆ :







R × T →R (r,t) 7→

Z Z

NΓ,∆(r,t)

time space

Additional scales could be added Some Examples

time space

16/ 30

(41)

Multi-Scale Data Aggregation – Time

1 Time interval defined during the analysis

2 Summary of events for each monitored entity

B A C E D

Blocked Execution

9 seconds

Time-integrated summary for processes

Numbers are in seconds (Execution, Blocked)

B=(7,2)

A=(4,5) C=(3,6) D=(9,0) E=(4,5)

17/ 30

(42)

Multi-Scale Data Aggregation – Time

1 Time interval defined during the analysis

2 Summary of events for each monitored entity

B A C E D

Blocked Execution

9 seconds

Time-integrated summary for processes

Numbers are in seconds (Execution, Blocked)

B=(7,2)

A=(4,5) C=(3,6) D=(9,0) E=(4,5)

17/ 30

(43)

Multi-Scale Data Aggregation – Space

1 Define a neighborhood for each monitored entity

2 Apply an aggregating operator on the neighborhood Neighborhood as ahierarchy

Resource-based Application groups

Deeper the hierarchy→higher the quality

B A

C

E D

M1

M2 M3

C1

C2 G

Space-integrated summary

Aggregating operator: addition (Execution, Blocked)

B=(7,2) A=(4,5)

C=(3,6)

E=(4,5) D=(9,0)

M1=(11,7)

M2=(12,6) M3=(4,5)

C1=(11,7)

C2=(16,11)

G=(27,18)

18/ 30

(44)

Multi-Scale Data Aggregation – Space

1 Define a neighborhood for each monitored entity

2 Apply an aggregating operator on the neighborhood Neighborhood as ahierarchy

Resource-based Application groups

Deeper the hierarchy→higher the quality

B A

C

E D

M1

M2 M3

C1

C2 G

Space-integrated summary

Aggregating operator: addition (Execution, Blocked)

B=(7,2) A=(4,5)

C=(3,6)

E=(4,5) D=(9,0)

M1=(11,7)

M2=(12,6) M3=(4,5)

C1=(11,7)

C2=(16,11)

G=(27,18)

18/ 30

(45)

Visualization Techniques

19/ 30

(46)

Visualization techniques

Squarified Treemap View

Observe outliers, differences of behavior Hierarchical aggregation

B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation

Hierarchical Graph View

Correlate application behavior to network topology Pin-point resource contention

All nodes Clusters Sites Full

aggregation

Host

20/ 30

(47)

Visualization techniques

Squarified Treemap View

Observe outliers, differences of behavior Hierarchical aggregation

B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation

Hierarchical Graph View

Correlate application behavior to network topology Pin-point resource contention

All nodes Clusters Sites Full

aggregation

Host

20/ 30

(48)

Squarified Treemap View

Scalablerepresentation for hierarchies

→bettervisualization scalabilityfor large trees

Space-filling top-down recursive layout algorithm Node value→space occupied in the screen

Squarified version→keeps rectangles ratio close to 1

T=6

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3 N=2

O=1

A=1 B=1 C=1

D=1 E=1 F=1

21/ 30

(49)

Squarified Treemap View

→bettervisualization scalabilityfor large trees Space-filling top-down recursive layout algorithm

Node value→space occupied in the screen

T=6

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3 N=2

O=1

A=1 B=1 C=1

D=1 E=1 F=1

21/ 30

(50)

Squarified Treemap View

T=6

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6

M=3 N=2 O=1

A=1 B=1 C=1

D=1 E=1 F=1

21/ 30

(51)

Squarified Treemap View

T=6

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3 N=2

O=1

A=1 B=1 C=1

D=1 E=1 F=1

21/ 30

(52)

Squarified Treemap View

T=6

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3 N=2

O=1

A=1 B=1 C=1

D=1 E=1 F=1

21/ 30

(53)

Squarified Treemap View – an example

Synthetic trace with 100 thousand processes Two states, four-level hierarchy

Visualization artifacts without spatial aggregation

A

Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)

22/ 30

(54)

Squarified Treemap View – an example

B

22/ 30

(55)

Squarified Treemap View – an example

C

22/ 30

(56)

Squarified Treemap View – an example

D

22/ 30

(57)

Squarified Treemap View – an example

E

Maximum Aggregation

22/ 30

(58)

Squarified Treemap View – KAAPI example

KAAPI (run DAGs, work stealing for load balacing) 188 processes running on five clusters

Rennes Toulouse

Porto Alegre Bordeaux Nancy

~43 s

~110 s

~148 s

~78 s ~65 s

~67 s

~17 s

Analysis: stealing requests depends on latency Porto Alegre – France: ~300 ms In France: ~10 ms

23/ 30

(59)

Squarified Treemap View – KAAPI example

KAAPI (run DAGs, work stealing for load balacing) 188 processes running on five clusters

Rennes Toulouse

Porto Alegre Bordeaux Nancy

~43 s

~110 s

~148 s

~78 s ~65 s

~67 s

~17 s

Analysis: stealing requests depends on latency Porto Alegre – France: ~300 ms In France: ~10 ms

23/ 30

(60)

Hierarchical Graph View

Scalablerepresentation for graphs Topology, with application-level metrics

Identify resourcebottleneckin space and time Use spatial-temporal aggregated traces

Interactive force-directed layout(Barnes-Hut algorithm)

Trace metrics→geometrical properties

→Size, shape, filling, colors

→Nodes: monitored entities

→Edges: relationship among entities

hostA hostB

link

link utilization time slice

24/ 30

(61)

Hierarchical Graph View

Interactive force-directed layout(Barnes-Hut algorithm) Trace metrics→geometrical properties

hostA hostB

link

24/ 30

(62)

Hierarchical Graph View

Interactive force-directed layout(Barnes-Hut algorithm) Trace metrics→geometrical properties

hostA hostB

link

24/ 30

(63)

Hierarchical Graph View - an example

Squares are hosts, diamonds are network links Colors represent different applications

or parts of it (task type, phase)

Two clusters interconnected by four network links

time slice

25/ 30

(64)

Hierarchical Graph View - a larger example

Squares are hosts, computer power defines size French Grid5000 platform: 2170 nodes

All nodes

26/ 30

(65)

Hierarchical Graph View - a larger example

Clusters

26/ 30

(66)

Hierarchical Graph View - a larger example

Sites

26/ 30

(67)

Hierarchical Graph View - a larger example

Host Full

aggregation

26/ 30

(68)

Tools and Framework

27/ 30

(69)

Open-source tools

PajeNG–Next Generation(Space/Time view),^GPL3 http://github.com/schnorr/pajeng/

Onlypj_dumpandpj_validate(partialpajeng) Link againstlibpaje.so– then use Paje API to get data

Data Liberation Front^®

$ ./pj_dump input.paje > output.csv

Then use your preferred visualization tool (gnuplot, R, GNU Octave, ...)

Viva(Aggregated Treemaps, Hierarchical graph),^GPL3 http://github.com/schnorr/viva/

Partial graph visualization for now (soon)

Deprecates Triva, serves asresearch framework Some tracers and converters: SimGrid, Akypuera (withScore-Psupport, such as otf22paje), Poti, XKaapi, EZTrace, rastro2paje, libRastro, GTG, JRastroand counting...

$ ./otf22paje ./scorep-20120827/traces.otf2 | ./viva ...

$ ./otf22paje ./scorep-20120827/traces.otf2 | pj_dump > output.csv ...

28/ 30

(70)

Open-source tools

Onlypj_dumpandpj_validate(partialpajeng) Link againstlibpaje.so– then use Paje API to get data Data Liberation Front^®

Viva(Aggregated Treemaps, Hierarchical graph),^GPL3 http://github.com/schnorr/viva/

28/ 30

(71)

Open-source tools

Viva(Aggregated Treemaps, Hierarchical graph),GPL3

http://github.com/schnorr/viva/

Deprecates Triva, serves asresearch framework

Some tracers and converters: SimGrid, Akypuera (withScore-Psupport, such as otf22paje), Poti, XKaapi, EZTrace, rastro2paje, libRastro, GTG, JRastroand counting...

28/ 30

(72)

Open-source tools

Viva(Aggregated Treemaps, Hierarchical graph),GPL3

http://github.com/schnorr/viva/

28/ 30

(73)

Conclusion and Future Work

Large scale traces – limited screen space Explicit Data aggregation

→Controlled + visual feedback

Concerns with behavior attenuation

Aggregation may remove important details Flexible aggregation: operators & neighborhood Visualization techniques for aggregated data

Continuous evaluation ofvisualization scalability With larger data-sets, does it remain useful?

Future Work

Revisit Space/Time representations

Aggregation operators to deal with time uncertainty To quantify loss of information when aggregating

29/ 30

(74)

Conclusion and Future Work

→Controlled + visual feedback Concerns with behavior attenuation

Aggregation may remove important details Flexible aggregation: operators & neighborhood

Visualization techniques for aggregated data Continuous evaluation ofvisualization scalability

With larger data-sets, does it remain useful?

Future Work

29/ 30

(75)

Conclusion and Future Work

Future Work

29/ 30

(76)

Conclusion and Future Work

Future Work

29/ 30

(77)

Thank you for your attention

Some references

Visualizing More Performance Data Than What Fits on Your Screen. Lucas Mello Schnorr, Arnaud Legrand. The 6th International Parallel Tools Workshop. Springer. 2012. (Invited paper)

Detection and Analysis of Resource Usage Anomalies in Large Distributed Systems Through Multi-scale Visualization. Lucas Mello Schnorr, Arnaud Legrand, Jean-Marc Vincent. Concurrency and Computation: Practice and Experience. Wiley. 2012.

A Hierarchical Aggregation Model to achieve Visualization Scalability in the analysis of Parallel Applications. Lucas Mello Schnorr, Guillaume Huard, Philippe Olivier Alexandre Navaux. Parallel Computing. Volume 38, Issue 3, March 2012, Pages 91-110.

More information

→http://mescal.imag.fr/membres/lucas.schnorr/

INFRA-SONGS Project (WP-7)

http://infra-songs.gforge.inria.fr/

Simulation of Next Generation Systems WP-7: Visualization and Analysis

30/ 30