ScientificWorkflow-v2

(1)

Scien,ﬁc Workﬂow

Fabio Porto

[email protected]

h?p://dexl.lncc.br

Outline

• From Wet-‐lab to in-‐silico

• Scien,ﬁc Workﬂows Concepts

• Scien,ﬁc Workﬂow Model (introduc,on)

• Example

Scien,ﬁc Workﬂow

15/05/12 2

From wet lab to in-‐silico research

• As science moves to in-‐silico, scien,ﬁc life-‐

cycle must be supported;

• The availability of data enables mul,ple

experiments/analyses;

• Scien,ﬁc notebook fruit of provenance

records;

In-‐silico science

• The overwhelming amount of produced data

to be analysed

• Ini,al strategies based on Unix-‐like scrip,ng

languages

• Complexity in managing, transforming,

(2)

Scien,ﬁc Workﬂows

A high level descrip1on of the process used to carry out computa1onal and analy1cal experiments

• Provide high-‐level modelling approaches for specifying analyses

• Oﬀer generic services to op,mize execu,on • Manage intermediate results

• Hide distribu,on and paralleliza,on • Store provenance informa,on

15/05/12 Scien,ﬁc Workﬂow 5

Scien,ﬁc Workﬂow Life-‐cycle

Ludascher et al 2009 updated F. Porto 2012

15/05/12 Scien,fic Workflow 6 Hypothesis, experiment Goals Experiment, Workflow Design Workflow Prepara,on Workflow Execu,on Post-‐ Execu,on analysis Workflow

repository Sources Data Provenance Store Monitoring Hypotheses database

Phases

•  Hypotheses Formula,on – Phenomenon Descrip,on

• Observed physical quan,,es

• Topology – Hypothesis descrip,on • Valida,on criteria • Constraints • Space-‐,me, scale •  Workﬂow Design

– Iden,fy exis,ng resources to be reused:

• Workﬂow templates

• Programs

– Model ac,vi,es and dependencies; – Deﬁne data requirements

• Formats, localiza,on, selec,on criteria;

• Specify data transforma,on procedures

Phases (cont.)

•  Workﬂow Prepara,on

– Choose data sources and parameter values;

– Select execu,on environment;

– Reserve physical resources (e.g. Cluster);

– Deﬁne execu,on model of ac,va,ons;

•  Workﬂow Execu,on

– Run workﬂows;

– Store provenance informa,on;

– Keep ﬁnal and intermediate results;

(3)

Phases (cont.)

• Post-‐execu,on analysis

– Does the result makes sense?

• Examine execu,on traces

– Which results were tainted by this input dataset?

• Data dependencies

– Why did this step failed?

• Debug runs

– Which step took the longest ,me

• performance

Roles

• A workﬂow involves diﬀerent roles:

– Domain scien,sts

• Act as (high-‐level) workﬂow designers;

• May act as workﬂow operators;

– Workﬂow Engineers

• Implement new ac,vi,es

• Integrate the workﬂow into a par,cular workﬂow

system

• Analyse and ﬁx bugs

Types of Scien,ﬁc Workﬂows

•  There is no agreement on a classiﬁca,on but

– Exploratory workﬂows

• The design phase is very preliminary • Language shall allow easy reformula,on • Techniques to stop and analyse intermediate results

– Produc,on workﬂows

• Detailed design phase • Less prone to modiﬁca,ons

• Models a known scien,ﬁc procedure or protocol

– Science oriented workﬂows

• Reﬂect scien,ﬁc experiments

– Engineering workﬂows

• Deal with data movement and job management

Concepts and System Func,ons

• Integrated Workﬂow Environment

– Environment to support the scien,ﬁc life cycle – Use Visual programming interface for workﬂow

design

– Provide libraries of exis,ng components – Templates and workﬂows from workﬂow

repositories

(4)

Concepts and System Func,ons

• Workﬂow Prepara,on and Execu,on Support

– Support to parameter sweep;

– Smart rerun avoiding costly recomputa,on;

– Run,me monitoring;

– Fault-‐tolerance with “smart-‐resume”

– Mapping to distributed nodes;

Concepts and System Func,ons

• Data-‐driven

– Although workﬂow design focuses on processing steps, workﬂow computa,ons are data driven

• Without workﬂow system scien,sts spend their ,me

– Reading, reformaWng, transfering and saving datasets

• Each step deﬁnes a data transforma,on ac,vity

– Dataflow oriented workflows [Lee,Parks 1995 Dataflow Process Network]

• Emphasize the central role of data

• Data ﬂow passes through workﬂow steps

Proccess vs Data driven

A

B

C

D

Step A is at the same ,me a processing step and a data spliWng step

A B C D s1 s2 s3 s4 Flow graph speciﬁes how data is guided through the processes

Workﬂows in diﬀerent levels of

abstrac,on (Ogasawara et al 2009)

•  Scien,fic workflows are usually defined in two levels of

abstrac,on: abstract and concrete;

•  Abstract – refers to the speciﬁca,on of ac,vi,es with

no reference to physical resources or par,cular implementa,on.

– Wokﬂow represented as a DAG of conceptual ac,vi,es;

•  Concrete – deﬁne technological characteris,cs and

computa,onal resources required to run the workﬂow;

– Is a par,cular instan,a,on of an abstract workﬂow

– Ac,vi,es are named concrete ac1vi1es or tasks

•  workﬂow system suppor,ng abstract workﬂow

(5)

Concrete vs abstract workﬂows

(taverna)

Sequence to

align Protein Type

Align Sequence

Alignment result

a) Concrete workflow in Scufl (Taverna) b) Abstract Workflow (GExpline)

Implica,ons Abstract and Concrete

Workﬂows

• The same abstract workﬂow may be

implemented diﬀerently:

– Changing tasks that implement ac,vi,es

• Implemen,ng AlignSequence using other alignment

tool;

• Keep provenance informa,on about the experiment

evolu,on

– Usually this is implemented designing a new workﬂow

Scien,ﬁc Workﬂow Model

Dealing with heterogeneity

• integra,on of autonomously deﬁned programs pose constraints on tasks integra,on;

• Proposed solu,ons

– Shims – processing steps that act as adaptors

changing format of data between heterogeneous format tasks (Oinn et al. 2006 – Taverna)

• Control operator in QEF

– Data Model – make it explicit the data model

• Describe inputs and output data schema;

• Using mappings between schema to handle data

transforma,on declara,vely

Scien,ﬁc workﬂow Model

•  Current State:

– No standard language for modelling scien,ﬁc workﬂow

• BPEL – use for business workﬂows

– Data has no schema and passed as ﬁles through ac,vi,es

•  General View:

– A set of ac,vi,es A={a1, a2,…, an}

• Ac,vi,es are:

– Data manipula,on (e.g. transforma,on, formaWng,…) – Data analyses

• par,al order among ac,vi,es

– modelled as a directed acyclic graph

• Some scien,ﬁc workﬂow systems support cycles: Kepler, QEF

– Data are modelled as input/output of ac,vi,es

• Data Unit is a set of elements that compose the input to an ac,vity

– Data Unit list -‐ DU={ du1,du2,…duk}

(6)

Scien,ﬁc Workﬂow Execu,on model

• Each ac,vity becomes a Task

• The workﬂow engine executes an instance of the workﬂow

–  schedules tasks based on the par,al ordering among

the corresponding ac,vi,es

– Evaluates a single data unit set (DU)

– If workﬂow system support cycles

• Parameter sweep, Turbulence simula,on, Par,cle tracing

• A single instance of workﬂow evaluates a set of data unit

sets

Risers Fa,gue Analysis (RFA) scien,ﬁc

workﬂow

Static Data (sdat) files, Dynamic Data (ddat) files sdat files, ddat files

SAV files, SsSai files

ddat files, SiSai files, SsSai files, SAV files

DiSai files Compressed Riser Data (rd.zip) (environmental

conditions, physical data and riser geometry)

DdSai files, Env files

SsSai files, DdSai files, Env files SiSai files, FTE files,

FTR files sdat files, FTR files

1. Extraction of Riser Data

call ExtractRD(rd.zip)

2. Run Static Analysis Preprocessing

For each rd in RD Set, x in sdat files,

y in ddat files call PSRiser(rd, x, y)

3. Run Static Analysis

For each rd in RD Set, x in sdat files, y

in FTR files call SRiser(rd, x, y)

6. Run Tension Analysis

For each rd in RD Set, x in DdSai files,

y in Env files call Tanalysis(rd ,x, y) 7. Run Curvature Analysis

For each rd in RD Set, z in SsSai files call Canalysis(rd, z) ddat files, DiSai files,

FTE files, SAV files

DdSai files, Env files SsSai files

8. Merge Data

For each rdT in Accepted Tension

RD Set, rdC in Accepted Curvature RD Set call Match (rdT, rdC) 9. Compression of Riser’s Results

call CompressRD(Merged Rd, SsSai

files, DdSai files, MEnv files)

Riser Data (RD) set RD set RD set RD Set RD Set RD Set Accepted Tension RD Set Accepted Curvature RD Set Merged RD Set Shared disk

5. Run Dynamic Analysis

For each rd in RD Set, x in ddat files,

y in DiSai files, z in FTE files, w in SAV files

call Driser (rd, x, y, z, w)

4. Run Dynamic Analysis Preprocessing

For each rd in RD Set, x in ddat files,

y in SiSai files, z in SsSai files, w in SAV files

call PDRiser(rd, x, y, z, w)

RdResult.zip

Scien,ﬁc Workﬂow Systems

• VisTrails – scien,ﬁc visualiza,on, design tool,

provenance management

• Kepler – high-‐level data model, design tool,

execu,on model, provenance management

• Taverna – biologia, design tools, web services,

MyExperiment , Xsculf, Proveniência;

• Swix – biology, HPC, scrip,ng language

• QEF – algebraic based scien,ﬁc workﬂow

Taverna Scuﬂ

•  Taverna language components:

– Inputs – entry points for the data for the workﬂow

– Output – exit point for the data for the workﬂow

– Processors – an individual step in the workﬂow (task)

• Input ports and output ports

– Data links – links data sources to data des,na,ons

• Data sources – workﬂow inputs or processor output ports

• Data des,na,ons – workﬂow outputs or processors input

(7)

Kepler (Al,ntas et al 2006)

•  Workﬂow language:

–  Actors – specify what processing occurs. • Kepler comes with more than 350 actors • Deﬁne the tasks to be done

• User-‐deﬁned actors can be added to the Kepler repository –  Directors – specify when the processing occurs

• Implements computa,onal model (e.g. synchronous, parallel) • Provides execu,on parameters

• Example: SDF – process simple pipeline type of workﬂows; CT-‐ workﬂows that evolve as con,nuous func,on of ,me

–  Composite/individual Actors

• Composite -‐ – subworkﬂows (reuse) • Individual –single task –  Parameters – conﬁgure actors behavior –  Rela,on – allow data to be sent to mul,ple consumers

•  Mul,disciplinary

•  Design and execu,on of workﬂows

•  Capture of Provenance data, both prospec,ve and retrospec,ve

The Lotka_Volterra Model

Predator-‐prey

• Two equa,ons modeling the modiﬁca,on of

the popula,ons of prey and predator

– dn1/dt = r*n1 -‐ a*n1*n2

– dn2/dt = -‐d*n2 + b*n1*n2

• The workﬂow:

– 6 actors= 2 plo?ers, 2 equa,ons and 2 integral func,ons

– 1 director – deﬁnes execu,on parameters

• ,me, step-‐size, max-‐itera,ons, ODESolver

(8)

QEF (LNCC)

• Algebra Based Workﬂow engine • Components

– Logical operators – implement applica,on logic

– Control operators – implement data manipula,on

– DataSources – Implement input/output data

– DataUnit – data communica,on between operators

• Paralelism, loops • No design tool

• No provenance management

15/05/12 Scien,ﬁc Workﬂow 29 eScience 2009

Adap,ve and Extensible Query Engine

• Extensible to data types

• Extensible to applica,on algebra

• Extensible to execu,on model

• Schedule opera,ons in grid nodes

• Adap,ve execu,on model

Objec(ve

•  Offer a query processing framework that can be extended to adapt to data centric grid application needs;

•  Offer transparency in using resources to answer queries;

•  Query optimization transparently introduced

•  Standardize remote communication using web services even when dealing with large amount of unstructured data •  Run-time performance monitoring and decision

Control Operators

•  Add data-flow and transformation operators •  Isolate application oriented operators from execution model data-flow concerns •  parallel grid based execution model:

•  Split/Merge - controls the routing of tuples to parallel nodes and the corresponding unification of multiple routes to a single flow

•  Send/Receive - marshalling/ unmarshalling of tuples and interface with communication mechanisms •  B2I/I2B - blocks and unblocks tuples •  Orbit - implements loop in a data-flow

(9)

eScience 2009

The Execu(on Model

Example of simple QEF Workflow

Data sources (Input)

Output Operator

Possibly distributed over a Grid environment

Integra,on unit (Tuple) containing data source units

eScience 2009

Itera(on Model

A B C DataSource OPEN OPEN OPEN A B C DataSource GETNEXT GETNEXT GETNEXT A B C DataSource

CLOSE CLOSE CLOSE

Results

Distribu(on and Paralleliza(on

Operator distribution

A Query Optimizer selects a set of operators in the QEP to execute over a Grid environment.

A B2 C DataSource B1 B3

General Parallel Execu(on Model

Remote QEP

In order to parallelize an execution, the initial QEP is modified and sent to remote nodes to handle the distributed execution. Control operator Distributed operator R : Receiver S : Sender Sp : Split Ini(al

(10)

eScience 2009

Modifying IQEP to adapt to execu(on

model

Par,cles Geometry Velocity A (TCP) SJ TJ Orbit merge Split Send Receive B2I Send I2B Receive B2I I2B

Query op,mizer adds control operators according to execu,on model and IQEP sta,s,cs Local dataﬂow Remote dataﬂow Logical operator Control operator Control node Remote nodei eScience 2009

Grid node alloca(on algorithm (G2N)

Grid Greedy Node scheduling algorithm (G2N)

•  Offers maximum usage of scheduled resources during query evaluation.

•  Basic idea : “an optimal parallel allocation strategy

for an independent query operator … is the one in which the computed elapsed-time of its execution is as close as possible to the maximum sequential time in each node evaluating an instance of the operator”.

A Bn € t1+ t2= tx

( )

Bn node on this cost operator ) (Bn t 1 t 2 t Introduc(on Applica(on Architecture Implem. Conclusion Principles

Implementa(on

•  Core development in Java 1.5. •  Globus toolkit 4.

•  Derby DBMS (catalog).

•  Tomcat, AJAX and Google Web Toolkit for user interface.

•  Runs on Windows, Unix and Linux. •  source code, demo, user guide available at:

ScientificWorkflow-v2

Scien,ﬁc Workﬂow

Fabio Porto

[email protected]

h?p://dexl.lncc.br

Outline

•

From Wet-­‐lab to in-­‐silico

•

Scien,ﬁc Workﬂows Concepts

•

Scien,ﬁc Workﬂow Model (introduc,on)

•

Example

From wet lab to in-­‐silico research

•

As science moves to in-­‐silico, scien,ﬁc life-­‐

cycle must be supported;

•

The availability of data enables mul,ple

experiments/analyses;

•

Scien,ﬁc notebook fruit of provenance

records;

In-­‐silico science

•

The overwhelming amount of produced data

to be analysed

•

Ini,al strategies based on Unix-­‐like scrip,ng

languages

•

Complexity in managing, transforming,

Scien,ﬁc Workﬂows

Scien,ﬁc Workﬂow Life-­‐cycle

Phases

Phases (cont.)

Phases (cont.)

•

Post-­‐execu,on analysis

Roles

•

A workﬂow involves diﬀerent roles:

Types of Scien,ﬁc Workﬂows

Concepts and System Func,ons

•

Integrated Workﬂow Environment

Concepts and System Func,ons

•

Workﬂow Prepara,on and Execu,on Support

Concepts and System Func,ons

•

Data-­‐driven

Proccess vs Data driven

Workﬂows in diﬀerent levels of

abstrac,on (Ogasawara et al 2009)

Concrete vs abstract workﬂows

(taverna)

Implica,ons Abstract and Concrete

Workﬂows

•

The same abstract workﬂow may be

implemented diﬀerently:

Scien,ﬁc Workﬂow Model

Dealing with heterogeneity

Scien,ﬁc workﬂow Model

Scien,ﬁc Workﬂow Execu,on model

Risers Fa,gue Analysis (RFA) scien,ﬁc

workﬂow

Scien,ﬁc Workﬂow Systems

•

VisTrails – scien,ﬁc visualiza,on, design tool,

provenance management

•

Kepler – high-­‐level data model, design tool,

execu,on model, provenance management

•

Taverna – biologia, design tools, web services,

MyExperiment , Xsculf, Proveniência;

•

• 

From Wet-‐lab to in-‐silico

• 

• 

• 

From wet lab to in-‐silico research

• 

As science moves to in-‐silico, scien,ﬁc life-‐

• 

• 

In-‐silico science

• 

• 

Ini,al strategies based on Unix-‐like scrip,ng

• 

Scien,ﬁc Workﬂow Life-‐cycle

• 

Post-‐execu,on analysis

• 

• 

• 

• 

Data-‐driven

• 

• 

• 

Kepler – high-‐level data model, design tool,

• 

• 

• 

Predator-‐prey

• 

• 

• 

• 

• 

• 

• 