This Project has received funding from the European Union´s Horizon 2020,
research and innovation programme under the Marie Sklodowska-Curie grant
agreement Nº 644202
Deliverable 4.3
Title: Scalability study and parallel IO
Authors: Octavio Castillo-Reyes (BSC) and Josep de la Puente (BSC) Associated Work Package (WP).
WP 4: Large Parallel Simulations of Geophysical Measurements Objectives.
The Parallel Edge-based Tool for Geophyiscal Electromagnetic Modelling (PETGEM) is the main product at the conclusion of the deliverable 4.1 (HPC Co-design of the best candidate application) and deliverable 4.2 (Porting and optimization to target HPC platforms).
In this deliverable we study the scalability and parallel IO performance of PETGEM by using a commonly available Intel Xeon multicore architecture. We verify the code scalability through a well-established 3D CSEM forward modelling case. At the same time, by means of code and data structures optimization, it makes it a competitive tool to simulate real-life scenarios of 3D CSEM forward modelling in geophysics on Intel Xeon-based architectures and thus can be used in a large variety of computer clusters and workstations. The last version of the code, together with extensive documentation, is readily available at http://petgem.bsc.es.
State of the Art, Developed Work, Obtained Results, and Main Innovative Aspects.
PETGEM is a Python code for the scalable solution of 3D CSEM forward modelling on tetrahedral meshes, as these are the easiest to scale-up to very large domains or arbitrary shape. It is written mostly in Python 3.5.2 and relies on the scientific Python software stack with heavy use of mpi4py and petsc4py packages for parallel computations. Other scientific Python packages used include: H5py for binary data format support, Numpy for efficient array manipulation and Scipy algorithms. PETGEM allow users to specify edge-based variational forms of H(curl) for the simulation of electromagnetic fields in real 3D CSEM FM on shared-memory/distributed-memory HPC platforms. Among others, the key drivers for the PETGEM development are the following:
1. Solve a relative scarcity of robust edge-based codes for CSEM modelling to reduce ambiguities in data interpretation for hydrocarbon exploration.
2. Provide synthetic results which can then compare to real data.
3. Simulate real scenarios, i.e. support for geologies structurally complex with a good trade-off between accuracy and number of DOFs.
4. The integration of Python, EFEM and geophysical methods such as 3D CSEM forward modelling is still limited, with pently room for improvement. These concepts has been systematically applied in PETGEM for running simulations using HPC resources.
Although there are specialised modelling tools for geophysical prospecting, details of their implemented methods are generally hidden behind a black box, which could lead to a situation in which the formulation could be unknown. Furthermore, not all numerical schemes are well suited for latest computing architectures or are well adapted to the problem. PETGEM is developed as open-source at Computer Applications in Science & Engineering (CASE) of the Barcelona Supercomputing Center - Centro Nacional de Supercomputación (BSC-CNS). Many features have gradually been included, such as modules for EFEM data structures and a set of Python wrappers for the use of efficient solvers and preconditioners suitable for the resulting matrix system. PETGEM is now a complete package particularly suited for the 3D CSEM forward modelling aiming to foster our understanding about EM in geophysics and its coupling with HPC technologies. Since it was intended tackle realistic problems, its data structure
This Project has received funding from the European Union´s Horizon 2020,
research and innovation programme under the Marie Sklodowska-Curie grant
agreement Nº 644202
was designed to cope simultaneously three key requirements: accuracy, flexibility and efficiency. In addition the adopted algorithms has the posibility to easily add or remove components without having to rewrite large parts of the code. This approach leads to optimal performance in terms of development and computation time. In other words, PETGEM was written based on an architecture-aware design effort in order to ensure a good capacity for large scale computations, thus competence to deal with real models without losing versatility offered by programming language.
We have studied the PETGEM scalability by using the 3D CSEM forward model of work package 4.2 (Porting and optimization to target HPC platforms), namely the canonical model by [1] which consists in four-layers: 1000 m thick seawater (3.3 S/m), 1000 m thick sediments (1 S/m), 100 m thick oil (0.01 S/m) and 1400 m thick sediments (1 S/m). The computational domain is a [0, 3500]3m cube. For this model, a 2 Hz x-directed dipole source is located at z=975, x=1750, y=1750. The receivers are placed in-line to the source position and along its orientation, directly above the seafloor (z=990).
For the scalability test we have been obtained a large tetrahedral mesh using Gmsh which includes local refinement close to the source and receivers position as show Fig. 1. The resulting mesh is composed by 25,653,760 tetrahedral elements and 30,137,768 edges.
Fig. 1 Unstructured tetrahedral mesh for y>1750
In deliverable 4.2 we presented scalability tests at node level on the Marenostrum supercomputer with two-8 cores Intel Xeon processors E52670 at 2.6GHz. Although that code version is efficient and flexible for small 3D CSEM forward modelling, the solution of models at real-scale requires greater computing resources than that provided at node level. To overcome this situation, in this deliverable we present
This Project has received funding from the European Union´s Horizon 2020,
research and innovation programme under the Marie Sklodowska-Curie grant
agreement Nº 644202
scalability results on distributed-memory platforms. Current version of PETGEM allows an automatic parallel matrix and parallel vectors creation when the code is run on many processors. Similarly, if only one processor is specified the PETGEM will run in a sequential mode.
For this 3D CSEM forward modelling, we achieved a quasi-linear scaling for the assembly stage (96 % of parallel efficiency) as show Fig. 2. At the same time, Fig. 3 shows the solver scalability (89 % of parallel efficiency). The solution of the system has been obtained using the GMRES solver with a SOR preconditioner from the PETSc framework.
Fig. 2 Scalability for the assembly task
Fig. 3 Scalabilty for the solving task
Regarding the I/O scalability, we have measured the time for I/O of the dataset associated with the model. A summary of this results is included in Table 1. As we can see, the I/O scalability is not linear but this will improve in future releases.
This Project has received funding from the European Union´s Horizon 2020,
research and innovation programme under the Marie Sklodowska-Curie grant
agreement Nº 644202
# of MPI tasks I/O (Seconds)
1 5.6825e+03 16 3.9241e+03 32 2.4741e+03 48 1.6713e+03 64 1.4765e+03 80 1.3471e+03 96 1.2118e+03 128 1.1103e+03 256 0.9552e+03 512 0.8476e+03 768 0.7184e+03 1024 0.6888e+03 1152 0.6189e+03 1536 0.5592e+03 1664 0.4946e+03 1792 0.4107e+03 1920 0.3174e+03
Table 1Summary of results for I/O Impact for Science and Society.
In past 2 decades the modelling tools have become one of the pillars for the simulation of numerical methods which main goal is elucidating the fundamental mechanisms behind simplified abstractions of complex phenomena in different areas. The 3D CSEM forward modelling in geophysics is no exception and the scientific community has developed important contributions in this field. However, the tools that full fit needs for the solution of real models are commercial and often are inaccessible to the wider scientific community. Due to the discretization method employed, not all codes that are affordable to community are capable of dealing with complex geometries such as models including bathymetries. Additionally there are few parallel codes that are efficient, scalable and can deliver good performance. PETGEM aims to solve a relative scarcity of robust edge-based codes to simulate these problems on HPC architectures. PETGEM is implemented in current state-of-art platforms such as Intel Haswell and Intel Xeon Phi processors, which offer high performance, flexibility and power efficiency. Nevertheless, PETGEM support older architectures such as SandyBridge, for the sake of usability and to be able to compare performance. Current version of PETGEM is available at http://petgem.bsc.es. The code is supplied in a manner to ease the immediate execution under Linux platforms. Examples, user’s manual and technical documentation (developer’s guide) are provided in the PETGEM archive as well.
Dissemination and Transfer of Knowledge.
Octavio Castillo-Reyes has participated in 5 international conferences and meetings, discussing results directly related to this deliverable. Similarly, Josep de la Puente has presented results of this Project in two international meetings.
Main Participants.
The main participants of this task were Octavio Castillo-Reyes and Josep de la Puente (BSC), and Helene Barucq and Julien Diaz (INRIA). Furthermore, actual version of the code was enriched from discussions within BSC team (Mauricio Hanzich, David Modesto and Samuel Rodríguez).
Intellectual Property Rights (IPR). Public.
This Project has received funding from the European Union´s Horizon 2020,
research and innovation programme under the Marie Sklodowska-Curie grant
agreement Nº 644202
Additional Info / References:
[1] Constable, S. and Weiss, C. J.: Mapping thin resistors and hydrocarbons with marine EM methods: Insights from 1D modeling, Geophysics, 71, G43-G51, 2006.
