The rise and fall of ThunderSTORM

ThunderSTORM is the codename of our first analytical approach for single-particle detection and super-resolution image reconstruction, in which I have tried to implement state-of-the-art software methods for high-performance analysis of the data. The challenge was clear, as a single super-resolution dataset was typically constituted by hundreds of thousands of images (a single image corresponds to around 0,5 MB), we required a software framework able to deal with multiple datasets and process them in a short amount of time, preferably in real-time.

At the core of ThunderSTORM was an HDF5 data management system. HDF5 is a file format designed for efficient high-speed data storage and exchange. It is specially developed to cache multi-dimensional matrix data and is one of the preferred formats for most physicists collecting massive amounts of information, such as in particle-accelerators or research nuclear reactors (my first contact with HDF5 was through collaborations with an accelerator in the European Organization for Nuclear Research,

unprocessed data acquired from our camera, and supply it as hyper-dimensional packages on demand to requesting “workers”.

To be noted, a “worker” in this context corresponds to an independent instance of an analysis algorithm working within a single CPU. In comparison, a “job-server” is an algorithm specialized in dispatching workloads (data packages to be analysed) to volunteer “workers”.

The ThunderSTORM HDF5 database acted as an independent software instance able to communicate with other algorithms over the network (via socket). It seated over a Redundant Array of Independent Disks (RAID) server with a large amount of free space available for data storage and consultation (see Figure 5.1). It observed the acquisition by monitoring a selected folder where new incoming images were streamed to and pulled each image for storage, in parallel it communicated to any

“worker” that a new image was available for treatment.

On a different level, an algorithm on a workstation within our lab would work as a first layer of data-analysis. It would pull new available images from the HDF5 database, denoise each and find particles within the image without engaging high-precision localization (in essence, this algorithm was a simplified version of the latter developed and published QuickPALM – see section 2.1). The particles were then classified by a k-means clustering algorithm in order to remove misshaped objects such as particle packs too dense to be resolved. The clustering procedure also allowed extracting the most common shape of the particle in order to estimate a parametric description latter given to the more precise (and slower) particle localization estimators. The particles were then fed to available “workers” in the cluster (see Figure 5.1).

Finally, available nodes (workers) in the cluster would receive the segmented particles and localize each through a maximum likelihood estimator and return the particle position back to HDF5 database. I had other features for this level, such as particle tracking and movement correction, but these never got to be implemented.

Appendix

Figure 5.1. ThunderSTORM data analysis framework. ThunderSTORM was composed of 4 levels, each corresponding to an independent program: (i) on the first level an observer program detected new incoming image data streamed from the camera, typically controlled by the µManager software (103), and passed it to the HDF5 database; (ii) the HDF5 database stores data, generates events informing any listeners that new data is available for processing; (iii) a first processing computer denoises each image and segments any found particles, they are then passed to available workers on the cluster and in parallel clustered together to generate a parametric estimation of the PSF; (iv) the workers on the cluster super-resolve the position of each particle and retrieve all additional possible data.

At this point we got into one vital pitfall, namely the way the cluster was set and managed.

The C4 cluster is a system shared by all the research teams at the CSIR. It corresponds to a 3-in-1 cluster (a 184 Opeteron-CPU cluster, a 64 Xeon-CPU cluster and a 64-Itanium2 CPU cluster). The cluster was designed with the goal of having a

of data for large periods of time. For this reason it had a dedicated storage system shared by all the worker nodes and the worker nodes could not be accessed directly by users or programs neither could communicate between each other. All communication was managed through a central node that then passed instructions to the worker nodes (see Figure 5.2).

Oppositely, we required a large number of inter-communicating programs (one per node) to process a high amount of data supplied by our own external database. This on-demand basis directly conflicted with the queuing/scheduling procedure and the required nodes to interchange information dynamically.

I solved these setbacks in a non-trivial way, by hacking the cluster. Normally, users would need to login to the central node to then push a job to the worker nodes. In addition, the worker nodes could also simulate a user login to the central node and this would be the trick I used. When we needed to use the cluster on-demand, we would load a certain share of worker nodes with a sleeper program. The program would then login to the central node and set a port-forward between both nodes (via ssh). Once this connection was established, the sleeper program would then extend the port-forwarding between the central node and our own workstation where the analysis program sit (creating a connection that otherwise would not exist). This way, we actually reversed the connections, instead of being our workstation connecting to the cluster nodes (a procedure not possible), it were the cluster nodes that connected to us. In parallel, once our workstation was made aware of the nodes through their initial connection, they were then also made aware of each other, allowing their needed inter-communication (see Figure 5.2).

Appendix

Storage!

System!

(HDF5)!

Processing computer!

(job-server)!

Central!

Node!

Worker!

Node!

Worker!

Node!

Worker!

Node!

C4 Cluster!

Figure 5.2. ThunderSTORM connection scheme to the cluster. Black lines corresponds to the “canonical” communication method. Red lines correspond to the

“non-standard” connections created on-demand by our algorithms.

This process was not optimal for several reasons. The main one was that the cluster was not designed to be run this way, and although we had the blessing of the cluster manager, we were exploiting security flaws in the cluster management software that allowed us to be in advantage in comparison with other users. On the other side, the port-forwarding system forced us to encrypt our connections, generating severe data transmission latency and lag in parallel to a CPU overhead.

At this time, I joined as a visitor scientist the group of Christophe Zimmer at Institut Pasteur (Paris, France). Zimmer’s group was at that time starting projects within the field of super-resolution.

The downfall of ThunderSTORM came about when we discovered a final fatal problem, the very slow Internet bandwidth speed in South Africa. My plan was to continue the development of ThunderSTORM in Paris by connecting remotely to the CSIR and pushing the results between the two institutes. Unfortunately, up to until recently, the Internet speed in South Africa was forbiddingly slow making the full remote analysis effort a fruitless effort. In fact, the Internet bandwidth speed was such a problem that latter we became aware of research groups in astrophysics using carrier pigeons to transport memory sticks with critical data between remote institutes, an alternative faster than the direct network data transfer itself (see http://news.bbc.co.uk/2/hi/8248056.stm for a news report).

I publically released a critical component out of the core of ThunderSTORM that made all the system possible, the Simple Remote Python (SRPy) package (see http://code.google.com/p/srpy/), which corresponds to the engine that enabled ThunderSTORM to seamless communicate with multiple computers and divide tasks over several CPUs.

My own description of SRPy can be found on its site and is as follows: “Simple Remote Python (SRPy) intends to harness the power of the Python language by allowing multiple python programs (instances) to seamlessly communicate and share information between each other. As such, each CPU (or core) in each computer can be considered as an individual that can request for other individuals to store information or run processing tasks. In this context, SRPy abstracts itself from the physical boundaries between processors and computers by looking at them simply as volunteer workers that are able to deal with workloads - this workers can even work in a social manner by communicating with each other on a non-centralized way, sharing information and processing requests.”

The SRPy package has gained increasing popularity over time. Today it has over 500 known users and is part of several software packages, such as the Andor iQ acquisition software developed by Andor Technology.