Developing microservice-based distributed workflow engine.

I noticed the need for this simpler approach while working on the Open Source Archiving project called OSA. OSA-project is the open source archiving project which is the main client for this project.

History

Workflow components

Workflow engine

Sometimes it can also allow the user to change the process without programming knowledge [2, 6, 7].

Microservice architectural style

Some examples are that with microservices architecture, services can be deployed independently; this means that if, for example, an error is found in the system, the part of a system that has the error can be removed while the rest of it continues to function. As stated before, the application is intelligently scalable which means that services that need more resources can be scaled independently.

Benefits of microservice architecture

Microservice-based software is not a new invention, but it has only recently made its way into enterprise applications. Make every program do one thing well”, this is pretty self-explanatory, but few points exist. To do a new job, build new rather than complicate old programs by adding new features; this principle helps software stay "recent".

Expect the output of each program to become input to another, as-yet-unknown program,” this principle exists because without it building software using microservices architecture would be very difficult. Another goal of this project was to produce a generic workflow engine component that can also be used in projects that do not deal with archiving in any way.

Simple usage

10 From these points, it can be concluded that microservice architecture makes the life of a developer easier and brings great improvement in code quality. This chapter deals with required features of a workflow engine, since the OSA project was the primary customer for the project, the features are mostly planned to support the needs of digital archiving systems. This workflow engine on the other hand only provides basis for building workflows, with the plugin system developer can create new microservices that can then be used in a workflow.

Open source

It was decided that the workflow engine would be open-source, especially when combined with microservice architecture. This is because if the developer creates a microservice that is compatible with this workflow engine and publishes it as open source, anyone using the engine can use the same service in their workflows. Another reason to create an open source workflow engine is that the main client for the project was the OSA archiving project, and in archiving, especially long-term preservation, a trust between the system provider and the client is very important, and the best way to make a profit. that confidence is to say that “our systems are open source, you can always check how we process your data and how it is stored.” Another point is that when archiving it is very important not to be dependent on one company that offers the service with their closed source software.

Because the goal of the project was to create only a prototype workflow engine and only a single developer worked in the project, publishing the workflow engine as an open source component gives it the possibility to evolve into something which could be more widely used if enough developers contribute to make the engine better. To get enough developers to contribute to the workflow engine, it needs to be generic enough to interest people from different areas of expertise, so even the primary goal was to create the workflow engine for OSA project, for use with archiving and working with Java programming language, the engine also had to work with other domains and other programming languages.

Communication

This is because the time that objects spend to reach can be hundreds of years in long-term archiving, and after that the company may no longer exist, and if the data is kept in the closed source archive, it is not certain that you will find it back. This brings one key advantage to SOAP over REST because the protocol is tightly typed and structured so that the processing of messages can be more standardized and the overall feel is more rigid. This makes REST API easily scalable because new nodes can be started and they do not need to exchange state information with existing nodes.

When the application conforms to all these constraints, it can be considered "RESTful", which means that the application conforms to the REST architectural style. From these points it can be concluded that even REST is not perfect and other protocols have some features that can be considered "better" than REST's equivalents, it was decided that this project will use the REST API since the primary goal was to create a lightweight workflow engine that can be easily adapted to fit into different projects and that is something that REST delivers.

Self-hosted

Also the author of the project had good knowledge of Java, so it was chosen.

Core

These options can be used to perform static configuration for microservices, such as providing database configurations. The tag must contain information about which command the microservice should execute, but it can also be empty if the functionality was programmed in a Java controller. The tag is used to provide the path to the microservice to store the log files, this can also be a url.

For this reason every microservice must have some kind of controller written in java, the controller can then start the actual microservice which can be written in any programming language. The run method must perform the actual task of the microservice, it can be the start of an external program, or the functionality can be programmed directly into the method.

Application programming interface (API)

This method was developed for logging and debugging purposes, but also just to monitor what the workflow engine is doing. This method is for use with some kind of UI so that the UI will know what kinds of actions the workflow engine can perform. This method is not useful in the actual operation of the workflow engine, but it exists as a convenience since it is very likely that the person creating the microservices and/or workflows is not the same person using them.

20 /start/testworkflow?input=/var/files&ouput=/temp these parameters are passed to the workflow in the form of a key -> value map. I call these parameters dynamic parameters since they are provided when the workflow starts and are not configured statically.

Timing/Triggers (QUARTZ)

Also, if the parameter named "schedule" is present, the workflow does not start immediately, but is scheduled according to the cron expression expected to be the value of this "schedule" parameter.

Example workflow and microservices

When transforming this process into microservices, each blue box represents a single microservice that would need to be implemented. The reason files are not automatically archived in Fedora is that all metadata that is important may not be automatically available, but must be entered by human. This phase exists because at this point the metadata may not be accurate and complete so it cannot be archived, however it must be stored somewhere and MongoDB provides excellent support for unstructured data.

This microservice stores documents with collected and user-entered metadata in Fedora Commons. However, the OSA project needs the archived object to be in a specific format, and this functionality is already implemented in the OSA software, but there is no API yet.

Clustering

The system currently only supports clustering within the same local network, but this is something that could be improved in the future. Division of work within the group works using the built-in clustering functions of Quartz programmers. It works in a way that all nodes in a cluster must share the common database which acts as a storage for the workflows.

As shown in Figure 15, the system currently supports clustering only at the workflow level, which is not ideal, as it would be useful if, for example, CPU-intensive microservices were running on a node that has a powerful CPU. However, since this was not the main goal of the project, it was left for further development.

Testing

Although it took more effort to create a custom test environment using Arch Linux, it was decided that it was worth it, as this way it would be certain that any external service or software would not interfere with the testing and cause false results. The basic non-clustering system was found to be working and usable, more testing should be done in the future as the time spent on testing was very limited during this project. There is also a lot more testing to be done in terms of performance testing, which means testing with some large file, testing with a large amount of small files, and testing with a large amount of large files.

Client library

Results

Future development

Documentation should be a bit better and more accurate and client library should cover all the functionality that workflow engine offers. For example, at the beginning it was said that it should not have any extensive XML configuration to work, and the goal during the project was not to make it, but in the end, very small XML configuration was added, mostly because this way the workflow engine is more configurable and can be more flexible. Even the primary customer was the OSA project and their needs are in the archive area, the workflow engine remained generic enough to serve other areas as well.

This was very important and sometimes also difficult to keep the workflow engine generic, but it had to be done to make this interesting for other open source developers. Development will be actively continued by the OSA project to make the engine more production-ready and adapt it more to their needs.