Metadata-Version: 2.1
Name: streamflow
Version: 0.2.0.dev3
Summary: StreamFlow framework
Author-email: Iacopo Colonnelli <iacopo.colonnelli@unito.it>
License: LGPL-3.0-or-later
Project-URL: Homepage, https://streamflow.di.unito.it
Project-URL: Package, https://pypi.org/project/streamflow
Project-URL: Repository, https://github.com/alpha-unito/streamflow
Project-URL: Docker, https://hub.docker.com/r/alphaunito/streamflow
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp (==3.8.4)
Requires-Dist: aiosqlite (==0.18.0)
Requires-Dist: antlr4-python3-runtime (==4.12.0)
Requires-Dist: asyncssh (==2.13.1)
Requires-Dist: bcrypt (==4.0.1)
Requires-Dist: cachetools (==5.3.0)
Requires-Dist: cwltool (==3.1.20230302145532)
Requires-Dist: cwl-utils (==0.23)
Requires-Dist: importlib-metadata (==6.0.0)
Requires-Dist: Jinja2 (==3.1.2)
Requires-Dist: jsonref (==1.1.0)
Requires-Dist: jsonschema (==4.17.3)
Requires-Dist: kubernetes-asyncio (==24.2.2)
Requires-Dist: psutil (==5.9.4)
Requires-Dist: rdflib (==6.2.0)
Requires-Dist: yattag (==1.15.1)
Provides-Extra: bandit
Requires-Dist: bandit (==1.7.5) ; extra == 'bandit'
Provides-Extra: docs
Requires-Dist: sphinx (==6.1.3) ; extra == 'docs'
Requires-Dist: sphinx-jsonschema (==1.19.1) ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme (==1.2.0) ; extra == 'docs'
Provides-Extra: lint
Requires-Dist: black (==23.1.0) ; extra == 'lint'
Requires-Dist: codespell (==2.2.4) ; extra == 'lint'
Requires-Dist: flake8-bugbear (==23.2.13) ; extra == 'lint'
Requires-Dist: pyupgrade (==3.3.1) ; extra == 'lint'
Provides-Extra: report
Requires-Dist: pandas (==1.5.2) ; extra == 'report'
Requires-Dist: plotly (==5.11.0) ; extra == 'report'
Requires-Dist: kaleido (==0.2.1.post1) ; extra == 'report'
Provides-Extra: test
Requires-Dist: cwltest (==2.3.20230108193615) ; extra == 'test'
Requires-Dist: pytest (==7.2.2) ; extra == 'test'
Requires-Dist: pytest-asyncio (==0.20.3) ; extra == 'test'
Requires-Dist: pytest-cov (==4.0.0) ; extra == 'test'
Requires-Dist: pytest-xdist (==3.2.0) ; extra == 'test'

# StreamFlow

[![CI Tests](https://github.com/alpha-unito/streamflow/actions/workflows/ci-tests.yaml/badge.svg?branch=master)](https://github.com/alpha-unito/streamflow/actions/workflows/ci-tests.yaml)

The [StreamFlow](https://streamflow.di.unito.it/) framework is a container-native *Workflow Management System (WMS)* written in Python 3.
It has been designed around two main principles:
* Allow the execution of tasks in **multi-container environments**, in order to support concurrent execution
of multiple communicating tasks in a multi-agent ecosystem.
* Relax the requirement of a single shared data space, in order to allow for **hybrid workflow** executions on top of
multi-cloud or hybrid cloud/HPC infrastructures.

## Use StreamFlow

#### PyPI
 
The StreamFlow module is available on [PyPI](https://pypi.org/project/streamflow/), so you can install it using pip.

```bash
pip install streamflow
```

Please note that StreamFlow requires `python >= 3.8`. Then you can execute it directly from the CLI

```bash
streamflow run /path/to/streamflow.yml
```

#### Docker

StreamFlow Docker images are available on [Docker Hub](https://hub.docker.com/r/alphaunito/streamflow). In order to run
a workflow inside the StreamFlow image
 - A StreamFlow project, containing a `streamflow.yml` file and all the other relevant dependencies (e.g. a CWL
   description of the workflow steps and a Helm description of the execution environment) needs to be mounted as a volume
   inside the container, for example in the `/streamflow/project` folder
 - Workflow outputs, if any, will be stored in the `/streamflow/results` folder. Therefore, it is necessary to mount
   such location as a volume in order to persist the results
 - StreamFlow will save all its temporary files inside the `/tmp/streamflow` location. For debugging purposes, or in
   order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume
 - The path of the `streamflow.yml` file **inside the container** (e.g. `/streamflow/project/streamflow.yml`) must be
   passed as an argument to the Docker container

The script below gives an example of StreamFlow execution in a Docker container

```bash
docker run -d \
    --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
    --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
    --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
    alphaunito/streamflow run /streamflow/project/streamflow.yml
```

#### Kubernetes

It is also possible to execute the StreamFlow container as a `Job` in [Kubernetes](https://kubernetes.io/).
In this case, StreamFlow is able to deploy `Helm` charts directly on the parent cluster through the
`ServiceAccount` credentials. In order to do that, the `inCluster` option must be set to `true` for each
involved module on the `streamflow.yml` file

```yaml
deployments:
  helm-deployment:
    type: helm
    config:
      inCluster: true
      ...
```

A `Helm` template of a StreamFlow `Job` can be found in the `helm/chart` folder.

Please note that, in case [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) is active on the
Kubernetes cluster, a proper `RoleBinding` must be attached to the `ServiceAccount` object, in order to give
StreamFlow the permissions to manage deployments of pods and executions of tasks.

## CWL Compatibility

StreamFlow relies on the [Common Workflow Language](https://www.commonwl.org/) (CWL) standard to design workflow models. CWL conformance badges for StreamFlow are reported below.

### CWL v1.0

#### Classes

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/command_line_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/expression_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/workflow.json)

#### Required features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/required.json)

#### Optional features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/docker.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/env_var.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/initial_work_dir.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/inline_javascript.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/multiple_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/resource.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/scatter.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/schema_def.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/shell_command.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/step_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.0/subworkflow.json)

### CWL v1.1

#### Classes

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/command_line_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/expression_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/workflow.json)

#### Required features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/required.json)

#### Optional features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/docker.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/env_var.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/format_checking.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/initial_work_dir.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/inline_javascript.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/inplace_update.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/input_object_requirements.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/multiple_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/networkaccess.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/resource.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/scatter.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/schema_def.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/shell_command.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/step_input_expression.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/step_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/subworkflow.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.1/timelimit.json)

### CWL v1.2

#### Classes

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/command_line_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/expression_tool.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/workflow.json)

#### Required features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/required.json)

#### Optional features

![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/conditional.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/docker.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/env_var.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/format_checking.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/initial_work_dir.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/inline_javascript.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/inplace_update.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/input_object_requirements.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/load_listing.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/multiple_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/multiple.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/networkaccess.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/resource.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/scatter.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/schema_def.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/shell_command.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/step_input.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/subworkflow.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/timelimit.json)
![](https://badgen.net/https/streamflow.di.unito.it/cwl-conformance/v1.2/work_reuse.json)

## Contribute to StreamFlow

As a first step, get StreamFlow from [GitHub](https://github.com/alpha-unito/streamflow) 
```bash
git clone git@github.com:alpha-unito/streamflow.git
```

Then you can install all the required packages using the `pip install` command

```bash
cd streamflow
pip install -r requirements.txt
```

StreamFlow relies on [GitHub Actions](https://github.com/features/actions) for PyPI and Docker Hub distributions. Therefore, in order to publish a
new version of the software, you only have to augment the version number in `version.py` file.

## StreamFlow Team

Iacopo Colonnelli <iacopo.colonnelli@unito.it> (creator and maintainer)  
Barbara Cantalupo <barbara.cantalupo@unito.it> (maintainer)  
Marco Aldinucci <aldinuc@di.unito.it> (maintainer)

Gaetano Saitta <gaetano.saitta@edu.unito.it> (contributor)  
Alberto Mulone <alberto.mulone@edu.unito.it> (contributor)
