Metadata-Version: 2.1
Name: tx-parallex
Version: 0.0.25
Summary: A job queue with data dependencies
Home-page: https://github.com/RENCI/tx-parallex
Author: Hao Xu
Author-email: xuhao@renci.org
License: MIT
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: autorepr (==0.3.0)
Requires-Dist: more-itertools (==8.2.0)
Requires-Dist: jsonschema (==3.2.0)
Requires-Dist: pyyaml (==5.3.1)
Requires-Dist: tx-functional (>=0.0.16)

# `parallex`
## Introduction
A queue with dependencies

## Usage

```
from parallex import run

ret = run(number_of_workers = 4, specf = "spec.yml", dataf = "data.yml")
```

## Spec
Each task is given a dict called `data`.

### `let`
The `let` task sets `data`
```
type: let
obj: 
  <var>: <value>
  ...
  <var>: <value>
sub: <subtask>
```

### `map`
The `map` task reads a list `coll` from `data` and applies a list of subtasks to each member of the list. The members will be assigned to `var` in `data` passed to those tasks

```
type: map
coll: <variable name for collection>
var: <variable name>
sub: <subtask>
```

### `top`

The `top` task toplogically sorts subtasks. 

```
type: top
sub: <subtasks>
```

It reads the `depends_on` property of subtasks, which has format:

```
<task name>: [<param>, ..., <param>]
...
<task name>: [<param>, ..., <param>]
```
The result of a task will be assigned to the parameters that it maps to.

### `python`

The `python` task runs a python function. It reads parameters from `data`.
```
type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
depends_on: <dependencies>
ret: <returns>
```
`params` are the same format as `depends_on`

### `dsl`
A dsl block contains a subset of python.
```
type: dsl
python: <python>
```

Available syntax:

#### assignment
```
<var> = <const> | <list> | <dict>
```
This translates to `let`.

#### function application
```
<var> = <module>.<func>(<param>=<arg>, ...)
```
This translate to `python`.
where `<var>` is `name`

#### return
```
return <dict>
```
this translates to `ret` in `python`.


## Data

data can be arbitrary yaml



