Metadata-Version: 2.1
Name: sqlpipe
Version: 0.2
Summary: A simple way to run multiple SQL script pipelines in parallel.
Home-page: https://github.com/mariotaddeucci/sqlpipe
Author: Mario Taddeucci
Author-email: mariotaddeucci@gmx.com
License: MIT
Keywords: sql pipeline etl parallel redshift postgres postgresql mysql
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
Requires-Dist: asyncpg (>=0.21.0)
Requires-Dist: aiomysql (>=0.0.20)

## SQLPipe - Easy way to build SQL pipelines

With **SQLPipe** it is possible to create a full pipeline and orchestrate dependencies in a simplified way using SQL. The task management uses the **asyncio** library, so execution is done with the python coroutine system for parallel executions.
![image](https://raw.githubusercontent.com/mariotaddeucci/sqlpipe/master/assets/diagram.png)
**Installation:** 

    pip install sqlpipe

## Basic Example
**Using SqlPipeBuilder**. 
Once the object has been instantiated, use the add_task method to add the scripts to be executed, you must specify a different id for each task, if one task depends on another, use the parent_id parameter to configure the dependency.

	from sqlpipe import SqlPipeBuilder

	con_str =  'postgres://user:pass@host:port/dbname'
	pipe =  SqlPipeBuilder(con_str, connections_limit=5)

	# Add custom script
	pipe.add_task(id='id_2', script='truncate table t1;', parent_id='id_1')

	# Add SQL file script
	pipe.add_task(id='id_1', script='queries/tables/t1.sql', file=True)

	# Add all .sql in directory
	pipe.map_directory('queries')

	# Execute all pipeline
	pipe.execute()
	# Start from specifc task
	pipe.execute('id_2')

**Using CLI**
Call the CLI and pass the directory (-d or --directory) parameter, after that all `.sql` files in the directory and subdirectories will be started. To configure a dependency just put the dependency in a comment on script file, example `/* @parent=my-parent-task */`.

    python -m sqlpipe -d queries
It's possible to call specific task passing the task parameter (-t or --task).

    python -m sqlpipe -d queries -t task-id


**Supported Databases**
 - MySQL
 - PostgreSQL
 - Redshift

**Licence 📜**
Code released under [the MIT license](https://github.com/mariotaddeucci/sqlpipe/blob/master/LICENSE).

