.. include:: header.txt

==============================
 processing package reference 
==============================

The `processing` package mostly replicates the API of the `threading`
module.


Classes and exceptions
----------------------

    **class** `Process(group=None, target=None, name=None, args=(), kwargs={})`
        An analogue of `threading.Thread`.

        See `Process objects`_.

    **exception** `ProcessExit`
        Exception raised in a target process when the `Process.stop()`
        method is used.  This is a subclass of `SystemExit`.

    **exception** `BufferTooShort`
        Exception raised by the `recvbytes_into()` method of a
        `connection object <connection-objects.html>`_
        when the supplied buffer object is too small for the message
        read.

        If `e` is an instance of `BufferTooShort` then `e.args[0]`
        will give the message as a byte string.


Pipes and Queues
----------------

When using multiple processes one generally uses message passing for
communication between processes and avoids having to use any
synchronization primitives like locks.  

For passing messages one can use a pipe (for a connection between two
processes) or a queue (which allows multiple producers and consumers).

Note that one can also create a shared queue by using a manager object
-- see `Managers`_.

For an example of the usage of queues for interprocess communication
see `test_workers.py <../test/test_workers.py>`_.

    `Pipe(duplex=True)`
        Returns a pair `(conn1, conn2)` of connection objects
        representing the ends of a pipe.  

        If `duplex` is true then the pipe is two way; otherwise
        `conn1` can only be used for receiving messages and `conn2`
        can only be used for sending messages.
        
        See `Connection objects <connection-objects.html>`_.

    `Queue(maxsize=0)`
        Returns a process shared queue object.  The usual `Empty` and
        `Full` exceptions from the standard library's `Queue` module
        are raised to signal timeouts.

        See `Queue objects <queue-objects.html>`_.

     
Synchronization primitives
--------------------------

Generally synchronization primitives are not as necessary in a
multiprocess program as they are in a mulithreaded program.  See the
documentation for the standard library's `threading` module.

Note that one can also create synchronization primitves by using a
manager object -- see `Managers`_.

    `BoundedSemaphore(value=1)`
        Returns a bounded semaphore object: a clone of
        `threading.BoundedSemaphore`.

    `Condition(lock=None)`
        Returns a condition variable: a clone of `threading.Condition`.
        
        If `lock` is specified then it should be a `Lock` or `RLock`
        object from `processing`.
        
    `Event()`
        Returns an event object: a clone of `threading.Event`.

    `Lock()` 
        Returns a non-recursive lock object: a near clone of `threading.Lock`.

        There are two differences from `threading.Lock`: trying to
        acquire a lock already owned by the current thread raises an
        exception instead of deadlocking; and trying to release a lock
        held by a different thread/process will raise and exception.
                        
    `RLock()`
        Returns a recursive lock object: a clone of `threading.RLock`.
        
    `Semaphore(value=1)`
        Returns a bounded semaphore object: a clone of
        `threading.Semaphore`.


Managers
--------

Managers provide a way to create data which can be shared between
different processes.

    `LocalManager()` 
        Returns a manager object which uses shared memory instead of a
        server process.  It has instance methods
        
            `SharedValue`, `SharedStruct`, `SharedArray` 

        for creating objects stored in shared memory map.  

        See `LocalManager <manager-objects.html#shared-memory-managers>`_.


    `Manager()`
        Returns a started `SyncManager` object which can be
        used for sharing objects between processes.  The returned
        manager object corresponds to a spawned child process and has
        methods which will create shared objects and return
        corresponding proxies.

        The methods for creating shared objects are

            `list()`, `dict()`, `Namespace()`, 
            `SharedValue()`, `SharedStruct()`, `SharedArray()`, 
            `Lock()`, `RLock()`, `Semaphore()`, `BoundedSemaphore()`, 
            `Condition()`, `Event()`, `Queue()`.
            
        See `SyncManager <manager-objects.html#sync-manager>`_.


Process Pools
-------------

One can create a pool of processes which will carry out tasks
submitted to it.

    `Pool(processes=None)`
        Returns a process pool object which controls a pool of worker
        processes to which jobs can be submitted.

        It supports asynchronous results with timeouts and
        callbacks and has a parallel map implementation.

        If `processes` is `None` then the number returned by
        `cpuCount()` is used.  

        See `Pool objects <pool-objects.html>`_.
        

Logging
-------

Some support for logging is available.  Note, however, that the
`logging` package does not use process shared locks so it is possible
(depending on the handler type) for messages from different processes
to get mixed up.

    `enableLogging(level=10, HandlerType=None, handlerArgs=(), format=None)`
        Enables logging and sets the debug level used by the package's
        logger to `level` -- see documentation for the `logging`
        package in the standard library.  The default logging level is
        `10` which is equivalent to `logging.DEBUG`.

        If `HandlerType` is specified then a handler is created using
        `HandlerType(*handlerArgs)` and this will be used by the
        logger -- any previous handlers will be discarded.  If
        `format` is specified then this will be used for the handler;
        otherwise `format` defaults to
        `'[%(levelname)s/%(processName)s] %(message)s'`.  (The logger
        used by `processing` allows use of the non-standard
        `'%(processName)s'` format.)

        If `HandlerType` is not specified and the logger has no
        handlers then a default one is created which prints to
        `sys.stderr`.

        *Note*: on Windows a child process does not directly inherit
        its parent's logger; instead it will automatically call
        `enableLogging()` with the same arguments which were used when
        its parent process last called `enableLogging()` (if it ever
        did).

    `getLogger()`
        Returns the logger used by `processing`.  If `enableLogging()`
        has not yet been called then `None` is returned.

Below is an example session with logging turned on::

    >>> import processing, logging
    >>> processing.enableLogging(level=logging.INFO)
    >>> processing.getLogger().warn('doomed')
    [WARNING/MainProcess] doomed
    >>> m = processing.Manager()
    [INFO/SyncManager-1] process starting up
    [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-1352-0-r97d0b'
    >>> del m
    [INFO/MainProcess] sending shutdown message to manager
    [INFO/SyncManager-1] manager received shutdown message
    [INFO/SyncManager-1] running all "atexit" finalizers
    [INFO/SyncManager-1] process exiting with `os.exit(0)`


Miscellaneous
-------------

    `activeChildren()`
        Return list of all live children of the current process.
        
        Calling this has the side affect of "joining" any processes
        which have already finished.

    `cpuCount()`
        Returns the number of CPUs in the system.  May raise
        `NotImplementedError`.

    `currentProcess()`
        An analogue of `threading.currentThread`

        Returns the object corresponding to the current process.

    `freezeSupport()`
        Adds support for when a program which uses the `processing`
        package has been frozen to produce a Windows executable.  (Has
        been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.)

        One needs to call this function straight after the `if __name__
        == '__main__'` line of the main module.  For example ::

            from processing import Process, freezeSupport

            def f():
                print "hello world!"

            if __name__ == '__main__':
                freezeSupport()
                p = Process(target=f)
                p.start()

        If the `freezeSupport()` line is missed out then trying to run
        the frozen executable will raise `RuntimeError`.

        If the module is being run normally by the python interpreter
        then `freezeSupport()` has no effect.
        

.. note::
   * The `processing.dummy` package replicates the API of `processing`
     but is no more than a wrapper around the `threading` module.
     
   * `processing` contains no analogues of `activeCount`,
     `enumerate`, `settrace`, `setprofile`, `Timer`, or
     `local` from the `threading` module.


Subsections
-----------

+ `Process objects <process-objects.html>`_
+ `Queue objects <queue-objects.html>`_
+ `Connection objects <connection-objects.html>`_
+ `Manager objects <manager-objects.html>`_
+ `Proxy objects <proxy-objects.html>`_
+ `Pool objects <pool-objects.html>`_
+ `Shared ctypes object <sharedctypes.html>`_
+ `Listeners and Clients <connection-ref.html>`_

.. _Prev: intro.html
.. _Up: index.html
.. _Next: process-objects.html
