.. _debug-drive:

Debugging and Driving YT
========================

There are several different convenience functions that allow you to control YT
in perhaps unexpected and unorthodox manners.  These will allow you to conduct
in-depth debugging of processes that may be running in parallel on multiple
processors, as well as providing a mechanism of signalling to YT that you need
more information about a running process.  Additionally, YT has a built-in
mechanism for optional reporting of errors to a central server.  All of these
allow for more rapid development and debugging of any problems you might
encounter.

Additionally, ``yt`` is able to leverage existing developments in the IPython
community for parallel, interactive analysis.  This allows you to initialize
multiple YT processes through ``mpirun`` and interact with all of them from a
single, unified interactive prompt.  This enables and facilitates parallel
analysis without sacrificing interactivity and flexibility.

.. _pastebin:

The Pastebin
------------

At http://paste.enzotools.org/ a pastebin is available for placing scripts.
With ``yt`` the script ``yt_lodgeit.py`` is distributed, which allows for
commandline uploading and downloading of pasted snippets.  To upload script you
would supply it to the command:

.. code-block:: bash

   $ yt_lodgeit.py some_script.py

The URL will be returned.  If you'd like it to be marked 'private' and not show
up in the list of pasted snippets, supply the argument ``--private``.  All
snippets are given either numbers or hashes.  To download a pasted snippet, you
would use the ``--download`` option:

.. code-block:: bash

   $ yt_lodgeit.py --download=216

The snippet will be output to the window, so output redirection can be used to
store it in a file.

.. _error-reporting:

Error Reporting with the Pastebin
+++++++++++++++++++++++++++++++++

If you are having troubles with ``yt``, you can have it paste the error report
to the pastebin by running your problematic script with the ``--paste`` option:

.. code-block:: bash

   $ python2.6 some_problematic_script.py --paste

The ``--paste`` option has to come after the name of the script.  When the
script dies and prints its error, it will also submit that error to the
pastebin and return a URL for the error.  When reporting your bug, include this
URL and then the problem can be debugged more easily.

For more information on asking for help, see `asking-for-help`.

Signaling YT to Do Something
----------------------------

During startup, ``yt`` inserts handlers for two operating system-level signals.
These provide two diagnostic methods for interacting with a running process.
Signalling the python process that is running your script with these signals
will induce the requested behavior.  

   SIGUSR1
     This will cause the python code to print a stack trace, showing exactly
     where in the function stack it is currently executing.
   SIGUSR2
     This will cause the python code to throw a RuntimeError.  If you are
     running with the ``--rpdb`` options (see :ref:`remote-debugging`) this
     will also cause the interpreter to sit inside a debug loop.  If you are
     running inside ``pdb`` (see :mod:`pdb` ) then
     ``pdb`` will produce a debug loop.

If your ``yt``-running process has PID 5829, you can signal it to print a
traceback with:

.. code-block:: bash

   $ kill -SIGUSR1 5829

Note, however, that if the code is currently inside a C function, the signal
will not be handled, and the stacktrace will not be printed, until it returns
from that function.

.. _remote-debugging:

Remote and Disconnected Debugging
---------------------------------

If you are running a parallel job that fails, often it can be difficult to do a
post-mortem analysis to determine what went wrong.  To facilitate this, ``yt``
has implemented an `XML-RPC <http://en.wikipedia.org/wiki/XML-RPC>`_ interface
to the Python debugger (``pdb``) event loop.  

Running with the ``--rpdb`` command will cause any uncaught exception during
execution to spawn this interface, which will sit and wait for commands,
exposing the full Python debugger.  Additionally, a frontend to this is
provided through the ``yt`` command.  So if you run the command:

.. code-block:: bash

   $ mpirun -np 4 python2.6 some_script.py --parallel --rpdb

and it reaches an error or an exception, it will launch the debugger.
Additionally, instructions will be printed for connecting to the debugger.
Each of the four processes will be accessible via:

.. code-block:: bash

   $ yt rpdb 0

where ``0`` here indicates the process 0.

For security reasons, this will only work on local processes; to connect on a
cluster, you will have to execute the command ``yt rpdb`` on the node on which
that process was launched.

Interactive Parallel Processing with IPython
--------------------------------------------

IPython is a powerful mechanism not only for interactive usage, but also for
task delegation and parallel analysis driving.  Using the 
`IPython Parallel Multi Engine <http://ipython.scipy.org/doc/manual/html/parallel/parallel_multiengine.html>`_
interface, you can launch multiple 'engines' for computation which can then be
driven by ``yt``.  However, to do so, you will have to ensure that the IPython
dependencies for parallel computation are met -- this requires the installation
of a few components.

  * `PyOpenSSL <http://pyopenssl.sourceforge.net/>`_
  * `Twisted <http://twistedmatrix.com/trac/>`_
  * `Foolscap <http://foolscap.lothar.com/trac>`_

Both Twisted and Foolscap can be installed using ``easy_install`` but PyOpenSSL
requires manual installation.  Of course, ``yt`` itself requires `mpi4py
<http://code.google.com/p/mpi4py/>`_ to be installed as well, which is
described in :ref:`parallel-computation`.

The entire section in the IPython manual on
`parallel computation <http://ipython.scipy.org/doc/manual/html/parallel/index.html>`_
is essential reading, but for a quick start, you need to launch the engines:

.. code-block:: bash

   $ ipcontroller
   $ mpirun -np 4 ipengine

This will launch the controller, which interfaces with the new computation
engines launched afterward.  Note that you can launch an arbitrary number of
compute processes.  Now, launch IPYthon:

.. code-block:: bash

   $ ipython

and execute the commands:

.. code-block:: python

   ipcontroller mpirun -np 4 ipengine
   mec = client.MultiEngineClient()
   mec.activate()

You have now obtained an object, ``mec``, which is able to interact with and
control all of the launched engines.  Any command prefixed with the string
``%px`` will now be issued on all processors.  Any action that would be
executed in parallel in ``yt`` will be executed in parallel here.  For
instance,

.. code-block:: python

   %px from yt.mods import *
   %px pf = load("data0050")
   %px pc = PlotCollection(pf)
   %px pc.add_projection("Density", 0)

This will load up the name space, the parameter file, and project through
``data0050`` in parallel utilizing all of our processors.  IPython can also
execute commands on a limited subset of hosts, and it can also turn on
auto-execution, to send all of your commands to all of the compute engines,
using the ``%autopx`` directive.
