Benchmarking addon
===================

Benchmarking addon is a tool that can be used for calculating metrics for
``parityos.ParityOSOutputs`` and ``parityos.api_interface.compiler_run.CompilerRuns``.
Here we will show how one can use the benchmarking addon.

Benchmarking ParityOSOutputs
-----------------------

For benchmarking we should choose a group of problems for which we will calculate the metrics.
For example we can consider all the 3-regular graphs with 8 nodes::

    from parityos import ProblemRepresentation
    from parityos import Qubit

    # here we have lists of interactions. [0, 1] means that we have interaction between
    # qubit 0 and Qubit 1
    three_regular_graphs = [
        [[0, 1], [0, 7], [0, 6], [1, 3], [1, 7], [2, 7],
         [2, 5], [2, 4], [3, 6], [3, 4], [5, 6], [5, 4]],
        [[4, 7], [4, 6], [4, 0], [7, 5], [7, 1], [1, 3],
         [1, 5], [3, 6], [3, 0], [2, 6], [2, 5], [2, 0]],
        [[0, 1], [0, 7], [0, 6], [1, 6], [1, 7], [4, 7],
         [4, 5], [4, 2], [5, 2], [5, 3], [6, 3], [2, 3]],
        [[1, 2], [1, 6], [1, 7], [2, 3], [2, 4], [4, 5],
         [4, 0], [5, 6], [5, 7], [6, 3], [7, 0], [0, 3]],
        [[2, 7], [2, 3], [2, 5], [7, 4], [7, 6], [0, 1],
         [0, 6], [0, 4], [1, 3], [1, 5], [4, 3], [6, 5]]
    ]

    three_regular_problems = []

    for graph in three_regular_graphs:
        problem = ProblemRepresentation(
            interactions=[[Qubit(vertex_1), Qubit(vertex_2)] for vertex_1, vertex_2 in graph],
            coefficients=[1]*len(graph),
        )
        three_regular_problems.append(problem)

So, these are our problems that we should first compile to obtain the corresponding list of
``ParityOSOutputs``. We will do compilation for the ``parityos.RectangularDigitalDevice``::

    from parityos import CompilerClient, RectangularDigitalDevice

    from parityos_addons.benchmarking import benchmark_parityos_outputs

    client = CompilerClient()
    digital_device = RectangularDigitalDevice(4, 4)

    digitial_three_regular_outputs = [
        client.compile(problem, digital_device) for problem in three_regular_problems
    ]

    parityos_output_stats = benchmark_parityos_outputs(
        digitial_three_regular_outputs,
        calculate_circuit_statistics=True,
    )

where ``digital_statistics`` ``pd.DataFrame`` is going to contain different metrics that describe
the ``ParityOSOutputs``. The list of metrics is shown below::

    [
        'parityos_outputs',
        'constraint_count',
        'square_constraint_count',
        'triangle_constraint_count',
        'compilation_qubit_count',
        'compilation_ancilla_count',
        'interaction_count',
        'constraint_circuit_depth',
        'constraint_circuit_gate_count',
        'constraint_circuit_two_body_gate_count',
        'constraint_circuit_cnot_count',
        'constraint_circuit_rzz_count',
        'constraint_circuit_qubit_count',
    ]

Benchmarking CompilerRuns
--------------------------------
For benchmarking ``CompilerRuns`` one should submit compilations asynchronously and get the
corresponding ``CompilerRuns``::

    import time
    from parityos import CompilerClient, RectangularDigitalDevice
    from parityos_addons.benchmarking import benchmark_compiler_runs

    client = CompilerClient()
    digital_device = RectangularDigitalDevice(4, 4)

    submission_ids = [client.submit(problem, digital_device) for problem in three_regular_problems]

    time.sleep(40)

    compiler_runs = [client.get_compiler_runs(submission_id)[0] for submission_id in submission_ids]

    compiler_runs_stats = benchmark_compiler_runs(client, compiler_runs)

The ``compiler_runs_stats`` contains in addition to all the metrics mentioned for the
``parityos_output_stats`` some extra metrics that are mentioned bellow::

    [
       'CompilerRun',
       'compilation_time',
       'COMPLETED',
       'FAILED',
       'RUNNING',
       'SUBMITTED',
    ]

Visualization
--------------------------------
To visualize the results one can use ``matplotlib`` library. Here is an example code that
one can use to plot a histogram for a chosen metric::

    import matplotlib.pyplot as plt

    # change the column (the list of the columns can be found above)
    # to get figures also for other columns:
    column = compiler_runs_stats.compilation_time

    title = f'statistics from {len(compiler_runs_stats)} problems'
    x_label = f'{column.name.replace("_", " ")}'
    y_label = 'problems'
    number_of_bins = 5

    plt.hist(column, number_of_bins)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.title(title)
    plt.show()

