Metadata-Version: 2.4
Name: shared-ndarray2
Version: 2.0.2
Summary: Interface for NumPy ndarray using multiprocessing SharedMemory
Project-URL: Repository, https://gitlab.com/osu-nrsg/shared-ndarray2
Project-URL: Issues, https://gitlab.com/osu-nrsg/shared-ndarray2/-/issues
Project-URL: Changelog, https://gitlab.com/osu-nrsg/shared-ndarray2/-/blob/master/CHANGELOG.md
Author-email: Randall Pittman <randall.pittman@oregonstate.edu>
License: MIT License
        
        Copyright (c) 2021-2025 Randall Pittman, Oregon State University
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: ndarray,numpy,shared_memory
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development
Requires-Python: >=3.8
Requires-Dist: numpy>=1.21.1
Requires-Dist: packaging>=25.0
Requires-Dist: single-version>=1.5.1
Description-Content-Type: text/markdown

# shared-ndarray2  <!-- omit in toc -->

`SharedNDArray` encapsulates a NumPy `ndarray` interface for using shared memory in
multiprocessing, using `multiprocessing.shared_memory` in Python 3.8+.

- [Quick Start](#quick-start)
- [Requirements](#requirements)
- [Similar Projects](#similar-projects)
- [Usage](#usage)
  - [Creation](#creation)
    - [`SharedNDArray()`](#sharedndarray)
    - [`SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`](#sharedndarrayfrom_shape-or-shared_ndarrayfrom_shape)
    - [`SharedNDArray.from_array()` or `shared_ndarray.from_array()`](#sharedndarrayfrom_array-or-shared_ndarrayfrom_array)
  - [Using like `np.ndarray`](#using-like-npndarray)
  - [Releasing Shared Memory](#releasing-shared-memory)
  - [`.lock` attribute](#lock-attribute)
  - [Typed SharedNDArray](#typed-sharedndarray)
- [MyPy and NumPy typing compatibility](#mypy-and-numpy-typing-compatibility)

## Quick Start

```python
import multiprocessing
from multiprocessing.managers import SharedMemoryManager

import numpy as np

from shared_ndarray2 import SharedNDArray


def process_data(arr: SharedNDArray):
    # Work with data
    arr[:] += 1


with SharedMemoryManager() as mem_mgr:
    arr = SharedNDArray.from_array(mem_mgr, np.arange(1024))
    p = multiprocessing.Process(target=process_data, args=(arr,))
    p.start()
    p.join()
    assert np.all(arr[:] == np.arange(1, 1025))
```

## Requirements

- Python 3.8+
- NumPy 1.21+

## Similar Projects

- [SharedArray](https://pypi.org/project/SharedArray/) - POSIX-only. Quite a different
  paradigm, uses pre-Python 3.8 memory-sharing constructs, requires building a C module
  with `gcc`.
- [shared-ndarray](https://pypi.org/project/shared-ndarray/) - POSIX-only. Similar (uses
  NumPy ndarray `buffer` arg), uses pre-Python 3.8 memory-sharing constructs (requires
  `posix_ipc`).

## Usage

### Creation

There are three methods for constructing a `SharedNDArray`.

#### `SharedNDArray()`

To create a `SharedNDArray` object from existing shared memory that represents a NumPy
array, use the regular constructor providing _shape_ and _dtype_, either with an existing
`multiprocessing.SharedMemory` object or the name of one:

```python
shm = SharedMemory(create=True, size=1024)
arr = SharedNDArray(shm, (1024,), np.uint8)
# -or-
arr = SharedNDArray(shm.name, (1024,), np.uint8)
```

#### `SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`

This method allocates shared memory managed by a SharedMemoryManager to represent a NumPy
`ndarray` with some _shape_ and _dtype_.

```python
with SharedMemoryManager as mem_mgr:
    arr = SharedNDArray.from_shape(mem_mgr, (3, 1024, 1024), dtype=np.uint16)
    # ... Use arr with e.g. multiprocessing.Pool or multiprocess.Process
    # ... Be sure process instances join/terminate before exiting SharedMemoryManager context manager
```

<div style="font-size: small; margin-left: 4em">

Note: _`shared_ndarray.from_shape()` is a standlone function and correctly types the
`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_

</div>

#### `SharedNDArray.from_array()` or `shared_ndarray.from_array()`

This method allocates shared memory managed by a SharedMemoryManager to represent a some
provided NumPy `ndarray` and copies that ndarray into the shared memory

```python
x = np.arange(100.0).reshape(2, 2, 25)
with SharedMemoryManager as mem_mgr:
    arr = SharedNDArray.from_array(mem_mgr, x)
    assert np.all(arr[:] == x[:])
    # ... Use arr as above...
```

<div style="font-size: small; margin-left: 4em">

Note: _`shared_ndarray.from_array()` is a standlone function and correctly types the
`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_

</div>

### Using like `np.ndarray`

The point of `SharedNDArray` is to remove the boilerplate of creating shared memory,
passing around shapes and dtypes and reconstructing `np.ndarray` objects. `SharedNDArray`
does this last step with its `.get()` method, which creates a `np.ndarray` on-the-fly
using the shared memory buffer. The `__getitem__()` and `__setitem__()` methods use the
`.get()` method to get the np.ndarray to access the data, so multi-dimensional indexing
and slicing work the same as with an `ndarray`. Other `np.ndarray` methods are not
directly implemented but may be accessed by first calling `.get()`, e.g.
`arr.get().mean()`.

### Releasing Shared Memory

`SharedNDArray` implements a `__del__()` method that calls the
[`.close()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.close)
method on the `SharedMemory` when the instance is destroyed (i.e. at process exit). When
the shared memory is unlinked in the parent process (either manually with
[`shm.unlink()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.unlink)
or by exiting a `SharedMemoryManager` context manager) the shared_memory is properly
released. However if a sub-process is not joined or terminated before the shared memory is
unlinked a warning will be emitted about "`leaked shared_memory objects to clean up at
shutdown`".

### `.lock` attribute

The `__init__()`, `from_shape()`, and `from_array()` methods may be given a `lock=True`
argument that will also create a `multiprocessing.Lock` object and include it in the
`SharedNDArray`, accesible as the `.lock` attribute. It should be noted, however, that it
doesn't work well to pass a `multiprocessing.Lock` as an argument to a
`multiprocessing.Pool` function, for reasons described
[here](https://stackoverflow.com/questions/25557686/python-sharing-a-lock-between-processes#comment72803059_25558333).
Thus by default `.lock` is set to `None`.

### Typed SharedNDArray

`SharedNDArray` is able to be typed with NumPy types. When using the `from_array()`
constructor, it is also able to inherit the type of the `ndarray` if it is typed using
`numpy.typing.NDArray` (new in NumPy 1.21). Typing information does not pass through with
slicing (`__getitem__`), however.

```python
x: npt.NDArray[np.int_] = np.arange(1024)
arr = SharedNDArray(mem_mgr, x)  # type of x is SharedNDArray[int_]
arr2 = arr[:]  # arr2 is typing.Any
```

## MyPy and NumPy typing compatibility

This package includes type annotations and the `py.typed` marker file.

Due to the use of a private NumPy type annoation whose location moved in NumPy 1.23.0,
mypy has to be configured differently if using NumPy < 1.23.0.

| NumPy Version | mypy config                 | mypy cli                    | Pyright config                              |
|---------------|-----------------------------|-----------------------------|---------------------------------------------|
| < 1.23.0      | `always_false = NUMPY_1_23` | `--always-false NUMPY_1_23` | `defineConstant = { "NUMPY_1_23" = false }` |
| 1.23.0+       | `always_true = NUMPY_1_23`  | `--always-true NUMPY_1_23`  | `defineConstant = { "NUMPY_1_23" = true }`  |
