Metadata-Version: 2.4
Name: sop-python-beta-3
Version: 3.0.8
Summary: Scalable Objects Persistence for Python.
Author-email: Gerardo Recinto <gerardorecinto@yahoo.com>
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown

# What is SOP?

Scalable Objects Persistence (SOP) is a raw storage engine that bakes together a set of storage related features & algorithms in order to provide the most efficient & reliable (ACID attributes of transactions) technique (known) of storage management and rich search, as it brings to the application, the raw muscle of "raw storage", direct IO communications w/ disk drives. In a code library form factor.

Other "key advances" in database technology available in this SOP release:
* Serverless (or all servers!) operations mode in the cluster, i.e. - your applications that use SOP library are server-less
* Uses a new storage & L1/L2 caching strategy that is ground breaking in performance & efficiency
* New superfast realtime Orchestration engine fit for database transactions
* Support for small, medium to large/very large (multi-GBs/TB) data management
* Sports advanced/efficient/high tolerance software based (via Erasure Coding) replication
* Horizontal & Vertical scaling in storage and cluster processes
* New database storage strategy that uses segment files which avoids having to manage a huge file, resulting in higher IO efficiency & file system/disk drive friendly (medium sized) data files.
* Built-in data caching, your application data can be cached automatically via config setting
* Host your applications across platforms, e.g. - microservice cluster running in Linux, another cluster or instance running in Windows, and all SOPs inter-operating with one another seamlessly and data stored in same set of disk drives. Freedom to use popular hardware architecture & OS of your choice(s)!

# SOP supported Hardware/OS
SOP supports popular architectures & Operating Systems such as Linux, Darwin & Microsoft Windows, in both ARM64 & AMD64 architectures. For Windows, only AMD64 is supported since it is the only architecture Windows is available in.

# SOP Dependencies
* Redis, you will need to have one of the recent or latest version of Redis for use in SOP caching.
* More than one Disk Drives(recommended is around four or more, for replication) with plenty of drive space available, for storage management. Example:
```
    /disk1
    /disk2
    /disk3
    /disk4
```

# SOP for Python package
Following steps outlines how to use the Scalable Objects Persistence code library for Python:
* Install the package using: pip install sop-python-beta-3
* Follow standard Python package import and start coding to use the SOP for Python code library for data management. Import the sop package in your python code file.
* Specify Home base folders where Store info & Registry data files will be stored.
* Specify Erasure Coding (EC) configuration details which will be used by SOP's EC based replication.
* Open the global Redis connection
* Create a transaction
* Begin a transaction
* Create a new B-tree, or Open an existing B-tree
* Manage data, do some CRUD operations
* Commit the transaction

Below is an example code black for illustrating the above steps. For other SOP B-tree examples, you can checkout the code in the unit tests test_btree.py & test_btree_idx.py files that comes w/ the SOP package you downloaded from pypi.

```
from sop import transaction
from sop import btree
from sop import context
from sop import redis

stores_folders = ("/disk1", "/disk2")
ec = {
    # Erasure Config default entry(key="") will allow different B-tree(tables) to share same EC structure.
    "": transaction.ErasureCodingConfig(
        2,  # two data shards
        2,  # two parity shards
        (
            # 4 disk drives paths
            "/disk1",
            "/disk2",
            "/disk3",
            "/disk4",
        ),
        # False means Auto repair of failed reads from (shards') disk drive will not get repaired.
        False,
    )
}

# Transaction Options (to).
to = transaction.TransationOptions(
    transaction.TransactionMode.ForWriting.value,
    # commit timeout of 15mins
    15,
    # Min Registry hash mod value is 250, you can specify higher value like 1000. A 250 hashmod
    # will use 1MB sized file segments. Good for demo, but for Prod, perhaps a bigger value is better.
    transaction.MIN_HASH_MOD_VALUE,
    # Store info & Registry home base folders. Array of strings of two elements, one for Active & another, for passive folder.
    stores_folders,
    # Erasure Coding config as shown above.
    ec,
)

# Context object.
ctx = context.Context()

# initialize/open SOP global Redis connection. You can specify your Redis cluster host address(es) & port, etc
# but defaults to localhost and default Redis port # if none is specified.
ro = redis.RedisOptions()
redis.Redis.open_connection(ro)

t = transaction.Transaction(ctx, to)
t.begin()

cache = btree.CacheConfig()

# "barstoreec" is new b-tree name, 2nd parameter set to True specifies B-tree Key field to be native data type
bo = btree.BtreeOptions("barstoreec", True, cache_config=cache)
bo.set_value_data_size(btree.ValueDataSize.Small)

# create the new "barstoreec" b-tree store. Once created, you can just use the "open" method. Perhaps
# you can have a code set for creating B-trees like "admin only" script. Then all other app code uses
# the "open" method.
b3 = btree.Btree.new(ctx, bo, t)

# Since we've specified Native data type = True in BtreeOptions, we can use "integer" values as Key.
l = [
    btree.Item(1, "foo"),
]

# Add Item to the B-tree.
b3.add(ctx, l)

# Commit the transaction to finalize the new B-tree (store) change.
t.commit(ctx)
print("ended.")
```

# SOP in Github
SOP open source project (MIT license) is in github. You can checkout the "...sop/jsondb/" package which contains the Go code enabling general purpose JSON data management & the Python wrapper, coding guideline of which, was described above.

Please feel free to join the SOP project if you have the bandwidth and participate/co-own/lead! the project engineering.
