.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "gallery/misc/example0_common.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_gallery_misc_example0_common.py: Getting Started ======================== This example illustrates basic usage of ``dman``. .. GENERATED FROM PYTHON SOURCE LINES 10-35 Overview --------------------------- The ``dman`` packages allows for convenient data storing and loading. The focus is on human readability, hence ``dman`` is mostly a file hierarchy manager. It does allow storing some types to files by default, but for others you have to define yourself how files are read from and written to disk. More details are provided in other example. For the full overview of introducing new types into ``dman`` see :ref:`sphx_glr_gallery_fundamentals_example1_storables.py` and :ref:`sphx_glr_gallery_fundamentals_example0_serializables.py`. In all likelihood however you will not need to do so too often. Especially if your data is represented in terms of ``numpy`` arrays. We will be using those in this example. This example starts with some basic ``python`` types and how to store them to disk. We then show how ``dman`` extends on these types, allowing for specifying file paths. Finally we show how ``modelclasses``, the ``dman`` extension of a ``dataclass`` can be used. If you want some code that can run out-of-the-box instead of a detailed introduction to the important components, you can take a look at the other examples listed under :ref:`sphx_glr_gallery_patterns`. To run this example you will require ``numpy`` and ``rich``. .. GENERATED FROM PYTHON SOURCE LINES 37-42 Basic types ---------------------------- You can store most basic ``python`` types. Specifically those that can be handled by ``json``. .. GENERATED FROM PYTHON SOURCE LINES 42-53 .. code-block:: default # clear any old data import dman, shutil, os if os.path.isdir(dman.mount()): shutil.rmtree(dman.mount()) config = {'mode': 'automatic', 'size': 5} data = [i**2 for i in range(config['size'])] dman.save('result', {'config': config, 'data': data}) .. rst-class:: sphx-glr-script-out .. code-block:: none {'config': {'mode': 'automatic', 'size': 5}, 'data': [0, 1, 4, 9, 16]} .. GENERATED FROM PYTHON SOURCE LINES 54-58 If you receive an error about ``.dman`` not existing. This means you have to create one by executing ``dman init`` in your terminal or creating a ``.dman`` folder manually in your project root. Files will, by default, be stored in this folder: .. GENERATED FROM PYTHON SOURCE LINES 58-61 .. code-block:: default dman.tui.walk_directory(dman.mount(), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common ┗━━ 📂 result ┗━━ 📄 result.json (148 bytes) ────────────────────────────────────────────────────────────────────── { "config": { "mode": "automatic", "size": 5 }, "data": [ 0, 1, 4, 9, 16 ] } ────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 62-63 By default ``dman`` can also handle ``numpy`` arrays. .. GENERATED FROM PYTHON SOURCE LINES 63-68 .. code-block:: default import numpy as np dman.save('result', {'config': config, 'data': np.arange(config['size'])**2}) dman.tui.walk_directory(dman.mount(), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common ┗━━ 📂 result ┗━━ 📄 result.json (176 bytes) ────────────────────────────────────────────────────────────────────── { "config": { "mode": "automatic", "size": 5 }, "data": { "_ser__type": "_num__ndarray", "_ser__content": "[0, 1, 4, 9, 16]" } } ────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 69-75 We mentioned that ``dman`` is a file hierarchy manager. Already some convenience is provided since we didn't need to specify a path for our data. This path has been determined automatically by :func:`dman.mount` internally based on the name of the script. Of course if you want to read the file from a different script this can be inconvenient. So you can specify the generator yourself. .. GENERATED FROM PYTHON SOURCE LINES 75-79 .. code-block:: default dman.save('result', 'content', generator='example_common') dman.tui.walk_directory(dman.mount(generator='example_common'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/example_common ┗━━ 📂 result ┗━━ 📄 result.json (9 bytes) ────────────────────────────────────────────────────────────────────── "content" ────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 80-85 The signature of :func:`dman.mount` is similar to that of :func:`dman.save`, :func:`dman.load` and :func:`dman.track`. Hence if you want to know where your files go, you can always use it. We can also load files of course: .. GENERATED FROM PYTHON SOURCE LINES 85-87 .. code-block:: default print(dman.load('result', generator='example_common')) .. rst-class:: sphx-glr-script-out .. code-block:: none content .. GENERATED FROM PYTHON SOURCE LINES 88-91 If the ``generator`` is not specified then loading only works when executed in the same script as the one where ``save`` was called with the default ``generator``. .. GENERATED FROM PYTHON SOURCE LINES 91-94 .. code-block:: default print(dman.load('result')) .. rst-class:: sphx-glr-script-out .. code-block:: none {'config': {'mode': 'automatic', 'size': 5}, 'data': array([ 0, 1, 4, 9, 16])} .. GENERATED FROM PYTHON SOURCE LINES 95-96 Finally we can do both at the same time .. GENERATED FROM PYTHON SOURCE LINES 96-102 .. code-block:: default dman.save('updated', {'original': 0}) with dman.track('updated', default_factory=dict) as data: data['value'] = 42 dman.tui.walk_directory(dman.mount('updated'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common/updated ┗━━ 📄 updated.json (38 bytes) ────────────────────────────────────────────────────────────────────────── { "original": 0, "value": 42 } ────────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 103-114 Creating a File Hierarchy ---------------------------- Beyond just automatically determining a ``mount`` point to save files to, ``dman`` also allows creating a file hierarchy within this folder. To do so we need to use model types. Let's start with the model version of dictionaries and of lists. We will also use ``barray``, which is the first ``storable`` type. It can be written and read from disk. The second storable is ``smdict``, which is simply a dictionary that will be stored to a separate file. .. GENERATED FROM PYTHON SOURCE LINES 114-125 .. code-block:: default from dman.numeric import barray config = dman.smdict.from_dict(config) data = (np.arange(config['size'])**2).view(barray) files = dman.mdict(store_by_key=True) files.update(config=config, data=data) dman.save('files', files) dman.tui.walk_directory(dman.mount('files'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common/files ┣━━ 📄 config.json (71 bytes) ┃ ────────────────────────────────────────────────────────────────────────── ┃ { ┃ "store": { ┃ "mode": "automatic", ┃ "size": 5 ┃ } ┃ } ┃ ────────────────────────────────────────────────────────────────────────── ┣━━ 📄 data.npy (168 bytes) ┗━━ 📄 files.json (578 bytes) ────────────────────────────────────────────────────────────────────────── { "_ser__type": "_ser__mdict", "_ser__content": { "store": { "config": { "_ser__type": "_ser__record", "_ser__content": { "target": "config.json", "sto_type": "_sto__smdict" } }, "data": { "_ser__type": "_ser__record", "_ser__content": { "target": "data.npy", "sto_type": "_num__barray" } } }, "store_by_key": true } } ────────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 126-134 Now three files have been created - ``files.json`` contains meta-data, describing the content of the other files. - ``config.json`` is what became of our ``config`` object. - ``data.npy`` stores the contents of ``data``. Let us consider a more interesting example, using :class:`dman.mruns`. This object acts like a list, but creates file names for storables automatically. .. GENERATED FROM PYTHON SOURCE LINES 134-142 .. code-block:: default with dman.track('runs', default_factory=dman.mruns_factory(store_subdir=False)) as runs: runs: dman.mruns = runs # for type hinting runs.clear() # remove all previous runs for i in range(3): runs.append(np.random.uniform(size=i).view(barray)) dman.tui.walk_directory(dman.mount('runs'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common/runs ┣━━ 📄 run-0.npy (128 bytes) ┣━━ 📄 run-1.npy (136 bytes) ┣━━ 📄 run-2.npy (144 bytes) ┗━━ 📄 runs.json (825 bytes) ────────────────────────────────────────────────────────────────────────── { "_ser__type": "_ser__mruns", "_ser__content": { "stem": "run", "run_count": 3, "store_subdir": false, "store": [ { "_ser__type": "_ser__record", "_ser__content": { "target": "run-0.npy", "sto_type": "_num__barray" } }, { "_ser__type": "_ser__record", "_ser__content": { "target": "run-1.npy", "sto_type": "_num__barray" } }, { "_ser__type": "_ser__record", "_ser__content": { "target": "run-2.npy", "sto_type": "_num__barray" } } ] } } ────────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 143-144 If you don't care about file names, ``dman`` can generate them automatically: .. GENERATED FROM PYTHON SOURCE LINES 144-150 .. code-block:: default with dman.track('auto', default_factory=list) as lst: lst.clear() lst.extend([np.random.uniform(size=i).view(barray) for i in range(3)]) dman.tui.walk_directory(dman.mount('auto'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common/auto ┣━━ 📄 0cdfb3bf-799d-4ef8-998c-f6f7955ddb42.npy (144 bytes) ┣━━ 📄 307e91fd-c680-4718-85fa-8efdcd043dea.npy (136 bytes) ┣━━ 📄 6c28b730-efe5-4247-8e13-fd2d1e429967.npy (128 bytes) ┗━━ 📄 auto.json (840 bytes) ────────────────────────────────────────────────────────────────────────── { "_ser__type": "_ser__mlist", "_ser__content": { "store": [ { "_ser__type": "_ser__record", "_ser__content": { "target": "6c28b730-efe5-4247-8e13-fd2d1e429967.npy", "sto_type": "_num__barray" } }, { "_ser__type": "_ser__record", "_ser__content": { "target": "307e91fd-c680-4718-85fa-8efdcd043dea.npy", "sto_type": "_num__barray" } }, { "_ser__type": "_ser__record", "_ser__content": { "target": "0cdfb3bf-799d-4ef8-998c-f6f7955ddb42.npy", "sto_type": "_num__barray" } } ] } } ────────────────────────────────────────────────────────────────────────── .. GENERATED FROM PYTHON SOURCE LINES 151-167 .. warning:: Both specifying file names and having them be automatically generated have advantages and disadvantages. When specifying the file names you risk overwriting existing data, however ``dman`` will give a warning by default. See :ref:`sphx_glr_gallery_fundamentals_example4_path.py` for more info. Importantly if you want ``dman`` to prompt you for a new filename whenever it risks overwriting an existing file use: .. code-block:: python dman.params.store.on_retouch = 'prompt' When not specifying file names, files will likely not be removed. Instead ``dman`` keeps creating new files (unless if you use ``track`` correctly as illustrated above). .. GENERATED FROM PYTHON SOURCE LINES 169-173 Modelclasses --------------------- We finally briefly illustrate the usage of ``modelclass``. .. GENERATED FROM PYTHON SOURCE LINES 173-189 .. code-block:: default from dman.numeric import sarray @dman.modelclass(compact=True, storable=True) class Config: description: str size: int @dman.modelclass(storable=True, store_by_field=True) class Data: values: sarray[int] output: barray = None cfg = Config('Experiment generating numbers', 25) data = Data(np.logspace(0, 3, cfg.size)) data.output = np.random.uniform(size=cfg.size) .. GENERATED FROM PYTHON SOURCE LINES 190-192 The ``modelclass`` automatically converts the numpy arrays to the specified types: .. GENERATED FROM PYTHON SOURCE LINES 192-199 .. code-block:: default print( f'{type(data.values)=}', f'{type(data.values[0])=}', f'{type(data.output)=}', sep='\n' ) .. rst-class:: sphx-glr-script-out .. code-block:: none type(data.values)= type(data.values[0])= type(data.output)= .. GENERATED FROM PYTHON SOURCE LINES 200-201 We can now save the result .. GENERATED FROM PYTHON SOURCE LINES 201-207 .. code-block:: default dman.save( 'model', dman.mdict.from_dict({'cfg': cfg, 'data': data}, store_by_key=True) ) dman.tui.walk_directory(dman.mount('model'), show_content=True) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 .dman/cache/examples:misc:example0_common/model ┣━━ 📄 cfg.json (70 bytes) ┃ ────────────────────────────────────────────────────────────────────────── ┃ { ┃ "description": "Experiment generating numbers", ┃ "size": 25 ┃ } ┃ ────────────────────────────────────────────────────────────────────────── ┣━━ 📄 data.json (366 bytes) ┃ ────────────────────────────────────────────────────────────────────────── ┃ { ┃ "values": { ┃ "_ser__type": "_num__sarray", ┃ "_ser__content": "[1, 1, 1, 2, 3, 4, 5, 7, 10, 13, 17, 23, 31, 42, 5 ┃ }, ┃ "output": { ┃ "_ser__type": "_ser__record", ┃ "_ser__content": { ┃ "target": "output.npy", ┃ "sto_type": "_num__barray" ┃ } ┃ } ┃ } ┃ ────────────────────────────────────────────────────────────────────────── ┣━━ 📄 model.json (559 bytes) ┃ ────────────────────────────────────────────────────────────────────────── ┃ { ┃ "_ser__type": "_ser__mdict", ┃ "_ser__content": { ┃ "store": { ┃ "cfg": { ┃ "_ser__type": "_ser__record", ┃ "_ser__content": { ┃ "target": "cfg.json", ┃ "sto_type": "Config" ┃ } ┃ }, ┃ "data": { ┃ "_ser__type": "_ser__record", ┃ "_ser__content": { ┃ "target": "data.json", ┃ "sto_type": "Data" ┃ } ┃ } ┃ }, ┃ "store_by_key": true ┃ } ┃ } ┃ ────────────────────────────────────────────────────────────────────────── ┗━━ 📄 output.npy (328 bytes) .. GENERATED FROM PYTHON SOURCE LINES 208-209 More information on how to use ``modelclass`` can be found in :ref:`sphx_glr_gallery_fundamentals_example3_models.py`. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.090 seconds) .. _sphx_glr_download_gallery_misc_example0_common.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: example0_common.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: example0_common.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_