.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "gallery/fundamentals/example2_records.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_gallery_fundamentals_example2_records.py: Using Records ======================== We show how to use ``records`` to serialize storables. .. GENERATED FROM PYTHON SOURCE LINES 9-11 Basic usage ------------------------- .. GENERATED FROM PYTHON SOURCE LINES 11-24 .. code-block:: default # We will be using ``barray`` for this example so you should have ``numpy`` # installed. from tempfile import TemporaryDirectory import dman from dman.numeric import barray from dman import tui import numpy as np # turn of logging dman.log.default_config(level=dman.log.CRITICAL) .. GENERATED FROM PYTHON SOURCE LINES 25-26 By default a ``barray`` object is not serializable: .. GENERATED FROM PYTHON SOURCE LINES 26-30 .. code-block:: default array = np.arange(3).view(barray) ser = dman.serialize(array) tui.print_json(dman.sjson.dumps(ser, indent=4)) .. rst-class:: sphx-glr-script-out .. code-block:: none { "_ser__type": "__unserializable", "_ser__content": { "type": "barray", "info": "Unserializable type: barray." } } .. GENERATED FROM PYTHON SOURCE LINES 31-35 Note how ``dman`` does not throw an error here. This is to make sure that as much data is serialized as possible. You can turn on validation by setting ``dman.params.serialize.validate=True``. Further details are provided in :ref:`sphx_glr_gallery_fundamentals_example5_errors.py` .. GENERATED FROM PYTHON SOURCE LINES 37-47 Contexts will also be useful for storing purposes. They specify the directory in which files should be stored during serialization. To make sure a ``storable`` can be serialized it should be wrapped in a ``record``. This interface has a the following features: * File names and extensions can be specified manually or created automatically. * Sub folders can be specified. * Reading the object from file can be delayed until the content is accessed. The most basic usage is as follows: .. GENERATED FROM PYTHON SOURCE LINES 47-55 .. code-block:: default tdir = TemporaryDirectory() base = tdir.name ctx = dman.Context.from_directory(base) rec = dman.record(array) ser = dman.serialize(rec, context=ctx) tui.print_json(dman.sjson.dumps(ser, indent=4)) .. rst-class:: sphx-glr-script-out .. code-block:: none { "_ser__type": "_ser__record", "_ser__content": { "target": "8aa9fac7-f0b9-4462-a4e5-4cf81acf8966.npy", "sto_type": "_num__barray" } } .. GENERATED FROM PYTHON SOURCE LINES 56-58 You can see that the result of serialization now provides a pointer to the file where the array is stored. We can see that the file exists: .. GENERATED FROM PYTHON SOURCE LINES 58-61 .. code-block:: default tui.walk_directory(base) .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 /tmp/tmp5k2r9725 ┗━━ 📄 8aa9fac7-f0b9-4462-a4e5-4cf81acf8966.npy (152 bytes) .. GENERATED FROM PYTHON SOURCE LINES 62-63 And we can load its content again .. GENERATED FROM PYTHON SOURCE LINES 63-67 .. code-block:: default rec = dman.deserialize(ser, context=ctx) tui.print(rec) .. rst-class:: sphx-glr-script-out .. code-block:: none Record(UL[_num__barray], target=8aa9fac7-f0b9-4462-a4e5-4cf81acf8966.npy) .. GENERATED FROM PYTHON SOURCE LINES 68-72 Note how the record specifies that it contains a ``_num__barray`` which is the name for the storable type. However it also specifies ``UL`` implying that the file has not been loaded. We can load it by accessing the ``content`` field: .. GENERATED FROM PYTHON SOURCE LINES 72-77 .. code-block:: default array = rec.content tui.print(array) tui.print(rec) .. rst-class:: sphx-glr-script-out .. code-block:: none [0 1 2] Record(_num__barray, target=8aa9fac7-f0b9-4462-a4e5-4cf81acf8966.npy) .. GENERATED FROM PYTHON SOURCE LINES 78-83 Now the record no longer specifies that the ``content`` is unloaded. Also observe that the file name is still the same as the one specified in the original record. This means that when serializing again the old file will be overwritten instead of creating a new one. We can also remove the file: .. GENERATED FROM PYTHON SOURCE LINES 83-88 .. code-block:: default dman.remove(rec, context=ctx) tui.walk_directory(base) tdir.cleanup() # clean temporary directory .. rst-class:: sphx-glr-script-out .. code-block:: none 📂 /tmp/tmp5k2r9725 .. GENERATED FROM PYTHON SOURCE LINES 89-143 It is possible to be more precise when specifying a ``record``. To give an overview of the options available when creating a record we provide its documentation: .. autofunction:: dman.record :noindex: The way file names are specified is left flexible for internal use, but is hence somewhat complex. We list examples below. ================================================ ========================= options target ================================================ ========================= ``stem='test'`` ``./test`` ``stem='test', suffix='.txt'`` ``./test.txt`` ``name='test.txt'`` ``./test.txt`` ``name='test.txt', subdir='dir'`` ``./dir/test.txt`` ``name='test.txt', stem='test', suffix='.txt'`` ``ValueError`` ================================================ ========================= More details are provided in :ref:`sphx_glr_gallery_fundamentals_example4_path.py`, where `targets` are introduced. .. note:: It is also possible to automatically determine the ``suffix`` based on the class. .. code-block:: python @storable(name='manual') class ManualFile: __ext__ = '.obj' ... So if only a ``stem=test`` is specified the target will automatically become ``test.obj``. If a ``suffix`` is specified anyway, then the one specified through ``__ext__`` is overridden. When a ``storable`` is automatically created from a ``dataclass`` or a ``serializable`` the ``suffix`` will be set to ``.json`` by default. .. warning:: Be careful specifying the ``stem`` of a file. It often makes sense to omit it and leave the selection up to the ``record``. That way you will not accidentally re-use existing files. When a file is re-used, ``dman`` can automatically handle this when configured to do so or, by default, it gives a warning. See :ref:`sphx_glr_gallery_fundamentals_example4_path.py` for details. .. GENERATED FROM PYTHON SOURCE LINES 146-155 Contexts ----------------------- Sometimes storable objects can create more than one file. The types provided by ``dman`` that do so are referred to as ``models``. See :ref:`sphx_glr_gallery_fundamentals_example3_models.py` for examples. The files created by these models should also be stored somewhere. Their path is determined relative to the root storable. This process is handled by the context. As we will illustrate below: .. GENERATED FROM PYTHON SOURCE LINES 155-166 .. code-block:: default @dman.storable class Feedback: def __write__(self, path: str, context): print(context) @classmethod def __read__(cls, path: str): return cls() _ = dman.serialize(dman.record(Feedback(), subdir='folder'), context=ctx) .. rst-class:: sphx-glr-script-out .. code-block:: none Context(/tmp/tmp5k2r9725/folder) .. GENERATED FROM PYTHON SOURCE LINES 167-172 As you can see the context received in the ``__write__`` method now keeps track of the subfolder. Hence any further serializations happen relative to it. We can keep going: .. GENERATED FROM PYTHON SOURCE LINES 172-184 .. code-block:: default @dman.storable class SubSerialize: def __write__(self, path: str, context): dman.serialize(dman.record(Feedback(), subdir='folder2'), context=context) @classmethod def __read__(cls, path: str): return cls() _ = dman.serialize(dman.record(SubSerialize(), subdir='folder'), context=ctx) .. rst-class:: sphx-glr-script-out .. code-block:: none Context(/tmp/tmp5k2r9725/folder/folder2) .. GENERATED FROM PYTHON SOURCE LINES 185-191 This is used inside of model types to determine the file paths. We will not go into much more detail here and instead refer to :ref:`sphx_glr_gallery_fundamentals_example3_models.py`. Whenever you create additional files you should also add a ``__remove__`` method. See the end of :ref:`sphx_glr_gallery_fundamentals_example1_storables.py` for more details on this topic. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.016 seconds) .. _sphx_glr_download_gallery_fundamentals_example2_records.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: example2_records.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: example2_records.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_