.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "gallery/fundamentals/example6_errors.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_gallery_fundamentals_example6_errors.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_gallery_fundamentals_example6_errors.py:


Error handling
========================

We show how errors are handled within ``dman``.

.. GENERATED FROM PYTHON SOURCE LINES 9-12

Introduction
-----------------
To run the example you will need the following imports:

.. GENERATED FROM PYTHON SOURCE LINES 12-24

.. code-block:: default


    from dataclasses import dataclass
    import sys
    from tempfile import TemporaryDirectory
    import traceback
    import dman
    from dman import tui
    from dman import log

    # turn off warnings
    log.default_config(level=log.CRITICAL)  








.. GENERATED FROM PYTHON SOURCE LINES 25-29

Since `dman` is designed to store results generated by experiments, it 
always attempts to store as much information as possible. This implies 
that exceptions are handled internally. Often such exceptions 
can occur when encountering unserializable types.

.. GENERATED FROM PYTHON SOURCE LINES 29-48

.. code-block:: default


    @dataclass
    class Address:
        street: str
        number: int
        zip_code: int
        city: str
        country: str

        def __str__(self):
            return f'{self.street} {self.number}, {self.zip_code} {self.city}, {self.country}'

    home = Address('Kasteelpark Arenberg', 10, 3001, 'Leuven', 'Belgium')
    employee = {'name': 'John Doe', 'home': home}

    dman.save('employee', employee)
    tui.walk_directory(dman.mount('employee'), show_content=True)






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    📂 .dman/cache/examples:fundamentals:example6_errors/employee
    ┗━━ 📄 employee.json (209 bytes)
         ────────────────────────────────────────────────────────────────────────── 
          {                                                                         
            "name": "John Doe",                                                     
            "home": {                                                               
              "_ser__type": "__unserializable",                                     
              "_ser__content": {                                                    
                "type": "Address",                                                  
                "info": "Unserializable type: Address."                             
              }                                                                     
            }                                                                       
          }                                                                         
         ────────────────────────────────────────────────────────────────────────── 




.. GENERATED FROM PYTHON SOURCE LINES 49-52

As you can see only the serializable parts of the data structure were stored. 
The address has been replaced with a specification of the issue encountered.
When we load the employee from disk again we get the following:

.. GENERATED FROM PYTHON SOURCE LINES 52-56

.. code-block:: default


    recovered = dman.load('employee')
    print(recovered['home'])





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Unserializable: Address
        Unserializable type: Address.




.. GENERATED FROM PYTHON SOURCE LINES 57-58

Usually a warning is raised when serialization fails:

.. GENERATED FROM PYTHON SOURCE LINES 58-62

.. code-block:: default


    with dman.log.logger_context(level=log.WARNING):
        dman.serialize(employee)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [01/04/23 10:12:24] WARNING  [@dict.Address | context]:     serializables.py:629
                                 Serialization resulted in an                       
                                 invalid object.                                    
                                 Unserializable: Address                            
                                     Unserializable type:                           
                                 Address.                                           




.. GENERATED FROM PYTHON SOURCE LINES 63-81

The warning will tell you where the serialization failed. In this case 
it specifies `@dict.Address` to indicate that the `Address` object 
was stored in a dictionary. 

.. note::
      If you do want an exception thrown you can do get one as follows:

      .. code-block:: python

          try:
              dman.serialize(employee, context=dman.BaseContext(validate=True))
          except dman.ValidationError as e:
              msg = traceback.format_exception(*sys.exc_info())
              print(''.join(msg))

      This is only recommended during debugging, since it interrupts 
      serialization while it is in progress and can therefore result 
      in loss of large amounts of data.

.. GENERATED FROM PYTHON SOURCE LINES 83-93

Usually the log messages should provide enough information to 
fix any issues. We nonetheless continue this example with more 
detailed examples describing the different things that can go wrong during 
serialization and what you can expect as output. 
We also illustrate how data can be recovered even if initial errors 
were present in the implementation.

We will be doing so in terms of the fundamental datastructures involved 
in `dman`. To understand how these fit into the bigger picture 
take a look at the other examples.

.. GENERATED FROM PYTHON SOURCE LINES 96-100

Serializables
---------------------------------------
Serialization
^^^^^^^^^^^^^^^^^

.. GENERATED FROM PYTHON SOURCE LINES 102-103

We first serialize an unserializable type:

.. GENERATED FROM PYTHON SOURCE LINES 103-113

.. code-block:: default



    class Base:
        ...


    ser = dman.serialize(Base())
    tui.print_serialized(ser)






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "__unserializable",
      "_ser__content": {
        "type": "Base",
        "info": "Unserializable type: Base."
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 114-116

Next let's try serialization of an object with an erroneous implementation
of `__serialize__`.

.. GENERATED FROM PYTHON SOURCE LINES 116-131

.. code-block:: default



    @dman.serializable(name="base")
    class Base:
        def __serialize__(self):
            raise RuntimeError("Invalid implementation")

        @classmethod
        def __deserialize__(cls, ser):
            ...


    ser = dman.serialize(Base())
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "__exc_unserializable",
      "_ser__content": {
        "type": "Base",
        "info": "Error during serialization:",
        "trace": [
          {
            "exc": "RuntimeError(Invalid implementation)",
            "frames": [
              "File 
    /home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 121, 
    in __serialize__: raise RuntimeError(\"Invalid implementation\")"
            ]
          }
        ]
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 132-133

After deserialization we can still view the traceback resulting in the exception:

.. GENERATED FROM PYTHON SOURCE LINES 133-136

.. code-block:: default

    err = dman.deserialize(ser)
    print(err)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ExcUnserializable: Base
        Error during serialization:
    Traceback (most recent call last):
      File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 121, in __serialize__
        raise RuntimeError("Invalid implementation")
    RuntimeError: Invalid implementation




.. GENERATED FROM PYTHON SOURCE LINES 137-138

We also detect `__serialize__` methods with invalid signatures.

.. GENERATED FROM PYTHON SOURCE LINES 138-153

.. code-block:: default


    @dman.serializable(name="base")
    class Base:
        # `__serialize__` should either have no or two arguments besides `self`.
        def __serialize__(self, arg1, arg2):
            return 'Base'

        @classmethod
        def __deserialize__(cls, ser):
            raise RuntimeError('Invalid implementation')


    ser = dman.serialize(Base())
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "__exc_unserializable",
      "_ser__content": {
        "type": "Base",
        "info": "Error during serialization:",
        "trace": [
          {
            "exc": "TypeError(Expected method that takes 0 or 1 positional arguments
    but got 2.)",
            "frames": []
          }
        ]
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 154-157

Deserialization
^^^^^^^^^^^^^^^^^
Now we move on to deserialization.

.. GENERATED FROM PYTHON SOURCE LINES 157-172

.. code-block:: default



    @dman.serializable(name="base")
    class Base:
        def __serialize__(self):
            return '<base>'

        @classmethod
        def __deserialize__(cls, ser):
            raise RuntimeError('Invalid implementation')


    ser = dman.serialize(Base())
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "base",
      "_ser__content": "<base>"
    }




.. GENERATED FROM PYTHON SOURCE LINES 173-174

Even though the serialized object is valid, we cannot recover the object.

.. GENERATED FROM PYTHON SOURCE LINES 174-178

.. code-block:: default


    err = dman.deserialize(ser)
    print(err)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ExcUndeserializable: base
        Exception encountered while deserializing <class '__main__.Base'>:
    Traceback (most recent call last):
      File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 166, in __deserialize__
        raise RuntimeError('Invalid implementation')
    RuntimeError: Invalid implementation

    Serialized
    "<base>"




.. GENERATED FROM PYTHON SOURCE LINES 179-181

Note how the error contains information about what was serialized.
The error is serializable.

.. GENERATED FROM PYTHON SOURCE LINES 181-184

.. code-block:: default

    ser = dman.serialize(err)
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "__exc_undeserializable",
      "_ser__content": {
        "type": "base",
        "info": "Exception encountered while deserializing <class 
    '__main__.Base'>:",
        "trace": [
          {
            "exc": "RuntimeError(Invalid implementation)",
            "frames": [
              "File 
    /home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 166, 
    in __deserialize__: raise RuntimeError('Invalid implementation')"
            ]
          }
        ],
        "serialized": "<base>",
        "expected": "base"
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 185-187

Moreover when we try to deserialize it again, now with a valid 
class definition, things work. This sometimes allows for data restoration.

.. GENERATED FROM PYTHON SOURCE LINES 187-201

.. code-block:: default



    @dman.serializable(name="base")
    class Base:
        def __serialize__(self):
            return '<base>'

        @classmethod
        def __deserialize__(cls, ser):
            return cls()

    base = dman.deserialize(ser)
    print(base)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <__main__.Base object at 0x7fe4a1bf7b50>




.. GENERATED FROM PYTHON SOURCE LINES 202-211

Records
---------------------------------------

Since a `record` is also serializable it should suppress errors as well
such that the data structure is not damaged by a single erroneous item.
We provide an overview of the errors that typically occur 
and how they are represented within the result of the serialization.

Let's begin with a storable class that cannot write to disk

.. GENERATED FROM PYTHON SOURCE LINES 211-222

.. code-block:: default

    @dman.storable(name='base')
    class Base:
        def __write__(self, path: str):
            raise RuntimeError('Cannot write to disk.')

        @classmethod
        def __read__(cls, path: str):
            return cls()

    rec = dman.record(Base())








.. GENERATED FROM PYTHON SOURCE LINES 223-224

If we try to serialize the object without a context we run into some issues

.. GENERATED FROM PYTHON SOURCE LINES 224-227

.. code-block:: default

    ser = dman.serialize(rec)
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "_ser__record",
      "_ser__content": {
        "target": "db45b45e-79b5-4cee-9fb7-b84e76457603.sto",
        "sto_type": "base",
        "exceptions": {
          "write": {
            "_ser__type": "__un_writable",
            "_ser__content": {
              "type": "base",
              "info": "Invalid context passed to Record(base, 
    target=db45b45e-79b5-4cee-9fb7-b84e76457603.sto)."
            }
          }
        }
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 228-229

And when serializing it with a context we run into another.

.. GENERATED FROM PYTHON SOURCE LINES 229-234

.. code-block:: default

    root = TemporaryDirectory()
    ctx = dman.Context.from_directory(root.name)
    ser = dman.serialize(rec, context=ctx)
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "_ser__record",
      "_ser__content": {
        "target": "db45b45e-79b5-4cee-9fb7-b84e76457603.sto",
        "sto_type": "base",
        "exceptions": {
          "write": {
            "_ser__type": "__exc_un_writable",
            "_ser__content": {
              "type": "base",
              "info": "Exception encountered while writing.",
              "trace": [
                {
                  "exc": "RuntimeError(Cannot write to disk.)",
                  "frames": [
                    "File 
    /home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 214, 
    in __write__: raise RuntimeError('Cannot write to disk.')"
                  ]
                }
              ]
            }
          }
        }
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 235-236

When we cannot write instead then we get the following behavior:

.. GENERATED FROM PYTHON SOURCE LINES 236-251

.. code-block:: default


    @dman.storable(name='base')
    class Base:
        def __write__(self, path: str):
            with open(path, 'w') as f:
                f.write('<base>')

        @classmethod
        def __read__(cls, path: str):
            raise RuntimeError('Cannot read from disk.')

    rec = dman.record(Base(), preload=True)
    ser = dman.serialize(rec, context=ctx)
    tui.print_serialized(ser)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "_ser__record",
      "_ser__content": {
        "target": "3c3c8c7d-6718-4b30-82c5-8d4260254673.sto",
        "sto_type": "base",
        "preload": true
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 252-253

When deserializing we get the following.

.. GENERATED FROM PYTHON SOURCE LINES 253-257

.. code-block:: default

    dser = dman.deserialize(ser, context=ctx)
    print(dser)
    print(dser.content)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Record(UL[base], target=3c3c8c7d-6718-4b30-82c5-8d4260254673.sto, preload)
    ExcUnReadable: base
        Exception encountered while reading.
    Traceback (most recent call last):
      File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 245, in __read__
        raise RuntimeError('Cannot read from disk.')
    RuntimeError: Cannot read from disk.




.. GENERATED FROM PYTHON SOURCE LINES 258-263

Note how the contents are still unloaded. We could fix `Base` and try loading 
again. We also simulate a more advanced scenario where we first serialize 
the invalid record again, reproducing what would happen 
if objects became invalid and the remainder of the data structure 
was saved to disk again. We also define a corrected version of `Base`.

.. GENERATED FROM PYTHON SOURCE LINES 263-277

.. code-block:: default

    ser = dman.serialize(dser, context=ctx)
    tui.print_serialized(ser)
    tui.walk_directory(root.name, show_content=True, console=tui.Console(width=200))

    @dman.storable(name='base')
    class Base:
        def __write__(self, path: str):
            with open(path, 'w') as f:
                f.write('<base>')

        @classmethod
        def __read__(cls, path: str):
            return cls()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "_ser__type": "_ser__record",
      "_ser__content": {
        "target": "3c3c8c7d-6718-4b30-82c5-8d4260254673.sto",
        "sto_type": "base",
        "preload": true,
        "exceptions": {
          "read": {
            "_ser__type": "__exc_un_readable",
            "_ser__content": {
              "type": "base",
              "info": "Exception encountered while reading.",
              "trace": [
                {
                  "exc": "RuntimeError(Cannot read from disk.)",
                  "frames": [
                    "File 
    /home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 245, 
    in __read__: raise RuntimeError('Cannot read from disk.')"
                  ]
                }
              ],
              "target": "<un-serializable: Target>"
            }
          }
        }
      }
    }
    📂 /tmp/tmpgkez2wvo
    ┗━━ 📄 3c3c8c7d-6718-4b30-82c5-8d4260254673.sto (6 bytes)
         ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
          <base>                                                                                                                                                                                            
         ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 




.. GENERATED FROM PYTHON SOURCE LINES 278-279

Now we can load the contents of the original invalid record.

.. GENERATED FROM PYTHON SOURCE LINES 279-281

.. code-block:: default

    print(dser.content)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <__main__.Base object at 0x7fe4a1e42f80>




.. GENERATED FROM PYTHON SOURCE LINES 282-284

And we can load the contents of the one that was serialized when
it was invalid.

.. GENERATED FROM PYTHON SOURCE LINES 284-287

.. code-block:: default

    dser = dman.deserialize(ser, context=ctx)
    print(dser)
    print(dser.content)




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Record(base, target=3c3c8c7d-6718-4b30-82c5-8d4260254673.sto, preload)
    <__main__.Base object at 0x7fe4a1bf7d60>





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  0.046 seconds)


.. _sphx_glr_download_gallery_fundamentals_example6_errors.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: example6_errors.py <example6_errors.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: example6_errors.ipynb <example6_errors.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_