Note
Click here to download the full example code
Error handling¶
We show how errors are handled within dman.
Introduction¶
To run the example you will need the following imports:
from dataclasses import dataclass
import sys
from tempfile import TemporaryDirectory
import traceback
import dman
from dman import tui
from dman import log
# turn off warnings
log.default_config(level=log.CRITICAL)
Since dman is designed to store results generated by experiments, it always attempts to store as much information as possible. This implies that exceptions are handled internally. Often such exceptions can occur when encountering unserializable types.
@dataclass
class Address:
street: str
number: int
zip_code: int
city: str
country: str
def __str__(self):
return f'{self.street} {self.number}, {self.zip_code} {self.city}, {self.country}'
home = Address('Kasteelpark Arenberg', 10, 3001, 'Leuven', 'Belgium')
employee = {'name': 'John Doe', 'home': home}
dman.save('employee', employee)
tui.walk_directory(dman.mount('employee'), show_content=True)
📂 .dman/cache/examples:fundamentals:example6_errors/employee
┗━━ 📄 employee.json (209 bytes)
──────────────────────────────────────────────────────────────────────────
{
"name": "John Doe",
"home": {
"_ser__type": "__unserializable",
"_ser__content": {
"type": "Address",
"info": "Unserializable type: Address."
}
}
}
──────────────────────────────────────────────────────────────────────────
As you can see only the serializable parts of the data structure were stored. The address has been replaced with a specification of the issue encountered. When we load the employee from disk again we get the following:
recovered = dman.load('employee')
print(recovered['home'])
Unserializable: Address
Unserializable type: Address.
Usually a warning is raised when serialization fails:
with dman.log.logger_context(level=log.WARNING):
dman.serialize(employee)
[01/04/23 10:12:24] WARNING [@dict.Address | context]: serializables.py:629
Serialization resulted in an
invalid object.
Unserializable: Address
Unserializable type:
Address.
The warning will tell you where the serialization failed. In this case it specifies @dict.Address to indicate that the Address object was stored in a dictionary.
Note
If you do want an exception thrown you can do get one as follows:
try:
dman.serialize(employee, context=dman.BaseContext(validate=True))
except dman.ValidationError as e:
msg = traceback.format_exception(*sys.exc_info())
print(''.join(msg))
This is only recommended during debugging, since it interrupts serialization while it is in progress and can therefore result in loss of large amounts of data.
Usually the log messages should provide enough information to fix any issues. We nonetheless continue this example with more detailed examples describing the different things that can go wrong during serialization and what you can expect as output. We also illustrate how data can be recovered even if initial errors were present in the implementation.
We will be doing so in terms of the fundamental datastructures involved in dman. To understand how these fit into the bigger picture take a look at the other examples.
Serializables¶
Serialization¶
We first serialize an unserializable type:
class Base:
...
ser = dman.serialize(Base())
tui.print_serialized(ser)
{
"_ser__type": "__unserializable",
"_ser__content": {
"type": "Base",
"info": "Unserializable type: Base."
}
}
Next let’s try serialization of an object with an erroneous implementation of __serialize__.
@dman.serializable(name="base")
class Base:
def __serialize__(self):
raise RuntimeError("Invalid implementation")
@classmethod
def __deserialize__(cls, ser):
...
ser = dman.serialize(Base())
tui.print_serialized(ser)
{
"_ser__type": "__exc_unserializable",
"_ser__content": {
"type": "Base",
"info": "Error during serialization:",
"trace": [
{
"exc": "RuntimeError(Invalid implementation)",
"frames": [
"File
/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 121,
in __serialize__: raise RuntimeError(\"Invalid implementation\")"
]
}
]
}
}
After deserialization we can still view the traceback resulting in the exception:
err = dman.deserialize(ser)
print(err)
ExcUnserializable: Base
Error during serialization:
Traceback (most recent call last):
File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 121, in __serialize__
raise RuntimeError("Invalid implementation")
RuntimeError: Invalid implementation
We also detect __serialize__ methods with invalid signatures.
@dman.serializable(name="base")
class Base:
# `__serialize__` should either have no or two arguments besides `self`.
def __serialize__(self, arg1, arg2):
return 'Base'
@classmethod
def __deserialize__(cls, ser):
raise RuntimeError('Invalid implementation')
ser = dman.serialize(Base())
tui.print_serialized(ser)
{
"_ser__type": "__exc_unserializable",
"_ser__content": {
"type": "Base",
"info": "Error during serialization:",
"trace": [
{
"exc": "TypeError(Expected method that takes 0 or 1 positional arguments
but got 2.)",
"frames": []
}
]
}
}
Deserialization¶
Now we move on to deserialization.
@dman.serializable(name="base")
class Base:
def __serialize__(self):
return '<base>'
@classmethod
def __deserialize__(cls, ser):
raise RuntimeError('Invalid implementation')
ser = dman.serialize(Base())
tui.print_serialized(ser)
{
"_ser__type": "base",
"_ser__content": "<base>"
}
Even though the serialized object is valid, we cannot recover the object.
err = dman.deserialize(ser)
print(err)
ExcUndeserializable: base
Exception encountered while deserializing <class '__main__.Base'>:
Traceback (most recent call last):
File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 166, in __deserialize__
raise RuntimeError('Invalid implementation')
RuntimeError: Invalid implementation
Serialized
"<base>"
Note how the error contains information about what was serialized. The error is serializable.
ser = dman.serialize(err)
tui.print_serialized(ser)
{
"_ser__type": "__exc_undeserializable",
"_ser__content": {
"type": "base",
"info": "Exception encountered while deserializing <class
'__main__.Base'>:",
"trace": [
{
"exc": "RuntimeError(Invalid implementation)",
"frames": [
"File
/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 166,
in __deserialize__: raise RuntimeError('Invalid implementation')"
]
}
],
"serialized": "<base>",
"expected": "base"
}
}
Moreover when we try to deserialize it again, now with a valid class definition, things work. This sometimes allows for data restoration.
@dman.serializable(name="base")
class Base:
def __serialize__(self):
return '<base>'
@classmethod
def __deserialize__(cls, ser):
return cls()
base = dman.deserialize(ser)
print(base)
<__main__.Base object at 0x7fe4a1bf7b50>
Records¶
Since a record is also serializable it should suppress errors as well such that the data structure is not damaged by a single erroneous item. We provide an overview of the errors that typically occur and how they are represented within the result of the serialization.
Let’s begin with a storable class that cannot write to disk
@dman.storable(name='base')
class Base:
def __write__(self, path: str):
raise RuntimeError('Cannot write to disk.')
@classmethod
def __read__(cls, path: str):
return cls()
rec = dman.record(Base())
If we try to serialize the object without a context we run into some issues
ser = dman.serialize(rec)
tui.print_serialized(ser)
{
"_ser__type": "_ser__record",
"_ser__content": {
"target": "db45b45e-79b5-4cee-9fb7-b84e76457603.sto",
"sto_type": "base",
"exceptions": {
"write": {
"_ser__type": "__un_writable",
"_ser__content": {
"type": "base",
"info": "Invalid context passed to Record(base,
target=db45b45e-79b5-4cee-9fb7-b84e76457603.sto)."
}
}
}
}
}
And when serializing it with a context we run into another.
root = TemporaryDirectory()
ctx = dman.Context.from_directory(root.name)
ser = dman.serialize(rec, context=ctx)
tui.print_serialized(ser)
{
"_ser__type": "_ser__record",
"_ser__content": {
"target": "db45b45e-79b5-4cee-9fb7-b84e76457603.sto",
"sto_type": "base",
"exceptions": {
"write": {
"_ser__type": "__exc_un_writable",
"_ser__content": {
"type": "base",
"info": "Exception encountered while writing.",
"trace": [
{
"exc": "RuntimeError(Cannot write to disk.)",
"frames": [
"File
/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 214,
in __write__: raise RuntimeError('Cannot write to disk.')"
]
}
]
}
}
}
}
}
When we cannot write instead then we get the following behavior:
@dman.storable(name='base')
class Base:
def __write__(self, path: str):
with open(path, 'w') as f:
f.write('<base>')
@classmethod
def __read__(cls, path: str):
raise RuntimeError('Cannot read from disk.')
rec = dman.record(Base(), preload=True)
ser = dman.serialize(rec, context=ctx)
tui.print_serialized(ser)
{
"_ser__type": "_ser__record",
"_ser__content": {
"target": "3c3c8c7d-6718-4b30-82c5-8d4260254673.sto",
"sto_type": "base",
"preload": true
}
}
When deserializing we get the following.
dser = dman.deserialize(ser, context=ctx)
print(dser)
print(dser.content)
Record(UL[base], target=3c3c8c7d-6718-4b30-82c5-8d4260254673.sto, preload)
ExcUnReadable: base
Exception encountered while reading.
Traceback (most recent call last):
File "/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py", line 245, in __read__
raise RuntimeError('Cannot read from disk.')
RuntimeError: Cannot read from disk.
Note how the contents are still unloaded. We could fix Base and try loading again. We also simulate a more advanced scenario where we first serialize the invalid record again, reproducing what would happen if objects became invalid and the remainder of the data structure was saved to disk again. We also define a corrected version of Base.
ser = dman.serialize(dser, context=ctx)
tui.print_serialized(ser)
tui.walk_directory(root.name, show_content=True, console=tui.Console(width=200))
@dman.storable(name='base')
class Base:
def __write__(self, path: str):
with open(path, 'w') as f:
f.write('<base>')
@classmethod
def __read__(cls, path: str):
return cls()
{
"_ser__type": "_ser__record",
"_ser__content": {
"target": "3c3c8c7d-6718-4b30-82c5-8d4260254673.sto",
"sto_type": "base",
"preload": true,
"exceptions": {
"read": {
"_ser__type": "__exc_un_readable",
"_ser__content": {
"type": "base",
"info": "Exception encountered while reading.",
"trace": [
{
"exc": "RuntimeError(Cannot read from disk.)",
"frames": [
"File
/home/runner/work/dman/dman/examples/fundamentals/example6_errors.py, line 245,
in __read__: raise RuntimeError('Cannot read from disk.')"
]
}
],
"target": "<un-serializable: Target>"
}
}
}
}
}
📂 /tmp/tmpgkez2wvo
┗━━ 📄 3c3c8c7d-6718-4b30-82c5-8d4260254673.sto (6 bytes)
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
<base>
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Now we can load the contents of the original invalid record.
print(dser.content)
<__main__.Base object at 0x7fe4a1e42f80>
And we can load the contents of the one that was serialized when it was invalid.
dser = dman.deserialize(ser, context=ctx)
print(dser)
print(dser.content)
Record(base, target=3c3c8c7d-6718-4b30-82c5-8d4260254673.sto, preload)
<__main__.Base object at 0x7fe4a1bf7d60>
Total running time of the script: ( 0 minutes 0.046 seconds)