Resumable

How to implement a resumable experiment.

Overview

We show how you can use dman to create a resumable script. To do so we introduce the following components:

• tui.stack used for resumable for loops (requires rich).

• dman.uninterrupted to allow for keyboard interrupt safe storing of files.

Setting up

To setup the example you will need the following imports:

import dman
from dman import tui
from dman.numeric import barray

import numpy as np
import numpy.random as npr

import time
from typing import Tuple

We will also be using the following modelclass to store our data. The field state will keep track of the current state of the script.

@dman.modelclass
class Experiment:
    data: barray = dman.recordfield(stem="data", default=None)
    state: Tuple[int] = None

Running the experiment

We load the experiment if it exists, otherwise we create a default one.

shape = (30, 10)
exp: Experiment = dman.load("experiment", default_factory=Experiment)
if exp.data is None:
    exp.data = np.zeros(shape)

We use a stack to iterate through the two nested for loops, populating the data array. You can alternatively use sg(range(shape[1]), ...), replacing range with any other iterable.

rg = npr.default_rng(1024)
with tui.stack(exp.state) as sg:
    it = sg.range(shape[0], log={'value': np.nan})
    for i in it:
        for j in sg.range(shape[1]):
            # generate new data point
            time.sleep(0.01)
            exp.data[i, j] = rg.normal()

            # update descriptors of tasks
            it.update(value=exp.data[i, j])

            # store the state and current result
            exp.state = sg.state
            with dman.uninterrupted():
                dman.save("experiment", exp)

Range [30/30] | value=-0.7685216292562801 | ... ━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

We used dman.uninterrupted to make sure that no keyboard interrupts occur while saving to disk. Instead they are captured and raised after dman.save is completed.

You can try running the script and seeing what happens when you press CTRL+C and resume.

No matter how many times you quit the script. Eventually the full array should be computed:

exp: Experiment = dman.load("experiment")
with np.printoptions(linewidth=80, formatter={"float": lambda f: f"{f:+0.2f}"}):
    tui.pprint(exp.data)

barray([[-0.45, -0.14, +2.42, +0.08, +1.49, +0.46, -0.21, -0.66, -0.36, +0.45],
│   │   [-1.47, -0.37, +1.56, +0.76, +0.25, -0.34, -0.89, +2.04, +0.59, -1.03],
│   │   [-0.84, -0.52, -0.46, +0.60, +1.16, -0.09, +1.10, -1.23, +0.52, +0.66],
│   │   [+1.17, +1.00, +0.95, -1.98, -0.51, -1.62, -0.25, -1.94, +0.88, -0.39],
│   │   [+0.95, -0.69, -0.12, +0.81, -0.92, -1.94, +2.43, -1.63, +0.37, +0.90],
│   │   [+0.55, -1.77, +0.66, +0.36, +0.72, -2.18, -0.07, -0.68, -1.36, -0.12],
│   │   [-1.51, -0.05, -0.90, -0.33, -0.23, -2.00, -0.07, +0.49, -0.68, -0.03],
│   │   [+0.23, +0.27, -0.66, -0.08, -0.46, +0.19, +2.17, +1.06, +0.51, +0.47],
│   │   [-0.12, +0.26, +0.52, +0.28, +2.12, -0.48, +0.65, -0.62, +0.84, -0.26],
│   │   [-1.31, +0.52, -0.04, -1.84, +0.14, +0.28, +0.37, +0.17, -1.39, +0.94],
│   │   [+0.05, -0.43, -0.31, +0.09, -0.28, +1.16, +0.25, -0.85, +0.23, -0.80],
│   │   [-0.19, -0.19, -0.43, +0.29, +0.54, +0.30, +1.18, +1.56, +0.70, -1.45],
│   │   [+0.26, -2.29, +0.04, +0.56, -0.94, -1.60, -0.84, -0.29, +1.21, -0.61],
│   │   [+1.19, +0.04, +0.46, +1.93, -0.44, -0.39, +1.42, -0.92, +2.12, +0.58],
│   │   [+0.50, +0.63, +0.79, -1.00, -1.46, -0.32, +0.97, -0.27, -0.36, -1.63],
│   │   [+0.11, -1.04, +0.36, -1.84, +0.91, +0.61, -1.09, -1.19, -0.16, -1.71],
│   │   [-0.79, -2.16, -0.61, +0.51, -1.76, +0.38, -0.37, -0.43, +0.14, +0.73],
│   │   [-2.06, -2.30, -0.32, -0.65, -1.66, -0.47, +0.12, -0.20, +0.17, -0.35],
│   │   [-0.79, -1.22, -0.72, -0.04, +0.15, -0.27, -1.15, +1.51, -0.20, -1.05],
│   │   [+0.70, -0.89, -0.58, +1.32, -0.92, -1.03, +0.79, -1.05, -0.34, +0.00],
│   │   [+1.32, +0.49, -0.14, +0.97, -0.06, +0.26, +0.43, +1.58, -0.15, +0.08],
│   │   [+0.75, -1.41, +0.30, -0.05, -0.44, +0.72, -0.90, -1.26, -0.40, -0.42],
│   │   [+1.32, +0.21, -0.30, +1.07, -1.47, +1.38, +0.04, -0.99, -0.67, -0.84],
│   │   [+1.67, -1.85, -0.48, -0.82, +1.79, -0.87, +1.55, -1.05, -1.35, -1.97],
│   │   [+0.66, +0.05, -1.12, -0.46, -1.14, +0.60, -0.57, -0.33, -0.32, -1.37],
│   │   [-0.10, -0.51, -0.41, -0.15, +1.04, -0.19, +1.62, -0.32, -0.59, +2.41],
│   │   [-1.18, -0.80, +2.57, -0.20, +0.47, +0.96, +0.84, -1.17, -1.10, -1.89],
│   │   [+0.99, +1.39, -0.15, -1.29, +0.11, -0.26, -0.41, -0.93, -0.63, -0.69],
│   │   [+1.15, -0.26, -0.88, +0.48, -1.88, -0.32, -0.02, +0.31, +0.29, -0.42],
│   │   [+0.91, +0.15, +0.15, +0.41, -0.52, -0.23, -0.67, +0.95, -0.80, -0.77]])