API¶

functional¶

A collection of function and variables used to define experiments using the functional api

class xmen.functional.Root(root=None, purpose='', copy_params=True, **kwargs)[source]¶

The first argument passed to a functional experiment and principally used to root an experiment instance to a particular directory:

def functional_experiment(root: Root, ...):
    with open(root.directory, 'w') as f:
        f.write('Running experiment)

    root.message({'time': time.time()})

Note

Root is nothing more than Experiment with a different name. Whilst principally offering exactly the same functionality, primarily the purpose of Root is to expose the directory property and messaging protocol of the Experiment class to functional experiment definitions. However, there is nothing stopping the user form using the full functionality of the Experiment class if they wish. Please consult the Experiment class documentation in this case.

xmen.functional.read_comments(fn)[source]¶

A helper function for reading comments from the function definition. This should not be generally needed as xmen takes care of this for you.

Parameters: fn – A functional experiment definition conforming to the xmen api
Returns: A help string generated for each function argument.
Return type: docs (str)

xmen.functional.functional_experiment(fn)[source]¶

Convert a functional experiment to a class definition. Generally this should not be needed as xmen takes care of this for you. Specifically:

The parameters of the experiment are added from the argument of the function

Comments next to each argument will be automatically added to the doc string of the experiment

The experiments run method will be set to fn

Parameters: fn (xmen.Root) – An experiment definition conforming to the xmen functioanl api (the function must take as its first argument an object inheriting from experiment)
Returns: A class equivalent definition of fn
Return type: Exp (class)

xmen.functional.autodoc(func)[source]¶: A decorator used to add parameter comments to the docstring of func.

experiment¶

class xmen.experiment.Experiment(root=None, purpose='', copy_params=True, **kwargs)[source]¶

Base class from which all other experiments derive. Experiments are defined by:

Parameters: class attributes declared with the special parameter # @p in a comment after the definition.
Execution: defined by overloading the run() method

For example:

class AnExperiment(Experiment):
     ''' Doc strings should be used to document the purpose of the experiment'''
    # experiment parameters are defined as class attributes with an @p tag in there comment
    a: str = 'Hello'   # @p a parameter
    b: str = 'World!'  # @p another parameter

    # experiment execution code is defined in the experiments run method
    def run(self):
        print(f'{self.a} {self.b})

__init__(root=None, purpose='', copy_params=True, **kwargs)[source]¶

Create a new experiment object.

Parameters

name (root,) – If not None then the experiment will be registered to a folder {root}\{name}
purpose (str) – An optional string giving the purpose of the experiment.
copy_params (bool) – If True then parameters are deep copied to the object instance from the class definition. Mutable attributes will no longer be shared.
**kwargs – Override parameter defaults.

property start¶: The time the experiment last started running.

property registered¶: The time the experiment was last registered.

property stopped¶: The time the experiment was last stopped.

property last¶: The time the experiment state was last communicated.

property notes¶: A dictionary containing the notes attached to the experiment

property root¶: The root directory to which the experiment belongs

property status¶: The status of the experiment. One of 'default', 'registered', 'running', 'finished' or 'error'.

property created¶: The date the experiment parameters was first registered.

property purpose¶: A string giving a purpose message for the experiment

property messages¶: A dictionary of messages logged by the experimet.

property version¶: A dictionary giving the version information for the experiment

property user¶: The current user of the experiment

property host¶: The host of the experiment

note(string, remove=False)[source]¶: Leave a note with the experiment. Will be removed if remove is False

get_param_helps()[source]¶: Get help for all attributes in class (including inherited and private).

to_defaults(defaults_dir)[source]¶

Create a defaults.yml file from experiment object. Any base class inheriting from Experiment can create a default file as:

MyExperiment().to_yaml('/dir/to/defaults/root')

debug()[source]¶: Inherited classes may overload debug. Used to define a set of open_socket for minimum example

from_yml(path, copy=False)[source]¶

Load state from either a params.yml or defaults.yml file (inferred from the filename). The status of the experiment will be updated to 'default' if 'defaults.yml' file else 'registered' if params.yml file.

If copy is True then only user defined parameters themselves will be copied from the params.yml file.

register(root, purpose='', force=True, same_names=100, restart=False, **_)[source]¶

Register an experiment to an experiment directory. Its status will be updated to registered. If an experiment called name exists in root and force==True then name will be appended with an int (eg. {name}_0) until a unique name is found in root. If force==False a ValueError will be raised.

If restart is also passed, and an experiment called name also exists, then the experiment will be loaded from the params.yml file found in '{root}/{name}'.

Raises: ValueError – if {root}/{name} already contains a params.yml file

to_root(root_dir, shell='/bin/bash')[source]¶

Generate a defaults.yml file and script.sh file in root_dir.

Parameters: root_dir (str) – A path to the root directory in which to generate a script.sh and defaults.yml to run the experiment.

update(kwargs)[source]¶: Update the parameters with a given dictionary

message(messages, keep='latest', leader=None)[source]¶

Add a message to the experiment (and an experiments params.yml file). If the experiment is not registered to a root then no messages will be logged.

Parameters

messages (dict) – A dictionary of messages. Keys are interpreted as subjects and values interpreted as messages. If the defaults.yml already contains subject then the message for subject will be updated.
keep (str) – which message to keep in the case of collision. One of [‘latest’, ‘min’, ‘max’]
leader (str) – If not None then all messages will be saved if the keep condition is met for the leader key.

Note

Only messages of type float, int and string are supported. Any other message will be converted to type float (if possible) then string thereafter.

parse_args()[source]¶: Configure the experiment instance from the command line arguments.

main(args=None)[source]¶

Take the command line args and execute the experiment (see parse_args for more

information). In order to expose the command line interface:

if __name__ == '__main__':
    exp = AnExperiment().main()

Note that for backwards compatibility it is also possible to pass args as an argument to main. This allows the experiment to be run from the commandline as:

if __name__ == '__main__':
    exp = AnExperiment()
    args = exp.parse_args()
    exp.main(args)

stdout_to_txt()[source]¶: Configure stdout to also log to a text file in the experiment directory

monitor¶

Monitor¶

class xmen.monitor.Monitor(*, hooks=[], log=None, log_fn=None, log_format='', time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶

Automate tracking and logging of experiments.

Configured through arguments of the form f'{regex}@{modulo}{trigger}'. Any variables in the local scope of the monitor which match the specified regular expression will be logged every modulo steps of the trigger. Triggers are incremented either manually (using the inc method) or automatically using nested iterators (see example). Supported triggers include [“step”, “epoch”, “era”, “eon”, “supereon”] or their abbreviations [“s”, “e”, “er”, “eo”, “se”].

If a hook is passed with modulo == None it can instead be triggered manually as monitor.to_hooks().

Example:

from xmen import Experiment
from xmen.monitor import Monitor

X = Experiment(..., ...)

a, b, c, d, e, f = 0, 0, 0, 0, 0, 0


def identity(x):
    return x


def mult(x):
    return 10 * x


m = Monitor(
    log=('a|b@2s', 'b@1e', 'c@1er', "d|e@1eo", "e@1se"),
    log_fn=(mult, identity, identity, identity, identity),
    log_format='.3f',
    msg='a->X@1s',
    time=('@2s', '@1e'),
    probe='X@10s',
    limit='@20s')
for _ in m(range(2)):
    for _ in m(range(2)):
        for _ in m(range(3)):
            for _ in m(range(4)):
                for _ in m(range(5)):
                    a += 1
                b += 1
            c += 1
        d += 1
    e += 1

__init__(*, hooks=[], log=None, log_fn=None, log_format='', time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶

Parameters

hooks (Iterable[Hook]) – User defined hooks used to extend the functionality of the Monitor class inheriting from Hook.
log (str, Iterable[str]) – A modulo string of the form "f{regex}" or "{regex}@{steps}s" (or list of) giving the variables to log @ a particular logging frequency to stdout.
log_fn (Callable, Iterable[Callable]) – Converts the variable into a string for logging.
log_format (str, Iterable[str]) – A format used to format a string as f"{string}:{format}"
time (str, Iterable[str]) – A string of the form f"@{steps}" to log timing statistics at (or list of for different triggers).
msg (str, Iterable[str]) – A modulo string of the form "{regex}->{exp_regex}@{steps}s" (or list of) giving the variables to log as messages with the experiments matching exp_regex.
msg_keep (str, Iterable[str]) – One of [‘latest’, ‘max’, ‘min’] giving the protocol to use on message collision
msg_leader (str, Iterable[str]) – A regex to a single variable. If supplied then this variable will be treated as the leader and all other variables will be logged only if the keep condition is met for the leader
msg_expand (bool, Iterable[str]) – If True then dictionary variables with K keys will be expanded to give K variables
msg_prep (bool, Iterable[str]) – If True then each variable in the dictionary will be prepended by the dictionary name.
probe (str, Iterable[str]) – A string of the form f"{regex}@{steps}" to log resource use to each experiment that matches regex (or list of for different triggers).
limit (str, Iterable[str]) – A modulo string of the form ``f”@{modulo}{triger}” used to limit the number of iterations of an experiment at a particular trigger level. This is useful if an experiment is restarted for example.

Note

All variables ckpt_keep, msg_leader, msg_expand and msg_prep can be supplied as a single or as a list of entries, one for each set of variables matching each modulo string in each case.

inc(trigger, back=1)[source]¶: Manually increment trigger. Will also run modulo hooks defined by the user.

to_hooks(**kwargs)[source]¶: Pass all keyword arguments to manual hooks (hooks where modulo is None)

log(x, format='', process_func=<class 'str'>)[source]¶

Log string with time, epoch and step.

Parameters

x – Value to be logged
format – The format used to convert x to a string as ‘{x:{format}}’
process_func – A callable able to convert x to a valid format for printing as {x:{format}}’

modulo(trigger, modulo, exclude_1st=False)[source]¶: Check if trigger % modulo == 0. If exclude_1st is True then modulo will return False for triggers at 0

summary(verbose=0)[source]¶: Summarise the current state of the experiment monitor

TorchMonitor¶

class xmen.monitor.TorchMonitor(directory=None, *, hooks=[], ckpt=None, ckpt_keep=None, log=None, log_fn=None, log_format='', img=None, img_fn=None, img_pref='', img_prep=True, img_options={}, sca=None, sca_fn=None, sca_pref='', sca_prep=True, hist=None, hist_fn=None, hist_pref='', hist_prep=True, fig=None, fig_fn=None, fig_pref='', fig_prep=True, txt=None, txt_fn=None, txt_pref='', txt_prep=True, vid=None, vid_fn=None, vid_pref='', vid_prep=True, time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶

Automate tracking, logging and saving of experiments with additional hooks for logging to tensorboard and checkpointing of experiments.

Manual logging and checkpoint are also supported as monitor.log(...) and monitor.checkpoint(...)

__init__(directory=None, *, hooks=[], ckpt=None, ckpt_keep=None, log=None, log_fn=None, log_format='', img=None, img_fn=None, img_pref='', img_prep=True, img_options={}, sca=None, sca_fn=None, sca_pref='', sca_prep=True, hist=None, hist_fn=None, hist_pref='', hist_prep=True, fig=None, fig_fn=None, fig_pref='', fig_prep=True, txt=None, txt_fn=None, txt_pref='', txt_prep=True, vid=None, vid_fn=None, vid_pref='', vid_prep=True, time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶

Parameters

directory (str) – The directory used to log checkpoints in
hooks (Iterable[Hook]) – User defined hooks used to extend the functionality of the Monitor class
ckpt (str, Iterable[str]) – A modulo string of the form "f{regex}" or "{regex}@{steps}s" (or list of) giving the variables to checkpoint @ a particular logging frequency to stdout. The regex much match objects inheriting form torch.Module.
ckpt_keep (int, Iterable[int]) – The number of checkpoints to keep. The most recent checkpoints will be kept. If None then all checkpoints will be kept.
log (str, Iterable[str]) – A modulo string of the form "f{regex}" or "{regex}@{steps}s" (or list of) giving the variables to log @ a particular logging frequency to stdout.
log_fn (Callable, Iterable[Callable]) – Converts the variable into a string for logging.
log_format (str, Iterable[str]) – A format used to format a string as f"{string}:{format}"
time (str, Iterable[str]) – A string of the form f"@{steps}" to log timing statistics at (or list of for different triggers).
img (str, Iterable[str]) – A modulo string of the form "f{regex}" or "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard images @ a particular logging frequency.
img_fn (Callable, Iterable[Callable]) – Converts the variable into an image of shape [B, C, H, W] or [C, H, W] for tensorboard. See TensorBoardLogger for more details in terms of automatic-processing. If img is a list then img_fn can also be passed as list with a callable for each entry in img or can be passed as a single callable used for all entries.
img_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
img_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
img – A modulo string of the form "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard scalars @ a particular logging frequency.
sca_fn (Callable, Iterable[Callable]) – Converts the variable into a scalar for tensorboard. See TensorBoardLogger for more details in terms of automatic-processing. If sca is a list then sca_fn can also be passed as list with a callable for each entry in sca or can be passed as a single callable used for all entries.
sca_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
sca_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
hist (str, Iterable[str]) – A modulo string of the form "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard histograms @ a particular logging frequency. If no logging frequency is supplied then any variable logged in the experiment which matches regex will be logged to tensorboard each time it is passed to the logger. This is useful for logging variables at the end of an epoch for example.
hist_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard. If hist is a list then hist_fn can also be passed as list with a callable for each entry in hist or can be passed as a single callable used for all entries.
hist_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
hist_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
fig (str, Iterable[str]) – A modulo string of the form or "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard figures @ a particular logging frequency.
fig_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard into a plt.figure(). If fig is a list then fig_fn can also be passed as list with a callable for each entry in fig or can be passed as a single callable used for all entries.
fig_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
fig_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
txt (str) – A modulo string of the form "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard text @ a particular logging frequency.
txt_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard. If txt is a list then txt_fn can also be passed as list with a callable for each entry in txt or can be passed as a single callable used for all entries.
txt_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
txt_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
vid (str, Iterable[str]) – A modulo string of the form "{regex}@{steps}s" (or list of) giving the variables to add as tensorboard videos @ a particular logging frequency.
vid_fn (Callable, Iterable[Callable]) – Preprocess variable to a tensor of shape [B, T, C, H, W] before logging to tensorboard. If vid is a list then vid_fn can also be passed as list with a callable for each entry in vid or can be passed as a single callable used for all entries.
vid_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
vid_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
msg (str, Iterable[str]) – A modulo string of the form "{regex}->{exp_regex}@{steps}s" (or list of) giving the variables to log as messages with the experiments matching exp_regex.
msg_keep (str, Iterable[str]) – One of [‘latest’, ‘max’, ‘min’] giving the protocol to use on message collision
msg_leader (str, Iterable[str]) – A regex to a single variable. If supplied then this variable will be treated as the leader and all other variables will be logged only if the keep condition is met for the leader
msg_expand (bool, Iterable[bool]) – If True then dictionary variables with K keys will be expanded to give K variables
msg_prep (bool, Iterable[bool]) – If True then each variable in the dictionary will be prepended by the dictionary name.
probe (str, Iterable[str]) – A string of the form f"{regex}@{steps}" to log resource use to each experiment that matches regex (or list of for different triggers).
limit (str) – A modulo string of the form ``f”@{modulo}{triger}” used to limit the number of iterations of an experiment at a particular trigger level. This is useful if an experiment is restarted for example.

Note

All variables …_fn, …_pref and …_prep as well as ckpt_keep and msg_leader, msg_expand and msg_prep can be supplied as a single or as a list of entries, one for each set of variables matching each modulo string in each case.

Example 1:

nn, opt, dataset = ..., ...

m = Monitor(
    directory,
    checkpoint=('model@1e', 'opt@100s'),   # Checkpoint the model once per epoch and opt every 100 steps
    log='^loss$@100s',   # Log the loss to stdout every 100 steps
    img='^x$@1000s', sca=('^loss$@100s', 'eval_.*@1e'), time=('@100s')     # Log to tensorboard
    time=('@100s', ),   # Generate timing statistics every 100 steps
    hooks=[  # Custom hooks are also supported
        MyVeryOwnHook(...)])

# The only modification needed to the training loop are the em calls.
# Nested loops corresponds to different triggers from inside out
# we have ["step" or "s", "epoch" or "e", "era" or "er", "eon" or "eo", "supereon" or "se"]
for epoch in m(range(10)):
    for x, y in m(datset):
        _y_ = model(x)
        opt.zero_grad()
        loss = loss_fn(y, _y_)
        loss.backward()
        loss.step()
        em.log('Manual Logging is also supported')
    eval_1, eval_2 = eval(model, ds)

# Steps and epoch have been incremented
assert em.step == len(ds) * 10
assert em.epoch == 10

# Lets reload the model at the 5th epoch
em.load(step=5*len(ds), model)
# The step and epoch will be updated
print(em.step, em.epoch)

Example 2:

from xmen.monitor import Monitor
import numpy as np
import torch
import os
import matplotlib.pyplot as plt
from torchvision.datasets.mnist import MNIST
from torch.utils.data import DataLoader
import torchvision.transforms as T

plt.style.use('ggplot')
ds = DataLoader(MNIST(os.getenv("HOME") + '/data/mnist', download=True,
      transform=T.Compose(
          [T.Resize([64, 64]), T.CenterCrop([64, 64]),
           T.ToTensor(), T.Normalize([0.5], [0.5])])), 8)

m = Monitor(
    directory='/tmp/tb_5',
    sca=['^z$|^X$@10s', '^a|^x$@5s'],
    img=['^mnist@10s', '^mnist@5s'], img_fn=[lambda x: x[:2], lambda x: x[:5]], img_pref=['2', '5'],
    hist='Z@1s',
    fig='fig@10s',
    txt='i@5s', txt_fn=lambda x: f'Hello at step {x}',
    vid='^mnist@10s', vid_fn=lambda x: (x.unsqueeze(0) - x.min()) / (x.max() - x.min())
)

# variables
x = 0.
a = [1, 2]
z = {'x': 5, 'y': 10}
for i, (mnist, _) in m(zip(range(31), ds)):
    # plot a figure
    fig = plt.figure(figsize=[10, 5])
    plt.plot(np.linspace(0, 1000), np.cos(np.linspace(0, 1000) * i))
    # random tensor
    Z = torch.randn([10, 3, 64, 64]) * i / 100
    # scalars
    x = (i - 15) ** 2
    z['i'] = i
    z['x'] += 1
    z['y'] = z['x'] ** 2

checkpoint(**kwargs)[source]¶: Checkpoint the torch.nn objects with step and epoch passed as name==variable_to_save

load(directory=None, step=None, attempt=True, update_triggers=True, **modules)[source]¶: Load the torch torch.nn objects passed as name=variable_to_load, from the directory and reset the state of the em (if update_triggers == True). If attempt == False then an Exception will be raised if either the directory does not contain checkpoints corresponding to modules.

Hooks¶

class xmen.monitor.Hook(spec)[source]¶: A base class defining a variable passing protocol with the Monitor class. Ever time the monitors count is divisible by modulo for a particular trigger then the hook is passed all the variables matching regex from the current stack. Users therefore define hooks by overloading the hooks __call__ method (with the same call signature).

class xmen.monitor.XmenMessenger(spec, keep='latest', leader=None, expand=False, prepend=None)[source]¶

A hook for logging messages with an xmen.Experiment object.

Example 1:

from xmen import Experiment
from xmen.monitor import Monitor

messenger = XmenMessenger('y.*->ex.*@10s')   # log all variables matching the loss to experiments matching ex
m = Monitor(hooks=[messenger])
y1, y2 = 0, 0
ex1, ex2 = Experiment(), Experiment()
ex1.link('/tmp', 'ex1')
ex2.link('/tmp', 'ex2')
for i in m(range(40)):
    y1 += 1
    y2 += 2
    if i % 10 == 1:
        print(', '.join(
            [f"ex1: {k} = {ex1.messages.get(k, None)}" for k in ('y1', 'y2')] +
            [f"ex2: {k} = {ex1.messages.get(k, None)}" for k in ('y1', 'y2')]))

# Output
# ex1: y1 = None, ex1: y2 = None, ex2: y1 = None, ex2: y2 = None
# ex1: y1 = 10, ex1: y2 = 20, ex2: y1 = 10, ex2: y2 = 20
# ex1: y1 = 20, ex1: y2 = 40, ex2: y1 = 20, ex2: y2 = 40
# ex1: y1 = 30, ex1: y2 = 60, ex2: y1 = 30, ex2: y2 = 60

Example 2:

from xmen import Experiment
from xmen.monitor import Monitor

ex = Experiment()
ex.link('/tmp', 'ex')
m = Monitor(
    hooks=[
        XmenMessenger('^y$|^i$->^ex$@10s', keep='min', leader='^y$')])

x = -50
for i in m(range(100)):
    x += 1
    y = x ** 2
    if i % 10 == 1:
        print([ex.messages.get(k, None) for k in ('i', 'y')])

# Output
# [None, None]
# [9, 1600]
# [19, 900]
# [29, 400]
# [39, 100]
# [49, 0]
# [49, 0]
# [49, 0]
# [49, 0]
# [49, 0]

Example 3:

from xmen import Experiment
from xmen.monitor import Monitor

ex = Experiment()
ex.link('/tmp', 'ex')
m = Monitor(
    hooks=[
        XmenMessenger('z->^ex$@10s', keep='min', leader='y', expand=True)])

z = {'x': 5, 'y': 10}
for i in m(range(100)):
    z['i'] = i
    z['x'] += 1
    z['y'] = z['x'] ** 2
    if i % 10 == 1:
        print([ex.messages.get(k, None) for k in ('i', 'y', 'x')])

# [None, None, None]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]
# [9, 225, 15]

Example 4:

from xmen import Experiment
from xmen.monitor import Monitor

ex = Experiment()
ex.link('/tmp', 'ex')
m = Monitor(
    hooks=[
        XmenMessenger('z->^ex$@10s', keep='min', leader='z_y', expand=True, prepend=True)])


z = {'x': 5, 'y': 10}
for i in m(range(100)):
    z['i'] = i
    z['x'] += 1
    z['y'] = z['x'] ** 2
    if i % 10 == 1:
        print([ex.messages.get(k, None) for k in ('z_i', 'z_y', 'z_x')])

# same output as above

__init__(spec, keep='latest', leader=None, expand=False, prepend=None)[source]¶

Parameters

spec (Spec) – hook specification in the form either {log_regex}->{exp_regex}@{modulo}{trigger} or {exp_regex}@{modulo}{trigger} where exp_regex is the experiments to message and log_regex are additional messages to log with the experiment. In the second case only timing and step information will be logged
keep (str) – One of [‘latest’, ‘max’, ‘min’] in the case of message collision with each experiment
leader (regex) – If leader is not None then messages will be logged according to keep as judged by the variable in var_dict which matches leader.
expand – If a dictionary variable with K keys matches log_regex then it is converted to K variables with names corresponding to the keys in dict if expand==True.
prepend – If prepend is True then in the case above the name of each variable will be prepended by the dict variable name. eg. each variable will be called ‘{name}_{k}’.

class xmen.monitor.Timer(spec)[source]¶

A simple timing hook used to log any timers open_socket by the experiment monitor

__init__(spec)¶: Initialize self. See help(type(self)) for accurate signature.

class xmen.monitor.Logger(spec, format='', process_func=None)[source]¶

A simple logging hook used to log variables to stdout.

Example:

from xmen.monitor import Monitor, Logger

m = Monitor(
    hooks=[
        Logger('x@2s', process_func=lambda x: '|'.join(x)),
        Logger('y@1e', format='.5f')])

x = ['cat', 'dog']
y = 0.
for _ in m(range(3)):
    for i in m(range(5)):
        y += i

# [01:17PM 18/11/20 0/3 2/15]: x = cat|do
# [01:17PM 18/11/20 0/3 4/15]: x = cat|do
# [01:17PM 18/11/20 1/3]: y = 10.0000
# [01:17PM 18/11/20 1/3 6/15]: x = cat|do
# [01:17PM 18/11/20 1/3 8/15]: x = cat|do
# [01:17PM 18/11/20 1/3 10/15]: x = cat|do
# [01:17PM 18/11/20 2/3]: y = 20.0000
# [01:17PM 18/11/20 2/3 12/15]: x = cat|do
# [01:17PM 18/11/20 2/3 14/15]: x = cat|do
# [01:17PM 18/11/20 3/3]: y = 30.0000

__init__(spec, format='', process_func=None)[source]¶

Parameters

spec (Spec) – a specification string of the form “{regex}@{modulo}{trigger}”. Variables matching regex will be logged when trigger % modulo == 0.
format (str) – a format string used as f”{var:format}” for logging string variables
process_func (callable) – used to convert variables to a string for logging to stdout. Format will be applied after.

class xmen.monitor.TensorboardLogger(type, spec, fn=None, prefix='', prepend=True, **options)[source]¶

A hook for logging tensorboard summaries. Currently image, scalar, histogram, figure, video, text, pr_curve, mesh are all supported.

Before being passed to the summary writer each variable is processes as follows:

First all variables are passed to fn (if supplied).

Variables of type list or dict with length K will be expanded to give K variables. The name of each variable will be the list name postpended with its index or the dictionary name postpended with its key.

If the summary type is image or scalar then some additional processing will be performed:

For scalars if variables are not already scalar variables they will be converted by calling var.mean()

For images the variable must be either a 3D [C, H, W] or 4D [B, C, H, W] tensor. Tensors are converted to images of shape [C, H, W]:

if the variable is 3D it will be converted to 4D

the variable is then converted to an image using torchvision.utils.make_grid. Additional options can be passed to torchvision.utils.make_grid using the options parameter. Available options include:

'nrow' # images per row

'padding'

'normalize'

'range'

'scale_each'

'pad_value'

(see torchvision.utils.make_grid for more info)

Note

Tensorboard does not allow summaries to have the same name. If you want to leave to different types of summary for the same variable then you will need to use the prefix argument.

__init__(type, spec, fn=None, prefix='', prepend=True, **options)[source]¶

Parameters

type (str) – The type of tensorboard summary to log. Should be one of [‘image’, ‘scalar’, ‘histogram’, ‘figure’, ‘video’, ‘text’, ‘pr_curve’, ‘mesh’]
spec (str) – A string in the form "{regex}@{modulo}{trigger}". tensorbaord summaries will be logged for all variables matching regex when ``monitor.{trigger} % modulo == 0`
fn (callable) – A function used to convert each variable to a summary
prefix (str) – prepend all summaries with this string
prepend (bool) – If true prepend dictionary variable names with the dictionary name.
**options – Keyword arguments passed to torchvision.utils.make_grid

make_compatible(v)[source]¶: Convert the variable v to a valid input for the summary writer

class xmen.monitor.Checkpointer(spec, to_keep=None, expand=True)[source]¶

__init__(spec, to_keep=None, expand=True)[source]¶: Initialize self. See help(type(self)) for accurate signature.

Helpers¶

xmen.monitor.load(load_dir, step=None, attempt=True, **modules)[source]¶: Load torch objects from a checkpoint object. If Step is None then the most recent checkpoint is loaded. Else the checkpoint is loaded at the specified step. If attempt == False then load will raise a ValueError if either one of the modules was not found in modules or if no checkpoints were found in load_dir.

Note

No check to ensure the meta information in each file is the same. The meta information returned corresponds to the first module encountered in modules.

lightning¶

utils¶

A module containing several utilitity functions, classes and Meta Classes used by the ExperimentManager and the Experiment classes.

xmen.utils.load_param(root, file='params.yml')[source]¶: Load parameters from a params.yml file.

xmen.utils.save_param(params, root, file='params.yml')[source]¶

Save a dictionary of parameters at {root}/params.yml

Parameters

params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel.yaml
root (str) – The root of the experiment

xmen.utils.load_params(roots)[source]¶: Load params.yml files into a list of dictionaries from a list of paths

exception xmen.utils.IncompatibleYmlException[source]¶

xmen.utils.flatten(d, parent_key='', sep='_')[source]¶: Flatten a nested dictionary to a single dictionary. The keys of nested entries will be joined using sep

xmen.utils.dic_to_yaml(dic, typ='rt', default_flow_style=False)[source]¶: Convert dictionary to a yaml string (dic can also be a CommentedMap)

xmen.utils.dic_from_yml(*, string=None, path=None)[source]¶: Load from either a yaml string or path to a yaml file

xmen.utils.get_size(bytes, suffix='B')[source]¶: Scale bytes to its proper format e.g:

1253656 => ‘1.20MB’ 1253656678 => ‘1.17GB’

xmen.utils.get_meta(get_platform=False, get_cpu=False, get_memory=False, get_disk=False, get_slurm=False, get_conda=False, get_network=False, get_gpu=False, get_environ=False, live=False)[source]¶: Get Meta information for the system

xmen.utils.get_attribute_helps(cls)[source]¶: Get all help files for class cls from doc strings from all inherited classes.

xmen.utils.get_docs(cl)[source]¶: Return all docs for every inherited class

class xmen.utils.TypedMeta(name, bases, attr_dict)[source]¶

A meta class helper used to generate automatic doc strings generated from typed class definitions. This allows the docstrings to be defined inside the __init__ body instead of inside the doc string iradicating the need to type attributes twice:

MyClass(metaclass=TypedMeta):
    '''This class is bound to do something...'''

    a int: 3   # @p My first attribute
    b int: 5   # @p My second attribute

 m = MyClass()
 print(m.__init.__.__doc__)

Now if we call:

>>> ob = print(MyClass.__doc__)

 This class is bound to do something...

 Parameters:
     a (int) : My first attribute (default=3)
     b (int) : My second attribute (default=5)

The doc string has automatically been updated.

xmen.utils.get_git(path)[source]¶

Get git information for the given path.

Returns

Empty if status == False otherwise with keys:

remote: A url to the remote repository
hash: A git hash to the current commit

Return type

(dict)

status (bool): False if either git is not available, the repo is not in a git repository or if there are: uncommited changes in the current repository. Otherwise true

xmen.utils.get_version(*, path=None, cls=None, fn=None)[source]¶

Get version information for either a path to a directory or a class. Git version information is loaded if available.

Parameters

path (str) – A path to a repository which is inspected for version information
cls (Class) – A Class object to be inspected for version information

Returns

A dictionary containing at least one of the following:

if path is not None:
- 'path': A copy of the path
if cls is not None:
- 'module': A path to the module in which the class was defined
- 'class': The name of the class
if git != {}: (i.e if path or module is in a git repository):
- 'git_local': The root of the local git repository
- 'git_commit': The hash of the commit
- 'git_remote': The remote url if has remote else None

Return type

version (dict)

xmen.utils.get_run_script(module, name, shell='/usr/bin/env python3', comment='#')[source]¶

Generate a run script for a particular experiment.

Parameters

module (str) – the module to look in
name (str) – the name of the experiment in the module. If name corresponds to a function it will be converted to an Experiment class

xmen.utils.dics_to_pandas(dics, reg)[source]¶: Convert a list of nested dictionaries into a pandas data frame keeping only keys that match reg (after flattening)

manager¶

class xmen.manager.ExperimentManager(root='', headless=False)[source]¶

A helper class with wrapped command line interface used to manage a set of experiments. It is compatible both with experiments generated by the Experiment.to_root method call as well as any experiment that can be represented as a bash script.sh which takes a set of parameters in a yaml file as input.

More Info:

At its core the experiment manager maintains a single root directory:

root
├── defaults.yml
├── experiment.yml
├── script.sh
├── {param}:{value}__{param}:{value}
│   └── params.yml
├── {param}:{value}__{param}:{value}
│   └── params.yml
...

In the above we have:

defaults.yml defines a set of parameter keys and there default values shared by each experiment. It generated from the Experiment class it will look like:

   # Optional additional Meta parameters
   _created: 06:58PM September 16, 2019    # The date the defaults were created
   _version:
       module: /path/to/module/experiment/that/generated/defaults/was/defined/in
       class: TheNameOfTheExperiment
       git:
          local: /path/to/git/repo/defaults/were/defined/in
          branch: some_branch
          remote: /remote/repo/url
          commit: 80dcfd98e6c3c17e1bafa72ee56744d4a6e30e80    # The git commit defaults were generatd at

   # Default parameters
   a: 3 #  This is the first parameter (default=3)
   b: '4' #  This is the second parameter (default='4')

The ``ExperimentManager`` is also compatible with generic experiments. In this the ``_version`` meta
field can be added manually, replacing ``module`` and ``class`` with ``path``. The git information for
each will be updated automatically provided ``path`` is within a git repository.

script.sh is a bash script. When run it takes a single argument 'params.yml' (eg. `script.sh params.yml`).
Note

Experiment objects are able to automatically generate script.sh files that look like this:
#!/bin/bash # File generated on the 06:34PM September 13, 2019 # GIT: # - repo /path/to/project/module/ # - remote {/path/to/project/module/url} # - commit 51ad1eae73a2082e7636056fcd06e112d3fbca9c export PYTHONPATH="${PYTHONPATH}:path/to/project" experiments /path/to/project/module/experiment.py --execute ${1}
Generic experiments are compatible with the ExperimentManager provided they can be executed with a shell script. For example a bash only experiment might have a script.sh script that looks like:
```
#!/bin/bash
echo "$(cat ${1})"
```

A set of experiment folders representing individual experiments within which each experiment has a params.yml with a set of override parameters changed from the original defaults. These overrides define the unique name of each experiment (in the case that multiple experiments share the same overrides each experiment folder is additionally numbered after the first instantiation). Additionally, each params.yml contains the following:

# Parameters special to params.yml
_root: /path/to/root  #  The root directory to which the experiment belongs (should not be set)
_name: a:10__b:3 #  The name of the experiment (should not be set)
_status: registered #  The status of the experiment (one of ['registered' | 'running' | 'error' | 'finished'])
_created: 07:41PM September 16, 2019 #  The date the experiment was created (should not be set)
_purpose: this is an experiment example #  The purpose for the experiment (should not be set)
_messages: {} #  A dictionary of messages which are able to vary throughout the experiment (should not be set)

# git information is updated at registration if ``_version['module']`` or `_version['path']``
# exists in the defaults.yml file and the path is to a valid git repo.
_version:
    module: /path/to/module   # Path to module where experiment was generated
    class: NameOfExperimentClass     # Name of experiment class params are compatible with
    git: #  A dictionary containing the git history corresponding to the defaults.yml file. Only
      local_path: path/to/git/repo/params/were/defined/in
      remote_url: /remote/repo/url
      hash: 80dcfd98e6c3c17e1bafa72ee56744d4a6e30e80
                    # Parameters from the default (with values overridden)
a: 3 #  This is the first parameter (default=3)
b: '4' #  This is the second parameter (default='4')

experiment.yml preserves the experiment state with the following entries:

root: /path/to/root
defaults: /path/to/root/defaults.yml
script: /path/to/root/script.sh
experiments:
- /private/tmp/new-test/a:10__b:3
- /private/tmp/new-test/a:20__b:3
- /private/tmp/new-test/a:10__b:3_1
overides:
- a: 10
  b: '3'
- a: 20
  b: '3'
- a: 10
  b: '3'
created: 07:41PM September 16, 2019   # The date the experiment manager was initialised

The ExperimentManager provides the following public interface for managing experiments:

__init__(root):
Link the experiment manager with a root directory and load the experiments.yml if it exists

initialise(script, defaults):
Initialise an experiment set with a given script and default parameters

link(string_pattern):
Register a number of experiments overriding parameters based on the particular string_pattern

list():
Print all the experiments and their associated information

unlink(pattern):
Relieve the experiment manager of responsibility for all experiment names matching pattern

clean():
Delete any experiments which are no longer the responsibility of the experiment manager

run(string, options):
Run an experiment or all experiments (if string is 'all') with options prepended.

Example:

experiment_manager = ExperimentManager(ROOT_PATH)   # Create experiment set in ROOT_PATH
experiment_manager.initialise(PATH_TO_SCRIPT, PATH_TO_DEFAULTS)
experiment_manager.link('parama: 1, paramb: [x, y]')      # Register a set of experiments
experiment_manger.unlink('parama_1__paramb_y')                # Remove an experiment
experiment_manager.clean()                                    # Get rid of any experiments no longer managed
experiment_run('parama:1__paramb:x', sh)                      # Run an experiment
experiment_run('all')                                         # Run all created experiments

__init__(root='', headless=False)[source]¶

Link an experiment manager to root. If root already contains an experiment.yml then it is loaded.

In order to link a new experiment with a defaults.yml and script.sh file then the initialise method must be called.

Parameters

root – The root directory within which to create the experiment. If “” then the current working directory is used. If the root directory does not exist it will be made.
root – The root directory of the experiment manger
defaults – A path to the defaults.yml file. Will be None for a fresh experiment manager (if experiments.yml has just been created).
script – A path to the script.sh file. Will be None for a fresh experiment manager (if experiments.yml has just been created).
created – A string giving the date-time the experiment was created
experiments – A list of paths to the experiments managed by the experiment manager
overides – A list of dictionaries giving the names (keys) and values of the parameters overridden from the defaults for each experiment in experiments.
notes – A set of notes attched to the experiment set
purpose – The purpose of the epxeriment

check_initialised()[source]¶: Make sure that 'experiment.yml', 'script.sh', 'defaults.yml' all exist in the directory

load_defaults()[source]¶: Load the defaults.yml file into a dictionary

save_params(params, root)[source]¶

Save a dictionary of parameters at {root}/{experiment_name}/params.yml

Parameters

params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel
experiment_name (str) – The name of the experiment

update_meta()[source]¶

Save a dictionary of parameters at {root}/{experiment_name}/params.yml

Parameters

params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel
experiment_name (str) – The name of the experiment

load_params(experiment_path, experiment_name=False)[source]¶: Load parameters for an experiment. If experiment_name is True then experiment_path is assumed to be a path to the folder of the experiment else it is assumed to be a path to the params.yml file.

initialise(*, defaults='', script='', purpose='', name=None)[source]¶

Link an experiment manager with a defaults.yml file and sript.sh.

Parameters

defaults (str) – A path to a defaults.yml. If “” then a defaults.yml is searched for in the current work directory.
script (str) – A path to a script.sh. If "" then a script.sh file is searched for in the current work directory.

note(pattern, msg, remove=False)[source]¶: Add a note to experiments matching pattern. If remove is True msg is deleted instead.

move(dest)[source]¶: Move the current experiment set from one location to another.

register(name=None, string_params=None, purpose='', header=None, shell='/bin/bash', repeats=1)[source]¶

Register a set of experiments with the experiment manager.

Experiments are created by passing a yaml dictionary string of parameters to overload in the params.yml file. The special symbol '|' can be thought of as an or operator. When encountered each of the parameters either side '|' will be created separately with all other parameter combinations.

Parameters

string_params (str) –
A yaml dictionary of parameters to override of the form '{p1: val11 | val12, p2: val2, p3: val2 | p4: val31 | val32 | val33, ...}'. The type of each parameter is inferred from its value in defaults. A ValueError will be raised if any of the parameter cannot be found in defaults. Parameters can be float (1.), int (1), str (a), None, and dictionaries {a: 1., 2.} or lists [1, 2] of these types_match. None parameters are specified using empty space. The length of list parameters must match the length of the parameter in default. Dictionary parameters may only be partially defined. Missing keys will be assumed to take there default value.

The special character ‘|’ is used as an or operator. All combinations of parameters either side of an | operator will be created as separate experiments. In the example above N = 2 * 2 * 3 = 12 experiments will be generated representing all the possible values for parameters p1, p3 and p4 can take with p2 set to val2 for all.
purpose (str) – An optional purpose message for the experiment.
header (str) – An optional header message prepended to each run script.sh

Note

This function is currently only able to link list or dictionary parameters at the first level. {a: {a: 1.. b: 2.} | {a: 2.. b: 2.}} works creating two experiments with over-ridden dicts in each case but {a: {a: 1. | 2., b:2.}} will fail.

The type of each override is inferred from the type contained in the defaults.yml file (ints will be cast to floats etc.) where possible. This is not the case when there is an optional parameter that can take a None value. If None (null yaml) is passed as a value it will not be cast. If a default.yml entry is given the value null the type of any overrides in this case will be inferred from the yaml string.

reset(pattern, status='registered')[source]¶

Update the status of the experiment. This is useful if you need to re-run an experiment from a latest saved checkpoint for example.

Parameters: pattern – The experiment name

clean()[source]¶: Remove directories no longer linked to the experiment manager

run(pattern, *flags)[source]¶: Run all experiments that match the global pattern using the run command given by args.

unlink(pattern='*')[source]¶: Unlink all experiments matching pattern. Does not delete experiment folders simply deletes the experiment paths from the experiment folder. To delete call method clean.

relink(pattern='*')[source]¶: Re-link all experiment folders that match pattern (and are not currently managed by the experiment manager)

config¶

class xmen.config.GlobalExperimentManager[source]¶

DEPRECIATED A helper class used to manage global configuration of the Experiment Manager

__init__()[source]¶: Initialize self. See help(type(self)) for accurate signature.

clean()[source]¶: Iteratively search through experiment sets and remove any that no longer exist.

Note

This is performed by checking that experiment folder is a valid set using using ExperimentManager(p).check_initialised()

find(mode='all', last=None, pattern='*', param_match=None, types_match=None, load_defaults=False, missing_entry='')[source]¶

Search through all experiments in the globally filtering by experiment location, parameters, and type.

Parameters

mode (str) – If ‘all’ then all experiments for each set will be concatenated and summarised individually whilst if ‘set’ then experiments will be summarised and searched for based on the default parameters.
paths (List[str]) – Extract all information from the passed list of path to experiments. If None then a search will be performed instead.
pattern (str) – A unix glob pattern of experiments to consider
param_match (list) – A list of strings providing a condition to filter experiments by their parameters. Compatible string formats are “regex”, “regex op val” or “val1 op1 regex op2 val2” where op is in [<=,==,>=,<,>,!=]. If just a regex is supplied then only experiments with parameters matching the regex will be returned. If an op is supplied then only experiments with all parameters which match the regex and satisfy the condition will be returned.
types_match (list) – A list of types of experiments to search for
load_defaults (bool) – If True then defaults.yml files will be loaded and the parameters of each experiment inferred from the overides defined in the experiment.yml file for each experiment set. This is potentially faster (less io) but means that the messages for each experiment will not be returned.
missing_entry – Assign this parameter to missing values (see notes).

Returns

A dict with possible keys ``[‘_root’, ‘_purpose’, ‘_type’, ‘_notes’, ‘_name’, ‘_created’,: ’_version’, ‘_status’, ‘_messages’, ‘_experiments’, *param_keys]`` where param_keys correspond to any parameters that satisfy the conditions in param_match. The keys ['_root', '_purpose', '_type', '_notes', '_created'] will always be present, ['_name', '_version'] will be added if load_defaults==True and ['_name', '_version', '_status', '_messages] if load_defaults==False. If [mode == ‘set’] then the key ‘_experiments’ will also be added (giving a list of paths to the experiments in each set). The created dates in this case correspond to the date the set was created (given in defaults.yml).

special_keys (dict): A list of keys starting with ‘_’ present in the dictionary.

Return type

matches (dict)

Note

Due to the use of regex and conditions it is possible for parameters to match in one experiment which are: not present in another. In this case missing parameters are given a value equal to missing_string.

find_to_dataframe(table, verbose=True, display_git=None, display_purpose=None, display_date=None, display_messages=None, display_status=None, display_meta=None, display_params=None)[source]¶: Convert the output of the find method to a formatted data frame configured by args. If verbose then all entries will be displayed (independent of the other args)

add_experiment(module, name)[source]¶: Add a experiments experiment name defined in module either as a class or function.