Tutorial

Note, this tutorial corresponds to the jupyter notebook found at xmen/examples/tutorial.ipynb. After installing xmen you should be able to run this notebook on your local machine.

Defining Experiments

Experiments can be defined in one of two ways: 1. As experiments functions 2. As experiments classes

Functional Experiments

In the functional API experiments are defined as functions:

from xmen import Root

def hello_world(
    root: Root,   # experiments are assigned a root before being executed
    a: str = 'Hello',  # the first
    # argument comment spans multiple lines
    b: str = 'World'   # the second argument
):
    """A hello world experiment designed to demonstrate
    defining experiments through the functional experiment api"""
    print(f'{a}  {b}')

    ...  #  Whatever other experiment code you want

    with open(root.directory + '/out.txt', 'w') as f:
        f.write(f'{a} {b}')
    root.message({'a': a, 'b': b})

Arguments to the function are treated as parameters, with comments next to each parameter treated as documentation. You are also encouraged to specify parameters with their type and a default value, alongside specifying the doc string of the experiment.

Before being run each experiment is first linked to a driectory. This is supplied to the experiment as the first argument - defined as root in the example above. Principally, two methods are exposed to the experiment through root: 1. root.directory gives the directory the experiment is linked to 2. root.message() allows the experiment to leave simple messages (either floats, ints or strings) with the experiment root

Class Experiments

Alternatively experiments can be specified as experiments classes:

from xmen import Experiment

class HelloWorld(Experiment):
    """A hello world experiment designed to demonstrate
    defining experiments through the class experiment api"""
    # Parameters
    a: str = 'Hello'  # @p The first argument
    b: str = 'World'  # @p The second argument

    def run(self):
        print(f'{self.a} {self.b}!')

        with open(self.directory + '/out.txt', 'w') as f:
            f.write(f'{a} {b}')
        self.message({'a': self.a, 'b': self.b})

In this case, all experiments must inherit from Experiment. Parameters are defined as class attributes marked with a # @p comment - any method not marked with # @p will not be regarded as a parameter. The directory and messaging method are directly available to the experiment as class attributes.

Defining experiment as experiments classes allows experiments to benefit from all the features of an object orientated programming approach including encapsulation and inheritance. Further examples showing the benefits of defining experiments as classes can be found in: - /xmen/examples/inheritance.py - /xmen/examples/torch/inheritance.py

and is documented further in the class-api.ipynb.

Running Experiments

1. xmen command line interface

Any experiment that conforms to the xmen api can automatically be run in the commandline. This is facilitated by the xmen command line tool which allows multiple experiments to be quickly configured and run from the command line and is designed to easily interface with the slurm job scheduler.

[1]:
%%bash
xmen
usage: xmen [-h]
            {experiments,config,init,link,run,note,reset,unlink,clean,rm,relink,list}
            ...

||||||||||||||||||||||||||| WELCOME TO |||||||||||||||||||||||||||
||                                                              ||
||          &@&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&@&%          ||
||         *@&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&         ||
||          &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&          ||
||           &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&          ||
||           &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#          ||
||           &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&.          ||
||           &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@.          ||
||           &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@*          ||
||           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          ||
||   #&@@@@@&%&&&&&&&@@@@@@@@@@@@@@@@@@@@@@@@@&&&&&&&&@@@@@@&#  ||
||  /#%%%%%%%%%&&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&&%%&%%%%%%#  ||
||   &%&&&&&&&&&&&@@@@@@@@@@@@@@@@@@@@@@@@@@@&&@&&&&&&&&&&&&&   ||
||     (@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&.    ||
||      ...,*/#%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&##(*,...      ||
||                                                              ||
||    \\\  ///  |||\\        //|||  |||||||||  |||\\   |||      ||
||     \\\///   |||\\\      ///|||  |||        |||\\\  |||      ||
||      ||||    ||| \\\    /// |||  ||||||     ||| \\\ |||      ||
||     ///\\\   |||  \\\  ///  |||  |||        |||  \\\|||      ||
||    ///  \\\  |||   \\\///   |||  |||||||||  |||   \\|||      ||
||                                                              ||
||                      %@@,     (@@/                           ||
||                     @@@@@@@@@@@@@@@@@@@@@                    ||
||        @@        @@@@@@@@@@@@@@@@@@@@@@@@@@/        @#       ||
||       @@#     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#     @@       ||
||        @@@@@@@@@@@@@@@@@@@@@@@.@@@@@@@@@@@@@@@@@@@@@@.       ||
||           ,@@@@@@@@@@@@@@@%       @@@@@@@@@@@@@@@@           ||
||                                                              ||
|||||||||||||| FAST - REPRODUCIBLE - EXPERIMENTATION |||||||||||||

positional arguments:
  {experiments,config,init,link,run,note,reset,unlink,clean,rm,relink,list}
    experiments              Python interface
    config              View / edit the global configuration
    init                Initialise an experiment set
    link            Register a set of experiments
    run                 Run experiments matching glob in experiment set that have not yetbeen run.
    note                add notes to an experiment
    reset               Reset an experiment to registered status
    unlink              Unlink experiments from experiment set
    clean               Remove unlinked experiments (DESTRUCTIVE)
    rm                  Remove an experiment set (DESTRUCTIVE)
    relink              Relink experiments to global configuration or to a set root
    list                list experiments to screen

optional arguments:
  -h, --help            show this help message and exit

2. Adding an Experiment to Xmen

Before being run an experiment must first be registered with xmen. This is achieved by passing the module and the name of each experiment. For example the experiments defined above which are copies of hello_world and HelloWorld in xmen.examples.hello_world can be added as:

[2]:
%%bash
xmen experiments --add xmen.examples.hello_world hello_world
xmen experiments -a xmen.examples.hello_world HelloWorld
Added experiment hello_world from module xmen.examples.hello_world
Added experiment HelloWorld from module xmen.examples.hello_world

To see that the experiments have been registered with xmen we can run:

[3]:
%%bash
xmen experiments --list
The following experiments experiments are currently linked
CycleGan: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/cycle_gan/cycle_gan.py
ThereAndBackCycleGan: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/cycle_gan/there_and_back.py
MaskingByMoving: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/masking_by_moving/experiment.py
ThereAndBack: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/there_and_back/there_and_back.py
AnExperiment: /Users/robweston/xmen/examples/experiment.py
HelloWorld: /Users/robweston/.xmen/experiments/xmen.examples.hello_world.HelloWorld
MnistCGan: /Users/robweston/.xmen/experiments/xmen.examples.object.MnistCGan
hello_world: /Users/robweston/.xmen/experiments/xmen.examples.hello_world.hello_world
dcgan: /Users/robweston/.xmen/experiments/xmen.examples.torch.functional.dcgan

They can now be accessed in xmen:

[4]:
%%bash
xmen experiments hello_world
A hello world experiment designed to demonstrate
    defining experiments through the functional experiment api

Parameters:
    a: str=Hello ~ the first argument
    b: str=World ~ the second argument

For more help use --help.

Xmen automatically takes care of the command line interface including automatically reading comments next to parameters and adding them to the experiments help, alonsidge the experiments __doc__ string.

3. Initialising an experiment folder

A set of experiments is initialised from a experiments experiment definition using the xmen init command. For example,

[5]:
%%bash
xmen init -n hello_world -r ~/tmp/xmen/hello_world
Generating experiment root at /Users/robweston/tmp/xmen/hello_world
hello_world
Experiment root created at /Users/robweston/tmp/xmen/hello_world

4. Update default parameters

After a set of experiments has been initialised, the default parameters of the experiment can be changed by changing the defaults.yml generated by xmen during the xmen init call:

[6]:
cd ~/tmp/xmen/hello_world
/Users/robweston/tmp/xmen/hello_world
[7]:
%%bash
cat defaults.yml
_created: 02-08-21-19:04:29  # _created: str=now_time ~ The date the experiment was created
_meta: # _meta: Optional[Dict]=None ~ The global configuration for the experiment manager
  mac: '0x6c96cfdb71b9'
  host: dhcp55.robots.ox.ac.uk
  user: robweston
  home: /Users/robweston
  system:
    system: Darwin
    node: dhcp55.robots.ox.ac.uk
    release: 16.7.0
    version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
    machine: x86_64
    processor: i386
  virtual:
    total: 16.00GB
    free: 6.03GB
    used: 9.09GB
    percentage: 62.3%
  swap:
    total: 2.00GB
    free: ' 1.54GB'
    used: 470.25MB
    percentage: '23.0'
_version: # _version: Optional[Dict[Any, Any]]=None ~ Experiment version information. See `get_version`
  module: xmen.examples.hello_world
  function: hello_world
  path: /Users/robweston/xmen/xmen/examples/hello_world.py
  git:
    local: /Users/robweston/xmen
    remote: https://github.com/robw4/xmen.git
    commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
    branch: master
a: Hello # a: str=Hello ~ the first argument
b: World # b: str=World ~ the second argument

5. Register experiments

After this experiments are registered to the experiment set using the xment link command:

[8]:
%%bash
xmen link -u "{a: Hello | Bye, b: World | Planet}"

The -u flag is used to specify the parameters that should be updated for each parameter instance. It is passed as a yaml dictionary with the | character used as an or operator. For example, in this case four experiments are registered corresponding to:

a

b

Hello

World

Hello

Planet

Bye

World

Bye

Planet

The experiments registered in the experiment set are summarised using the xmen list command:

[9]:
%%bash
xmen list -sd
/Users/robweston/tmp/xmen/hello_world*
          root               name      status              created
0  hello_world   a=Hello__b=World  registered  2021-02-08-19-04-30
1  hello_world  a=Hello__b=Planet  registered  2021-02-08-19-04-30
2  hello_world     a=Bye__b=World  registered  2021-02-08-19-04-30
3  hello_world    a=Bye__b=Planet  registered  2021-02-08-19-04-30

Roots relative to: /Users/robweston/tmp/xmen

6. Run experiments

After registering experiments are run using the xmen run command:

[10]:
%%bash
xmen run "*" bash
Bye  Planet
Bye  World
Hello  Planet
Hello  World

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh

If we now run

[11]:
%%bash
xmen list -sd -m "a|b"
/Users/robweston/tmp/xmen/hello_world*
          root               name    status              created      a       b
0  hello_world   a=Hello__b=World  finished  2021-02-08-19-04-35  Hello   World
1  hello_world  a=Hello__b=Planet  finished  2021-02-08-19-04-34  Hello  Planet
2  hello_world     a=Bye__b=World  finished  2021-02-08-19-04-33    Bye   World
3  hello_world    a=Bye__b=Planet  finished  2021-02-08-19-04-32    Bye  Planet

Roots relative to: /Users/robweston/tmp/xmen

the experiment status has been changed from registered to finished and each message has been logged. Once finished experiments cannot be re-run unless they are first re-registered. This is achieved using the xmen reset command:

[12]:
%%bash
xmen reset "*"
xmen run "*" bash
xmen list -sd -m "a|b"
Bye  Planet
Bye  World
Hello  Planet
Hello  World

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
/Users/robweston/tmp/xmen/hello_world*
          root               name    status              created      a       b
0  hello_world   a=Hello__b=World  finished  2021-02-08-19-04-41  Hello   World
1  hello_world  a=Hello__b=Planet  finished  2021-02-08-19-04-40  Hello  Planet
2  hello_world     a=Bye__b=World  finished  2021-02-08-19-04-39    Bye   World
3  hello_world    a=Bye__b=Planet  finished  2021-02-08-19-04-38    Bye  Planet

Roots relative to: /Users/robweston/tmp/xmen

In this case notice how the created time has been updated. This correpsonds to the time the experiment was last run. Various options exist for running experiments using the xmen run command. For example we could run each experiment in a screen command as:

[13]:
%%bash
xmen reset "*"
xmen run "*" screen -dm bash
xmen list -sd -m "a|b"

Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh

Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh

Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh

Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
/Users/robweston/tmp/xmen/hello_world*
          root               name    status              created      a       b
0  hello_world   a=Hello__b=World  finished  2021-02-08-19-04-45  Hello   World
1  hello_world  a=Hello__b=Planet  finished  2021-02-08-19-04-44  Hello  Planet
2  hello_world     a=Bye__b=World  finished  2021-02-08-19-04-44    Bye   World
3  hello_world    a=Bye__b=Planet  finished  2021-02-08-19-04-44    Bye  Planet

Roots relative to: /Users/robweston/tmp/xmen

Features

Experiment Visualisation

Experiments can be visualised using the xmen list command. For more info see:

[14]:
%%bash
xmen list --help
usage: xmen list [-h] [-p [PARAM_MATCH [PARAM_MATCH ...]]]
                 [-n [TYPE_MATCH [TYPE_MATCH ...]]] [-v] [-d] [-g] [-P] [-s]
                 [-m [DISPLAY_MESSAGES]] [-M [DISPLAY_META]] [-l]
                 [--load_defaults] [--max_width MAX_WIDTH]
                 [--max_rows MAX_ROWS] [--csv]
                 [pattern [pattern ...]]

positional arguments:
  pattern               List experiments which match pattern.

optional arguments:
  -h, --help            show this help message and exit
  -p [PARAM_MATCH [PARAM_MATCH ...]], --param_match [PARAM_MATCH [PARAM_MATCH ...]]
                        List only experiments with certain parameter
                        conditions of the form reg, reg==val or
                        val1==reg==val2. Here reg is a regex matching a set of
                        parameters. ==, <, >, !=, >=, <= are all supported
                        with meaning defined as in experiments. Eg. a.*==cat and
                        1.0<a.*<=2.0 will return any experiment that has
                        parameters that match a.* provided that each match
                        satisfies the condition.
  -n [TYPE_MATCH [TYPE_MATCH ...]], --type_match [TYPE_MATCH [TYPE_MATCH ...]]
                        List only experiments with this type (class).
  -v, --verbose         Display all information for each experiment
  -d, --display_date    Display created date for each experiment
  -g, --display_git     Display git commit for each experiment
  -P, --display_purpose
                        Display purpose for each experiment
  -s, --display_status  Display status for each experiment
  -m [DISPLAY_MESSAGES], --display_messages [DISPLAY_MESSAGES]
                        Display messages for each experiment
  -M [DISPLAY_META], --display_meta [DISPLAY_META]
                        Display meta information for each experiment. The
                        regex '^root$|^name$|^mac$|^host$|^user$|^home$' gives
                        basic meta information logged with every experiment.
                        Other information is separated into groups including
                        'network.*', 'gpu.*', 'cpu.*', 'system.*',
                        'virtual.*', 'swap.*'
  -l, --list            Display as list and not a table
  --load_defaults       Infer parameters from defaults.yml and overides
                        instead of params.yml. Potentially faster but no
                        messages are available.
  --max_width MAX_WIDTH
                        The maximum width of an individual collumn. If None
                        then will print for ever
  --max_rows MAX_ROWS   Display tables with this number of rows.
  --csv                 Display the table as csv.

Git support

When running experiments xmen logs lots of useful information automatically. For example, the git information for the repo in which the experiement is defined. This can be found in the params.yml folder and can be visualised as:

[15]:
%%bash
xmen list -g
/Users/robweston/tmp/xmen/hello_world*
          root               name                  local                             remote                                    commit  branch
0  hello_world   a=Hello__b=World  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master
1  hello_world  a=Hello__b=Planet  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master
2  hello_world     a=Bye__b=World  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master
3  hello_world    a=Bye__b=Planet  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master

Roots relative to: /Users/robweston/tmp/xmen

Automatic Record Keeping

Other meta information is also logged and can be visualised using the meta flag -M

[16]:
%%bash
# basic meta information
xmen list -M
/Users/robweston/tmp/xmen/hello_world*
          root               name             mac                    host       user              home
0  hello_world   a=Hello__b=World  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
1  hello_world  a=Hello__b=Planet  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
2  hello_world     a=Bye__b=World  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
3  hello_world    a=Bye__b=Planet  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston

Roots relative to: /Users/robweston/tmp/xmen
[17]:
%%bash
# system info
xmen list -M "system_.*"
/Users/robweston/tmp/xmen/hello_world*
          root               name system_system             system_node system_release                                               system_version system_machine system_processor
0  hello_world   a=Hello__b=World        Darwin  dhcp55.robots.ox.ac.uk         16.7.0  Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20...         x86_64             i386
1  hello_world  a=Hello__b=Planet        Darwin  dhcp55.robots.ox.ac.uk         16.7.0  Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20...         x86_64             i386
2  hello_world     a=Bye__b=World        Darwin  dhcp55.robots.ox.ac.uk         16.7.0  Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20...         x86_64             i386
3  hello_world    a=Bye__b=Planet        Darwin  dhcp55.robots.ox.ac.uk         16.7.0  Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20...         x86_64             i386

Roots relative to: /Users/robweston/tmp/xmen

Meta information is also logged for "cpu_.*", "swap_.*", "virtual_.*", "gpu_.*". In addition, a more extensive summary of the environment is logged when each experiment is run and can be found in the meta.yml file in the experiments folder including network information and global flags. For example:

[18]:
%%bash
cat ~/tmp/xmen/hello_world/a=Hello__b=World/meta.yml
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
  system: Darwin
  node: dhcp55.robots.ox.ac.uk
  release: 16.7.0
  version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
  machine: x86_64
  processor: i386
cpu:
  physical: 4
  total: 8
  max_freq: 2200.00Mhz
  min_freq: 2200.00Mhz
  cur_freq: 2200.00Mhz
virtual:
  total: 16.00GB
  free: 6.15GB
  used: 8.97GB
  percentage: 61.5%
swap:
  total: 2.00GB
  free: ' 1.54GB'
  used: 470.25MB
  percentage: '23.0'
disks:
  /dev/disk1:
    mountpoint: /
    fstype: hfs
    total: 464.78GB
    used: 430.99GB
    free: 33.54GB
    percent: 92.8%
  read: 90.18GB
  write: 147.87GB
network:
  lo0:
    AF_INET6:
      address: fe80::1%lo0
      netmask: 'ffff:ffff:ffff:ffff::'
      broadcast: None
  en0:
    AF_INET6:
      address: fe80::cf3:b64a:220b:17d8%en0
      netmask: 'ffff:ffff:ffff:ffff::'
      broadcast: None
  en6:
    AF_INET6:
      address: fe80::107d:1836:a969:5191%en6
      netmask: 'ffff:ffff:ffff:ffff::'
      broadcast: None
  en1:
    AF_LINK:
      address: 6a:00:03:5b:d3:f0
      netmask: None
      broadcast: None
  en2:
    AF_LINK:
      address: 6a:00:03:5b:d3:f1
      netmask: None
      broadcast: None
  bridge0:
    AF_LINK:
      address: 6a:00:03:5b:d3:f0
      netmask: None
      broadcast: None
  p2p0:
    AF_LINK:
      address: 0e:96:cf:db:71:b9
      netmask: None
      broadcast: None
  awdl0:
    AF_INET6:
      address: fe80::d86a:adff:fe3b:d95%awdl0
      netmask: 'ffff:ffff:ffff:ffff::'
      broadcast: None
  utun0:
    AF_INET6:
      address: fe80::77eb:8860:4aa1:d43b%utun0
      netmask: 'ffff:ffff:ffff:ffff::'
      broadcast: None
  sent: 21.37GB
  received: 37.36GB
environ:
  PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: cpp
  TERM_PROGRAM: iTerm.app
  TERM: screen
  SHELL: /bin/zsh
  CLICOLOR: '1'
  TMPDIR: /var/folders/3f/7jm6_wkj6rq0znrt1zgbg7hm0000gn/T/
  CONDA_SHLVL: '2'
  Apple_PubSub_Socket_Render: /private/tmp/com.apple.launchd.fPKHCGpHV7/Render
  CONDA_PROMPT_MODIFIER: '(python3.6) '
  TERM_PROGRAM_VERSION: 3.1.5
  TERM_SESSION_ID: w0t4p0:7B70FECE-A287-43A3-A009-150E7FF3AA01
  ZSH: /Users/robweston/.oh-my-zsh
  USER: robweston
  CONDA_EXE: /Users/robweston/miniconda3/bin/conda
  TERMCAP: "SC|screen|VT 100/ANSI X3.64 virtual terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\\
    E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\\
    E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\\
    E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#24:co#80:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\\
    E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\\
    E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\\
    E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:k0=\\E[10~:k1=\\EOP:k2=\\\
    EOQ:\\\n\t:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:\\\n\t:k8=\\E[19~:k9=\\\
    E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:\\\n\t:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\\
    E[4~:kN=\\E[6~:kP=\\E[5~:\\\n\t:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\\
    EOD:"
  SSH_AUTH_SOCK: /private/tmp/com.apple.launchd.c8iOoBTn71/Listeners
  KERNEL_LAUNCH_TIMEOUT: '40'
  __CF_USER_TEXT_ENCODING: 0x1F5:0x0:0x2
  JPY_PARENT_PID: '89428'
  PAGER: cat
  _CE_CONDA: ''
  LSCOLORS: Gxfxcxdxbxegedabagacad
  CONDA_PREFIX_1: /Users/robweston/miniconda3
  PATH: /Users/robweston/.rbenv/shims:/Users/robweston/.gem/ruby/2.7.0/bin:/usr/local/opt/ruby/bin:/Users/robweston/.rbenv/shims:/Users/robweston/miniconda3/envs/python3.6/bin:/Users/robweston/miniconda3/condabin:/Users/robweston/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/git/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
  STY: 2844..dhcp55
  _: /Users/robweston/tmp/xmen/hello_world/xmen.examples.hello_world.hello_world
  CONDA_PREFIX: /Users/robweston/miniconda3/envs/python3.6
  PWD: /Users/robweston/tmp/xmen/hello_world
  MPLBACKEND: module://ipykernel.pylab.backend_inline
  LANG: en_GB.UTF-8
  ITERM_PROFILE: Default
  XPC_FLAGS: '0x0'
  RBENV_SHELL: zsh
  _CE_M: ''
  CXX: /usr/bin/clang++
  XPC_SERVICE_NAME: '0'
  SHLVL: '4'
  HOME: /Users/robweston
  COLORFGBG: 12;8
  ITERM_SESSION_ID: w0t4p0:7B70FECE-A287-43A3-A009-150E7FF3AA01
  CONDA_PYTHON_EXE: /Users/robweston/miniconda3/bin/experiments
  PYTHONPATH: :/Users/robweston/code/experiments-mrg-viewer/build-conda-build-conda-py36/experiments:/Users/robweston/code/experiments-moos/build/lib:/Users/robweston/xmen/experiments:/Users/robweston/projects/rad2sim2rad/experiments
  LESS: -R
  LOGNAME: robweston
  WINDOW: '0'
  LC_CTYPE: en_GB.UTF-8
  CONDA_DEFAULT_ENV: python3.6
  PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION: '2'
  CC: /usr/bin/clang
  DISPLAY: /private/tmp/com.apple.launchd.9Ugqo2pZGO/org.macosforge.xquartz:0
  GIT_PAGER: cat
  COLORTERM: truecolor

Note Taking

Notes can be manually added to an experiment using the xmen note command:

[19]:
%%bash
xmen note --help
usage: xmen note [-h] [-r DIR] [-d] message

positional arguments:
  message             Add note to experiment set

optional arguments:
  -h, --help          show this help message and exit
  -r DIR, --root DIR  Path to the root experiment folder. If None then the
                      current work directory will be used
  -d, --delete        Delete the note corresponding to message.

For example to add a note to the experiments run:

[20]:
%%bash
xmen note "example of adding a note"

Xmens list command provides an easy way to viusalise notes attached to an experiment.

[21]:
%%bash
xmen list -l
 0 /Users/robweston/tmp/xmen/hello_world
     |- a=Hello__b=World
     |- a=Hello__b=Planet
     |- a=Bye__b=World
     |- a=Bye__b=Planet
     purpose:
     created: 02-08-21-19:04:29
     version:
       module: xmen.examples.hello_world
       function: hello_world
       path: /Users/robweston/xmen/xmen/examples/hello_world.py
       git:
         local: /Users/robweston/xmen
         remote: https://github.com/robw4/xmen.git
         commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
         branch: master
     notes:
       example of adding a note

Conda support

To make reproducing experiments as easy as possible, alongside the git version information, the conda environent is also logged as an experiment.yml file when each experiment is run. To turn this feature on run:

[22]:
%%bash
xmen config --enable_save_conda

Now lets re-run "a=Hello__b=World":

[23]:
%%bash
xmen reset "a=Hello__b=World"
xmen run "a=Hello__b=World" bash
/Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/environment.yml
Hello  World

Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh

In this case an environment.yml file has been generated automatically once the experiment was run

[24]:
%%bash
head -20 ~/tmp/xmen/hello_world/a=Hello__b=World/environment.yml
name: python3.6
channels:
- pytorch
- anaconda
- conda-forge
- defaults
dependencies:
- _tflow_select=2.3.0=mkl
- absl-py=0.8.0=py36_0
- alabaster=0.7.12=py36_0
- appnope=0.1.0=py36hf537a9a_0
- asn1crypto=1.0.1=py36_0
- astor=0.8.0=py36_0
- attrs=19.2.0=py_0
- babel=2.7.0=py_0
- backcall=0.1.0=py36_0
- beautifulsoup4=4.8.0=py36_0
- blas=1.0=mkl
- bleach=3.1.0=py36_0
- bzip2=1.0.8=h1de35cc_0

Note, that while this is a useful feature this does have some additional overhead particularly when working with bloated conda environments. The feature can therefore be disabled as:

[25]:
%%bash
xmen config --disable_save_conda

Interfacing with the slurm job scheduler

Another powerful option for running experiments on high performance computing clusters is the slurm job scheduler. To run experiments using the slurm job scheduler first add a header,

[26]:
%%bash
cat ~/xmen/examples/header.txt
#SBATCH --nodes=1
#SBATCH --job-name=single_job
#SBATCH --time=1-00:00:00
#SBATCH --gres=gpu:1
#SBATCH --constraint='gpu_sku:RTX|gpu_sku:V100-LS|gpu_sku:V100|gpu_sku:P100'
#SBATCH --partition=htc-nova,htc
#SBATCH --cpus-per-task=2
#SBATCH --mail-user=robw@robots.ox.ac.uk
#SBATCH --mail-type=ALL
#SBATCH --account=engs-a2i
#SBATCH --signal=SIGUSR1@90
# Author: Rob Weston
# Email: robw@robots.ox.ac.uk

to the global configuration:

[27]:
%%bash
xmen config -H ~/xmen/examples/header.txt

This will be appended to each run script generated by xmen. Experiments can then be submitted using the steps described before, but this time running each job with the sbatch command:

xmen run "*" sbatch

Note the header is used to set default paraeter values. They can still be overidden by passing command line flags to sbatch. For example the --job-name flag can be set to override the name of the experiment as:

xmen run "*" sbatch --job-name=new_name

Language agnostic experiment configuration

Whilst so far we have considered defining and running experiments in experiments xmen is designed to work with other types of experiments. A general experiment can be defined through a script.sh which takes a yaml file of parameters as its first argment. For example, a trivial bash experiment script.sh which takes a yaml file and prints to screen might look something like:

#! /bin/bash
echo "contents of params.yml file ${1}"
cat ${1}

Next to interface with the xmen command line tool we need to generate a set of default paramaters in a yaml file called defaults.yml. The script xmen.examples.bash.generate is provided to do this for you:

[28]:
%%bash
experiments -m xmen.examples.bash.generate ~/tmp/xmen/bash

Now when we look in ~/tmp/xmen/bash we find a copy of the script.sh defined above alongside a defaults.yml file.

[29]:
cd ~/tmp/xmen/bash
/Users/robweston/tmp/xmen/bash
[30]:
%%bash
ls -l ~/tmp/xmen/bash
total 16
-rw-r--r--  1 robweston  staff  109  8 Feb 19:05 defaults.yml
-rw-r--r--  1 robweston  staff  164  8 Feb 19:05 script.sh

The defaults.yml file defines two parameters a and b. The special parameters _version and _created can also be included. Here, _created gives the date the file was created whilst version is a dictionary with a single entry giving the path to the file in which the parameters were originally defined.

[31]:
%%bash
cat ~/tmp/xmen/bash/defaults.yml
_version:
  path: /Users/robweston/xmen/xmen/examples/bash/generate.py
_created: 8/2/2021
a: Hello
b: Planet

It is now possible to initialise, link and run experiments using the xmen command line tool in a similar way to before.

[32]:
%%bash
xmen init
None
Defaults from /Users/robweston/tmp/xmen/bash/defaults.yml
Script from /Users/robweston/tmp/xmen/bash/script.sh
Experiment root created at /Users/robweston/tmp/xmen/bash
[33]:
%%bash
xmen link -u "{a: Hello | Bye, b: World| Planet}"
[34]:
%%bash
xmen run "*" bash
#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Bye__b=Planet/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash  # The root directory of the experiment
_name: a=Bye__b=Planet # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
  path: /Users/robweston/xmen/xmen/examples/bash/generate.py
  git:
    local: /Users/robweston/xmen
    remote: https://github.com/robw4/xmen.git
    commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
    branch: master
_meta: # The global configuration for the experiment manager
  mac: '0x6c96cfdb71b9'
  host: dhcp55.robots.ox.ac.uk
  user: robweston
  home: /Users/robweston
  system:
    system: Darwin
    node: dhcp55.robots.ox.ac.uk
    release: 16.7.0
    version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
    machine: x86_64
    processor: i386
  cpu:
    physical: 4
    total: 8
    max_freq: 2200.00Mhz
    min_freq: 2200.00Mhz
    cur_freq: 2200.00Mhz
  virtual:
    total: 16.00GB
    free: 6.10GB
    used: 9.02GB
    percentage: 61.9%
  swap:
    total: 2.00GB
    free: ' 1.54GB'
    used: 470.25MB
    percentage: '23.0'
a: Bye
b: Planet

#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Bye__b=World/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash  # The root directory of the experiment
_name: a=Bye__b=World # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
  path: /Users/robweston/xmen/xmen/examples/bash/generate.py
  git:
    local: /Users/robweston/xmen
    remote: https://github.com/robw4/xmen.git
    commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
    branch: master
_meta: # The global configuration for the experiment manager
  mac: '0x6c96cfdb71b9'
  host: dhcp55.robots.ox.ac.uk
  user: robweston
  home: /Users/robweston
  system:
    system: Darwin
    node: dhcp55.robots.ox.ac.uk
    release: 16.7.0
    version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
    machine: x86_64
    processor: i386
  cpu:
    physical: 4
    total: 8
    max_freq: 2200.00Mhz
    min_freq: 2200.00Mhz
    cur_freq: 2200.00Mhz
  virtual:
    total: 16.00GB
    free: 6.11GB
    used: 9.02GB
    percentage: 61.8%
  swap:
    total: 2.00GB
    free: ' 1.54GB'
    used: 470.25MB
    percentage: '23.0'
a: Bye
b: World

#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Hello__b=Planet/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash  # The root directory of the experiment
_name: a=Hello__b=Planet # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
  path: /Users/robweston/xmen/xmen/examples/bash/generate.py
  git:
    local: /Users/robweston/xmen
    remote: https://github.com/robw4/xmen.git
    commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
    branch: master
_meta: # The global configuration for the experiment manager
  mac: '0x6c96cfdb71b9'
  host: dhcp55.robots.ox.ac.uk
  user: robweston
  home: /Users/robweston
  system:
    system: Darwin
    node: dhcp55.robots.ox.ac.uk
    release: 16.7.0
    version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
    machine: x86_64
    processor: i386
  cpu:
    physical: 4
    total: 8
    max_freq: 2200.00Mhz
    min_freq: 2200.00Mhz
    cur_freq: 2200.00Mhz
  virtual:
    total: 16.00GB
    free: 6.10GB
    used: 9.02GB
    percentage: 61.9%
  swap:
    total: 2.00GB
    free: ' 1.54GB'
    used: 470.25MB
    percentage: '23.0'
a: Hello
b: Planet

#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Hello__b=World/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash  # The root directory of the experiment
_name: a=Hello__b=World # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
  path: /Users/robweston/xmen/xmen/examples/bash/generate.py
  git:
    local: /Users/robweston/xmen
    remote: https://github.com/robw4/xmen.git
    commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
    branch: master
_meta: # The global configuration for the experiment manager
  mac: '0x6c96cfdb71b9'
  host: dhcp55.robots.ox.ac.uk
  user: robweston
  home: /Users/robweston
  system:
    system: Darwin
    node: dhcp55.robots.ox.ac.uk
    release: 16.7.0
    version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
    machine: x86_64
    processor: i386
  cpu:
    physical: 4
    total: 8
    max_freq: 2200.00Mhz
    min_freq: 2200.00Mhz
    cur_freq: 2200.00Mhz
  virtual:
    total: 16.00GB
    free: 6.10GB
    used: 9.02GB
    percentage: 61.8%
  swap:
    total: 2.00GB
    free: ' 1.54GB'
    used: 470.25MB
    percentage: '23.0'
a: Hello
b: World


Running: bash /Users/robweston/tmp/xmen/bash/a=Bye__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/bash/a=Bye__b=World/run.sh

Running: bash /Users/robweston/tmp/xmen/bash/a=Hello__b=Planet/run.sh

Running: bash /Users/robweston/tmp/xmen/bash/a=Hello__b=World/run.sh

Alongside allowing experiments to be initialised and run quickly, xmen still takes care of record keeping behind the scenes, automatically logging git and system information as before:

[35]:
%%bash
xmen list -dsgM
/Users/robweston/tmp/xmen/bash*
   root               name                  local                             remote                                    commit  branch      status              created             mac                    host       user              home
0  bash   a=Hello__b=World  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master  registered  2021-02-08-19-05-03  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
1  bash  a=Hello__b=Planet  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master  registered  2021-02-08-19-05-03  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
2  bash     a=Bye__b=World  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master  registered  2021-02-08-19-05-03  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston
3  bash    a=Bye__b=Planet  /Users/robweston/xmen  https://github.com/robw4/xmen.git  7b6e0734d93fdaad5cd8daac012ecef5afb237f5  master  registered  2021-02-08-19-05-03  0x6c96cfdb71b9  dhcp55.robots.ox.ac.uk  robweston  /Users/robweston

Roots relative to: /Users/robweston/tmp/xmen

Clean Up

[36]:
%%bash
# remove the ~/tmp/xmen folder
rm -rf ~/tmp/xmen

Next Steps

  • For further help running any of the command line tools try xmen --help, xmen hello_world --help

  • More exampels of the experiments can be found in xmen.examples

  • Check out the API docs and the cheat sheet

[ ]: