Tutorial¶
Note, this tutorial corresponds to the jupyter notebook found at xmen/examples/tutorial.ipynb
. After installing xmen you should be able to run this notebook on your local machine.
Defining Experiments¶
Experiments can be defined in one of two ways: 1. As experiments functions 2. As experiments classes
Functional Experiments¶
In the functional API experiments are defined as functions:
from xmen import Root
def hello_world(
root: Root, # experiments are assigned a root before being executed
a: str = 'Hello', # the first
# argument comment spans multiple lines
b: str = 'World' # the second argument
):
"""A hello world experiment designed to demonstrate
defining experiments through the functional experiment api"""
print(f'{a} {b}')
... # Whatever other experiment code you want
with open(root.directory + '/out.txt', 'w') as f:
f.write(f'{a} {b}')
root.message({'a': a, 'b': b})
Arguments to the function are treated as parameters, with comments next to each parameter treated as documentation. You are also encouraged to specify parameters with their type and a default value, alongside specifying the doc string of the experiment.
Before being run each experiment is first linked to a driectory. This is supplied to the experiment as the first argument - defined as root
in the example above. Principally, two methods are exposed to the experiment through root: 1. root.directory
gives the directory the experiment is linked to 2. root.message()
allows the experiment to leave simple messages (either floats, ints or strings) with the experiment root
Class Experiments¶
Alternatively experiments can be specified as experiments classes:
from xmen import Experiment
class HelloWorld(Experiment):
"""A hello world experiment designed to demonstrate
defining experiments through the class experiment api"""
# Parameters
a: str = 'Hello' # @p The first argument
b: str = 'World' # @p The second argument
def run(self):
print(f'{self.a} {self.b}!')
with open(self.directory + '/out.txt', 'w') as f:
f.write(f'{a} {b}')
self.message({'a': self.a, 'b': self.b})
In this case, all experiments must inherit from Experiment
. Parameters are defined as class attributes marked with a # @p
comment - any method not marked with # @p
will not be regarded as a parameter. The directory and messaging method are directly available to the experiment as class attributes.
Defining experiment as experiments classes allows experiments to benefit from all the features of an object orientated programming approach including encapsulation and inheritance. Further examples showing the benefits of defining experiments as classes can be found in: - /xmen/examples/inheritance.py
- /xmen/examples/torch/inheritance.py
and is documented further in the class-api.ipynb
.
Running Experiments¶
1. xmen command line interface¶
Any experiment that conforms to the xmen api can automatically be run in the commandline. This is facilitated by the xmen
command line tool which allows multiple experiments to be quickly configured and run from the command line and is designed to easily interface with the slurm job scheduler.
[1]:
%%bash
xmen
usage: xmen [-h]
{experiments,config,init,link,run,note,reset,unlink,clean,rm,relink,list}
...
||||||||||||||||||||||||||| WELCOME TO |||||||||||||||||||||||||||
|| ||
|| &@&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&@&% ||
|| *@&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@& ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@& ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@& ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&. ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@. ||
|| &@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@* ||
|| @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ||
|| #&@@@@@&%&&&&&&&@@@@@@@@@@@@@@@@@@@@@@@@@&&&&&&&&@@@@@@&# ||
|| /#%%%%%%%%%&&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&&%%&%%%%%%# ||
|| &%&&&&&&&&&&&@@@@@@@@@@@@@@@@@@@@@@@@@@@&&@&&&&&&&&&&&&& ||
|| (@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&. ||
|| ...,*/#%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@&##(*,... ||
|| ||
|| \\\ /// |||\\ //||| ||||||||| |||\\ ||| ||
|| \\\/// |||\\\ ///||| ||| |||\\\ ||| ||
|| |||| ||| \\\ /// ||| |||||| ||| \\\ ||| ||
|| ///\\\ ||| \\\ /// ||| ||| ||| \\\||| ||
|| /// \\\ ||| \\\/// ||| ||||||||| ||| \\||| ||
|| ||
|| %@@, (@@/ ||
|| @@@@@@@@@@@@@@@@@@@@@ ||
|| @@ @@@@@@@@@@@@@@@@@@@@@@@@@@/ @# ||
|| @@# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# @@ ||
|| @@@@@@@@@@@@@@@@@@@@@@@.@@@@@@@@@@@@@@@@@@@@@@. ||
|| ,@@@@@@@@@@@@@@@% @@@@@@@@@@@@@@@@ ||
|| ||
|||||||||||||| FAST - REPRODUCIBLE - EXPERIMENTATION |||||||||||||
positional arguments:
{experiments,config,init,link,run,note,reset,unlink,clean,rm,relink,list}
experiments Python interface
config View / edit the global configuration
init Initialise an experiment set
link Register a set of experiments
run Run experiments matching glob in experiment set that have not yetbeen run.
note add notes to an experiment
reset Reset an experiment to registered status
unlink Unlink experiments from experiment set
clean Remove unlinked experiments (DESTRUCTIVE)
rm Remove an experiment set (DESTRUCTIVE)
relink Relink experiments to global configuration or to a set root
list list experiments to screen
optional arguments:
-h, --help show this help message and exit
2. Adding an Experiment to Xmen¶
Before being run an experiment must first be registered with xmen
. This is achieved by passing the module and the name of each experiment. For example the experiments defined above which are copies of hello_world
and HelloWorld
in xmen.examples.hello_world
can be added as:
[2]:
%%bash
xmen experiments --add xmen.examples.hello_world hello_world
xmen experiments -a xmen.examples.hello_world HelloWorld
Added experiment hello_world from module xmen.examples.hello_world
Added experiment HelloWorld from module xmen.examples.hello_world
To see that the experiments have been registered with xmen
we can run:
[3]:
%%bash
xmen experiments --list
The following experiments experiments are currently linked
CycleGan: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/cycle_gan/cycle_gan.py
ThereAndBackCycleGan: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/cycle_gan/there_and_back.py
MaskingByMoving: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/masking_by_moving/experiment.py
ThereAndBack: /Users/robweston/projects/rad2sim2rad/experiments/rad2sim2rad/experiments/there_and_back/there_and_back.py
AnExperiment: /Users/robweston/xmen/examples/experiment.py
HelloWorld: /Users/robweston/.xmen/experiments/xmen.examples.hello_world.HelloWorld
MnistCGan: /Users/robweston/.xmen/experiments/xmen.examples.object.MnistCGan
hello_world: /Users/robweston/.xmen/experiments/xmen.examples.hello_world.hello_world
dcgan: /Users/robweston/.xmen/experiments/xmen.examples.torch.functional.dcgan
They can now be accessed in xmen
:
[4]:
%%bash
xmen experiments hello_world
A hello world experiment designed to demonstrate
defining experiments through the functional experiment api
Parameters:
a: str=Hello ~ the first argument
b: str=World ~ the second argument
For more help use --help.
Xmen automatically takes care of the command line interface including automatically reading comments next to parameters and adding them to the experiments help, alonsidge the experiments __doc__
string.
3. Initialising an experiment folder¶
A set of experiments is initialised from a experiments experiment definition using the xmen init
command. For example,
[5]:
%%bash
xmen init -n hello_world -r ~/tmp/xmen/hello_world
Generating experiment root at /Users/robweston/tmp/xmen/hello_world
hello_world
Experiment root created at /Users/robweston/tmp/xmen/hello_world
4. Update default parameters¶
After a set of experiments has been initialised, the default parameters of the experiment can be changed by changing the defaults.yml
generated by xmen during the xmen init
call:
[6]:
cd ~/tmp/xmen/hello_world
/Users/robweston/tmp/xmen/hello_world
[7]:
%%bash
cat defaults.yml
_created: 02-08-21-19:04:29 # _created: str=now_time ~ The date the experiment was created
_meta: # _meta: Optional[Dict]=None ~ The global configuration for the experiment manager
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
virtual:
total: 16.00GB
free: 6.03GB
used: 9.09GB
percentage: 62.3%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
_version: # _version: Optional[Dict[Any, Any]]=None ~ Experiment version information. See `get_version`
module: xmen.examples.hello_world
function: hello_world
path: /Users/robweston/xmen/xmen/examples/hello_world.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
a: Hello # a: str=Hello ~ the first argument
b: World # b: str=World ~ the second argument
5. Register experiments¶
After this experiments are registered to the experiment set using the xment link
command:
[8]:
%%bash
xmen link -u "{a: Hello | Bye, b: World | Planet}"
The -u
flag is used to specify the parameters that should be updated for each parameter instance. It is passed as a yaml dictionary with the |
character used as an or operator. For example, in this case four experiments are registered corresponding to:
a |
b |
---|---|
Hello |
World |
Hello |
Planet |
Bye |
World |
Bye |
Planet |
The experiments registered in the experiment set are summarised using the xmen list
command:
[9]:
%%bash
xmen list -sd
/Users/robweston/tmp/xmen/hello_world*
root name status created
0 hello_world a=Hello__b=World registered 2021-02-08-19-04-30
1 hello_world a=Hello__b=Planet registered 2021-02-08-19-04-30
2 hello_world a=Bye__b=World registered 2021-02-08-19-04-30
3 hello_world a=Bye__b=Planet registered 2021-02-08-19-04-30
Roots relative to: /Users/robweston/tmp/xmen
6. Run experiments¶
After registering experiments are run using the xmen run
command:
[10]:
%%bash
xmen run "*" bash
Bye Planet
Bye World
Hello Planet
Hello World
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
If we now run
[11]:
%%bash
xmen list -sd -m "a|b"
/Users/robweston/tmp/xmen/hello_world*
root name status created a b
0 hello_world a=Hello__b=World finished 2021-02-08-19-04-35 Hello World
1 hello_world a=Hello__b=Planet finished 2021-02-08-19-04-34 Hello Planet
2 hello_world a=Bye__b=World finished 2021-02-08-19-04-33 Bye World
3 hello_world a=Bye__b=Planet finished 2021-02-08-19-04-32 Bye Planet
Roots relative to: /Users/robweston/tmp/xmen
the experiment status has been changed from registered to finished and each message has been logged. Once finished experiments cannot be re-run unless they are first re-registered. This is achieved using the xmen reset
command:
[12]:
%%bash
xmen reset "*"
xmen run "*" bash
xmen list -sd -m "a|b"
Bye Planet
Bye World
Hello Planet
Hello World
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
/Users/robweston/tmp/xmen/hello_world*
root name status created a b
0 hello_world a=Hello__b=World finished 2021-02-08-19-04-41 Hello World
1 hello_world a=Hello__b=Planet finished 2021-02-08-19-04-40 Hello Planet
2 hello_world a=Bye__b=World finished 2021-02-08-19-04-39 Bye World
3 hello_world a=Bye__b=Planet finished 2021-02-08-19-04-38 Bye Planet
Roots relative to: /Users/robweston/tmp/xmen
In this case notice how the created time has been updated. This correpsonds to the time the experiment was last run. Various options exist for running experiments using the xmen run
command. For example we could run each experiment in a screen command as:
[13]:
%%bash
xmen reset "*"
xmen run "*" screen -dm bash
xmen list -sd -m "a|b"
Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=Planet/run.sh
Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Bye__b=World/run.sh
Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=Planet/run.sh
Running: screen -dm bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
/Users/robweston/tmp/xmen/hello_world*
root name status created a b
0 hello_world a=Hello__b=World finished 2021-02-08-19-04-45 Hello World
1 hello_world a=Hello__b=Planet finished 2021-02-08-19-04-44 Hello Planet
2 hello_world a=Bye__b=World finished 2021-02-08-19-04-44 Bye World
3 hello_world a=Bye__b=Planet finished 2021-02-08-19-04-44 Bye Planet
Roots relative to: /Users/robweston/tmp/xmen
Features¶
Experiment Visualisation¶
Experiments can be visualised using the xmen list
command. For more info see:
[14]:
%%bash
xmen list --help
usage: xmen list [-h] [-p [PARAM_MATCH [PARAM_MATCH ...]]]
[-n [TYPE_MATCH [TYPE_MATCH ...]]] [-v] [-d] [-g] [-P] [-s]
[-m [DISPLAY_MESSAGES]] [-M [DISPLAY_META]] [-l]
[--load_defaults] [--max_width MAX_WIDTH]
[--max_rows MAX_ROWS] [--csv]
[pattern [pattern ...]]
positional arguments:
pattern List experiments which match pattern.
optional arguments:
-h, --help show this help message and exit
-p [PARAM_MATCH [PARAM_MATCH ...]], --param_match [PARAM_MATCH [PARAM_MATCH ...]]
List only experiments with certain parameter
conditions of the form reg, reg==val or
val1==reg==val2. Here reg is a regex matching a set of
parameters. ==, <, >, !=, >=, <= are all supported
with meaning defined as in experiments. Eg. a.*==cat and
1.0<a.*<=2.0 will return any experiment that has
parameters that match a.* provided that each match
satisfies the condition.
-n [TYPE_MATCH [TYPE_MATCH ...]], --type_match [TYPE_MATCH [TYPE_MATCH ...]]
List only experiments with this type (class).
-v, --verbose Display all information for each experiment
-d, --display_date Display created date for each experiment
-g, --display_git Display git commit for each experiment
-P, --display_purpose
Display purpose for each experiment
-s, --display_status Display status for each experiment
-m [DISPLAY_MESSAGES], --display_messages [DISPLAY_MESSAGES]
Display messages for each experiment
-M [DISPLAY_META], --display_meta [DISPLAY_META]
Display meta information for each experiment. The
regex '^root$|^name$|^mac$|^host$|^user$|^home$' gives
basic meta information logged with every experiment.
Other information is separated into groups including
'network.*', 'gpu.*', 'cpu.*', 'system.*',
'virtual.*', 'swap.*'
-l, --list Display as list and not a table
--load_defaults Infer parameters from defaults.yml and overides
instead of params.yml. Potentially faster but no
messages are available.
--max_width MAX_WIDTH
The maximum width of an individual collumn. If None
then will print for ever
--max_rows MAX_ROWS Display tables with this number of rows.
--csv Display the table as csv.
Git support¶
When running experiments xmen logs lots of useful information automatically. For example, the git information for the repo in which the experiement is defined. This can be found in the params.yml
folder and can be visualised as:
[15]:
%%bash
xmen list -g
/Users/robweston/tmp/xmen/hello_world*
root name local remote commit branch
0 hello_world a=Hello__b=World /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master
1 hello_world a=Hello__b=Planet /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master
2 hello_world a=Bye__b=World /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master
3 hello_world a=Bye__b=Planet /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master
Roots relative to: /Users/robweston/tmp/xmen
Automatic Record Keeping¶
Other meta information is also logged and can be visualised using the meta flag -M
[16]:
%%bash
# basic meta information
xmen list -M
/Users/robweston/tmp/xmen/hello_world*
root name mac host user home
0 hello_world a=Hello__b=World 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
1 hello_world a=Hello__b=Planet 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
2 hello_world a=Bye__b=World 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
3 hello_world a=Bye__b=Planet 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
Roots relative to: /Users/robweston/tmp/xmen
[17]:
%%bash
# system info
xmen list -M "system_.*"
/Users/robweston/tmp/xmen/hello_world*
root name system_system system_node system_release system_version system_machine system_processor
0 hello_world a=Hello__b=World Darwin dhcp55.robots.ox.ac.uk 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20... x86_64 i386
1 hello_world a=Hello__b=Planet Darwin dhcp55.robots.ox.ac.uk 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20... x86_64 i386
2 hello_world a=Bye__b=World Darwin dhcp55.robots.ox.ac.uk 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20... x86_64 i386
3 hello_world a=Bye__b=Planet Darwin dhcp55.robots.ox.ac.uk 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 20... x86_64 i386
Roots relative to: /Users/robweston/tmp/xmen
Meta information is also logged for "cpu_.*", "swap_.*", "virtual_.*", "gpu_.*"
. In addition, a more extensive summary of the environment is logged when each experiment is run and can be found in the meta.yml
file in the experiments folder including network information and global flags. For example:
[18]:
%%bash
cat ~/tmp/xmen/hello_world/a=Hello__b=World/meta.yml
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
cpu:
physical: 4
total: 8
max_freq: 2200.00Mhz
min_freq: 2200.00Mhz
cur_freq: 2200.00Mhz
virtual:
total: 16.00GB
free: 6.15GB
used: 8.97GB
percentage: 61.5%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
disks:
/dev/disk1:
mountpoint: /
fstype: hfs
total: 464.78GB
used: 430.99GB
free: 33.54GB
percent: 92.8%
read: 90.18GB
write: 147.87GB
network:
lo0:
AF_INET6:
address: fe80::1%lo0
netmask: 'ffff:ffff:ffff:ffff::'
broadcast: None
en0:
AF_INET6:
address: fe80::cf3:b64a:220b:17d8%en0
netmask: 'ffff:ffff:ffff:ffff::'
broadcast: None
en6:
AF_INET6:
address: fe80::107d:1836:a969:5191%en6
netmask: 'ffff:ffff:ffff:ffff::'
broadcast: None
en1:
AF_LINK:
address: 6a:00:03:5b:d3:f0
netmask: None
broadcast: None
en2:
AF_LINK:
address: 6a:00:03:5b:d3:f1
netmask: None
broadcast: None
bridge0:
AF_LINK:
address: 6a:00:03:5b:d3:f0
netmask: None
broadcast: None
p2p0:
AF_LINK:
address: 0e:96:cf:db:71:b9
netmask: None
broadcast: None
awdl0:
AF_INET6:
address: fe80::d86a:adff:fe3b:d95%awdl0
netmask: 'ffff:ffff:ffff:ffff::'
broadcast: None
utun0:
AF_INET6:
address: fe80::77eb:8860:4aa1:d43b%utun0
netmask: 'ffff:ffff:ffff:ffff::'
broadcast: None
sent: 21.37GB
received: 37.36GB
environ:
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: cpp
TERM_PROGRAM: iTerm.app
TERM: screen
SHELL: /bin/zsh
CLICOLOR: '1'
TMPDIR: /var/folders/3f/7jm6_wkj6rq0znrt1zgbg7hm0000gn/T/
CONDA_SHLVL: '2'
Apple_PubSub_Socket_Render: /private/tmp/com.apple.launchd.fPKHCGpHV7/Render
CONDA_PROMPT_MODIFIER: '(python3.6) '
TERM_PROGRAM_VERSION: 3.1.5
TERM_SESSION_ID: w0t4p0:7B70FECE-A287-43A3-A009-150E7FF3AA01
ZSH: /Users/robweston/.oh-my-zsh
USER: robweston
CONDA_EXE: /Users/robweston/miniconda3/bin/conda
TERMCAP: "SC|screen|VT 100/ANSI X3.64 virtual terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\\
E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\\
E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\\
E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#24:co#80:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\\
E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\\
E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\\
E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:k0=\\E[10~:k1=\\EOP:k2=\\\
EOQ:\\\n\t:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:\\\n\t:k8=\\E[19~:k9=\\\
E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:\\\n\t:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\\
E[4~:kN=\\E[6~:kP=\\E[5~:\\\n\t:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\\
EOD:"
SSH_AUTH_SOCK: /private/tmp/com.apple.launchd.c8iOoBTn71/Listeners
KERNEL_LAUNCH_TIMEOUT: '40'
__CF_USER_TEXT_ENCODING: 0x1F5:0x0:0x2
JPY_PARENT_PID: '89428'
PAGER: cat
_CE_CONDA: ''
LSCOLORS: Gxfxcxdxbxegedabagacad
CONDA_PREFIX_1: /Users/robweston/miniconda3
PATH: /Users/robweston/.rbenv/shims:/Users/robweston/.gem/ruby/2.7.0/bin:/usr/local/opt/ruby/bin:/Users/robweston/.rbenv/shims:/Users/robweston/miniconda3/envs/python3.6/bin:/Users/robweston/miniconda3/condabin:/Users/robweston/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/git/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
STY: 2844..dhcp55
_: /Users/robweston/tmp/xmen/hello_world/xmen.examples.hello_world.hello_world
CONDA_PREFIX: /Users/robweston/miniconda3/envs/python3.6
PWD: /Users/robweston/tmp/xmen/hello_world
MPLBACKEND: module://ipykernel.pylab.backend_inline
LANG: en_GB.UTF-8
ITERM_PROFILE: Default
XPC_FLAGS: '0x0'
RBENV_SHELL: zsh
_CE_M: ''
CXX: /usr/bin/clang++
XPC_SERVICE_NAME: '0'
SHLVL: '4'
HOME: /Users/robweston
COLORFGBG: 12;8
ITERM_SESSION_ID: w0t4p0:7B70FECE-A287-43A3-A009-150E7FF3AA01
CONDA_PYTHON_EXE: /Users/robweston/miniconda3/bin/experiments
PYTHONPATH: :/Users/robweston/code/experiments-mrg-viewer/build-conda-build-conda-py36/experiments:/Users/robweston/code/experiments-moos/build/lib:/Users/robweston/xmen/experiments:/Users/robweston/projects/rad2sim2rad/experiments
LESS: -R
LOGNAME: robweston
WINDOW: '0'
LC_CTYPE: en_GB.UTF-8
CONDA_DEFAULT_ENV: python3.6
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION: '2'
CC: /usr/bin/clang
DISPLAY: /private/tmp/com.apple.launchd.9Ugqo2pZGO/org.macosforge.xquartz:0
GIT_PAGER: cat
COLORTERM: truecolor
Note Taking¶
Notes can be manually added to an experiment using the xmen note
command:
[19]:
%%bash
xmen note --help
usage: xmen note [-h] [-r DIR] [-d] message
positional arguments:
message Add note to experiment set
optional arguments:
-h, --help show this help message and exit
-r DIR, --root DIR Path to the root experiment folder. If None then the
current work directory will be used
-d, --delete Delete the note corresponding to message.
For example to add a note to the experiments run:
[20]:
%%bash
xmen note "example of adding a note"
Xmens list command provides an easy way to viusalise notes attached to an experiment.
[21]:
%%bash
xmen list -l
0 /Users/robweston/tmp/xmen/hello_world
|- a=Hello__b=World
|- a=Hello__b=Planet
|- a=Bye__b=World
|- a=Bye__b=Planet
purpose:
created: 02-08-21-19:04:29
version:
module: xmen.examples.hello_world
function: hello_world
path: /Users/robweston/xmen/xmen/examples/hello_world.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
notes:
example of adding a note
Conda support¶
To make reproducing experiments as easy as possible, alongside the git version information, the conda environent is also logged as an experiment.yml
file when each experiment is run. To turn this feature on run:
[22]:
%%bash
xmen config --enable_save_conda
Now lets re-run "a=Hello__b=World"
:
[23]:
%%bash
xmen reset "a=Hello__b=World"
xmen run "a=Hello__b=World" bash
/Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/environment.yml
Hello World
Running: bash /Users/robweston/tmp/xmen/hello_world/a=Hello__b=World/run.sh
In this case an environment.yml
file has been generated automatically once the experiment was run
[24]:
%%bash
head -20 ~/tmp/xmen/hello_world/a=Hello__b=World/environment.yml
name: python3.6
channels:
- pytorch
- anaconda
- conda-forge
- defaults
dependencies:
- _tflow_select=2.3.0=mkl
- absl-py=0.8.0=py36_0
- alabaster=0.7.12=py36_0
- appnope=0.1.0=py36hf537a9a_0
- asn1crypto=1.0.1=py36_0
- astor=0.8.0=py36_0
- attrs=19.2.0=py_0
- babel=2.7.0=py_0
- backcall=0.1.0=py36_0
- beautifulsoup4=4.8.0=py36_0
- blas=1.0=mkl
- bleach=3.1.0=py36_0
- bzip2=1.0.8=h1de35cc_0
Note, that while this is a useful feature this does have some additional overhead particularly when working with bloated conda environments. The feature can therefore be disabled as:
[25]:
%%bash
xmen config --disable_save_conda
Interfacing with the slurm job scheduler¶
Another powerful option for running experiments on high performance computing clusters is the slurm job scheduler. To run experiments using the slurm job scheduler first add a header,
[26]:
%%bash
cat ~/xmen/examples/header.txt
#SBATCH --nodes=1
#SBATCH --job-name=single_job
#SBATCH --time=1-00:00:00
#SBATCH --gres=gpu:1
#SBATCH --constraint='gpu_sku:RTX|gpu_sku:V100-LS|gpu_sku:V100|gpu_sku:P100'
#SBATCH --partition=htc-nova,htc
#SBATCH --cpus-per-task=2
#SBATCH --mail-user=robw@robots.ox.ac.uk
#SBATCH --mail-type=ALL
#SBATCH --account=engs-a2i
#SBATCH --signal=SIGUSR1@90
# Author: Rob Weston
# Email: robw@robots.ox.ac.uk
to the global configuration:
[27]:
%%bash
xmen config -H ~/xmen/examples/header.txt
This will be appended to each run script generated by xmen. Experiments can then be submitted using the steps described before, but this time running each job with the sbatch command:
xmen run "*" sbatch
Note the header is used to set default paraeter values. They can still be overidden by passing command line flags to sbatch. For example the --job-name
flag can be set to override the name of the experiment as:
xmen run "*" sbatch --job-name=new_name
Language agnostic experiment configuration¶
Whilst so far we have considered defining and running experiments in experiments xmen
is designed to work with other types of experiments. A general experiment can be defined through a script.sh
which takes a yaml file of parameters as its first argment. For example, a trivial bash experiment script.sh
which takes a yaml file and prints to screen might look something like:
#! /bin/bash
echo "contents of params.yml file ${1}"
cat ${1}
Next to interface with the xmen
command line tool we need to generate a set of default paramaters in a yaml file called defaults.yml
. The script xmen.examples.bash.generate
is provided to do this for you:
[28]:
%%bash
experiments -m xmen.examples.bash.generate ~/tmp/xmen/bash
Now when we look in ~/tmp/xmen/bash
we find a copy of the script.sh
defined above alongside a defaults.yml
file.
[29]:
cd ~/tmp/xmen/bash
/Users/robweston/tmp/xmen/bash
[30]:
%%bash
ls -l ~/tmp/xmen/bash
total 16
-rw-r--r-- 1 robweston staff 109 8 Feb 19:05 defaults.yml
-rw-r--r-- 1 robweston staff 164 8 Feb 19:05 script.sh
The defaults.yml
file defines two parameters a
and b
. The special parameters _version
and _created
can also be included. Here, _created
gives the date the file was created whilst version is a dictionary with a single entry giving the path to the file in which the parameters were originally defined.
[31]:
%%bash
cat ~/tmp/xmen/bash/defaults.yml
_version:
path: /Users/robweston/xmen/xmen/examples/bash/generate.py
_created: 8/2/2021
a: Hello
b: Planet
It is now possible to initialise, link and run experiments using the xmen command line tool in a similar way to before.
[32]:
%%bash
xmen init
None
Defaults from /Users/robweston/tmp/xmen/bash/defaults.yml
Script from /Users/robweston/tmp/xmen/bash/script.sh
Experiment root created at /Users/robweston/tmp/xmen/bash
[33]:
%%bash
xmen link -u "{a: Hello | Bye, b: World| Planet}"
[34]:
%%bash
xmen run "*" bash
#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Bye__b=Planet/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash # The root directory of the experiment
_name: a=Bye__b=Planet # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
path: /Users/robweston/xmen/xmen/examples/bash/generate.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
_meta: # The global configuration for the experiment manager
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
cpu:
physical: 4
total: 8
max_freq: 2200.00Mhz
min_freq: 2200.00Mhz
cur_freq: 2200.00Mhz
virtual:
total: 16.00GB
free: 6.10GB
used: 9.02GB
percentage: 61.9%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
a: Bye
b: Planet
#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Bye__b=World/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash # The root directory of the experiment
_name: a=Bye__b=World # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
path: /Users/robweston/xmen/xmen/examples/bash/generate.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
_meta: # The global configuration for the experiment manager
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
cpu:
physical: 4
total: 8
max_freq: 2200.00Mhz
min_freq: 2200.00Mhz
cur_freq: 2200.00Mhz
virtual:
total: 16.00GB
free: 6.11GB
used: 9.02GB
percentage: 61.8%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
a: Bye
b: World
#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Hello__b=Planet/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash # The root directory of the experiment
_name: a=Hello__b=Planet # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
path: /Users/robweston/xmen/xmen/examples/bash/generate.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
_meta: # The global configuration for the experiment manager
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
cpu:
physical: 4
total: 8
max_freq: 2200.00Mhz
min_freq: 2200.00Mhz
cur_freq: 2200.00Mhz
virtual:
total: 16.00GB
free: 6.10GB
used: 9.02GB
percentage: 61.9%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
a: Hello
b: Planet
#######################################
contents of params.yml file /Users/robweston/tmp/xmen/bash/a=Hello__b=World/params.yml
#######################################
_root: /Users/robweston/tmp/xmen/bash # The root directory of the experiment
_name: a=Hello__b=World # The name of the experiment (under root)
_status: registered # One of ['default' | 'created' | 'running' | 'error' | 'finished']
_created: 2021-02-08-19-05-03 # The date the experiment was created
_purpose: # A description of the experiment purpose
_messages: {} # Messages left by the experiment
_version: # Experiment version information. See `get_version`
path: /Users/robweston/xmen/xmen/examples/bash/generate.py
git:
local: /Users/robweston/xmen
remote: https://github.com/robw4/xmen.git
commit: 7b6e0734d93fdaad5cd8daac012ecef5afb237f5
branch: master
_meta: # The global configuration for the experiment manager
mac: '0x6c96cfdb71b9'
host: dhcp55.robots.ox.ac.uk
user: robweston
home: /Users/robweston
system:
system: Darwin
node: dhcp55.robots.ox.ac.uk
release: 16.7.0
version: 'Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64'
machine: x86_64
processor: i386
cpu:
physical: 4
total: 8
max_freq: 2200.00Mhz
min_freq: 2200.00Mhz
cur_freq: 2200.00Mhz
virtual:
total: 16.00GB
free: 6.10GB
used: 9.02GB
percentage: 61.8%
swap:
total: 2.00GB
free: ' 1.54GB'
used: 470.25MB
percentage: '23.0'
a: Hello
b: World
Running: bash /Users/robweston/tmp/xmen/bash/a=Bye__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/bash/a=Bye__b=World/run.sh
Running: bash /Users/robweston/tmp/xmen/bash/a=Hello__b=Planet/run.sh
Running: bash /Users/robweston/tmp/xmen/bash/a=Hello__b=World/run.sh
Alongside allowing experiments to be initialised and run quickly, xmen
still takes care of record keeping behind the scenes, automatically logging git and system information as before:
[35]:
%%bash
xmen list -dsgM
/Users/robweston/tmp/xmen/bash*
root name local remote commit branch status created mac host user home
0 bash a=Hello__b=World /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master registered 2021-02-08-19-05-03 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
1 bash a=Hello__b=Planet /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master registered 2021-02-08-19-05-03 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
2 bash a=Bye__b=World /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master registered 2021-02-08-19-05-03 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
3 bash a=Bye__b=Planet /Users/robweston/xmen https://github.com/robw4/xmen.git 7b6e0734d93fdaad5cd8daac012ecef5afb237f5 master registered 2021-02-08-19-05-03 0x6c96cfdb71b9 dhcp55.robots.ox.ac.uk robweston /Users/robweston
Roots relative to: /Users/robweston/tmp/xmen
Clean Up¶
[36]:
%%bash
# remove the ~/tmp/xmen folder
rm -rf ~/tmp/xmen
Next Steps¶
For further help running any of the command line tools try
xmen --help
,xmen hello_world --help
More exampels of the experiments can be found in
xmen.examples
Check out the API docs and the cheat sheet
[ ]: