model_organization.config module¶

Classes

`Archive`	Just a dummy string subclass to identify archived experiments
`Config`(name)	Configuration class for one model organizer
`ExperimentsConfig`(projects[, d, project_map])	The configuration of the experiments
`ProjectsConfig`(conf_dir[, d])	The project configuration

Functions

`get_configdir`(name)	Return the string representing the configuration directory.
`ordered_yaml_dump`(data[, stream, Dumper])	Dumps the stream from an OrderedDict.
`ordered_yaml_load`(stream[, Loader, …])	Loads the stream into an OrderedDict.
`safe_dump`(d, fname, args, *kwargs)	Savely dump d to fname using yaml
`safe_load`(fname)	Load the file fname and make sure it can be done in parallel
`setup_logging`([default_path, default_level, …])	Setup logging configuration

class Archive[source]¶

Bases: str

Just a dummy string subclass to identify archived experiments

Attributes

`project`	The name of the project inside this archive
`time`	The time when this project has been archived

project = None¶: The name of the project inside this archive

time = None¶: The time when this project has been archived

class Config(name)[source]¶

Bases: object

Configuration class for one model organizer

Attributes

`experiments`	`ExperimentConfig`. The configuration of the experiments
`global_config`	`OrderedDict`. The global configuration that applies to all
`projects`	`ProjectsConfig`. The configuration of the projects

Methods

`remove_experiment`(experiment)
`save`()	Save the entire configuration files

experiments = {}¶: ExperimentConfig. The configuration of the experiments

global_config = {}¶: OrderedDict. The global configuration that applies to all projects

projects = {}¶: ProjectsConfig. The configuration of the projects

remove_experiment(experiment)[source]¶

save()[source]¶: Save the entire configuration files

class ExperimentsConfig(projects, d=None, project_map=None)[source]¶

Bases: collections.OrderedDict

The configuration of the experiments

This class acts like a collections.OrderedDict but loads the experiment configuration only when you access the specific item (i.e. via d['exp_id'])

Parameters

projects (ProjectConfig) – The project configuration
d (dict) – An alternative dictionary to initialize from. If not given, the experiments are loaded on the fly from the exp_files attribute
project_map (dict) – A mapping from project to experiments. If not given, it is created when accessing the project_map experiment

Methods

`as_ordereddict`()	Convenience method to convert this object into an OrderedDict
`fix_paths`(d[, root, project])	Fix the paths in the given dictionary to get absolute paths
`items`()	Notes
`iteritems`()	Notes
`itervalues`()	Notes
`load`()	Load all experiments in this dictionary into memory
`rel_paths`(d[, root, project])	Fix the paths in the given dictionary to get relative paths
`remove`(experiment)	Remove the configuration of an experiment
`save`()	Save the experiment configuration
`values`()	Notes

Attributes

`exp_file`	The path to the file containing all experiments in the configuration
`exp_files`	A mapping from experiment to experiment configuration file
`paths`	list of str. The keys describing paths for the model. Note that these
`project_map`	A mapping from project name to experiments

as_ordereddict()[source]¶: Convenience method to convert this object into an OrderedDict

exp_file¶: The path to the file containing all experiments in the configuration

exp_files¶

A mapping from experiment to experiment configuration file

Note that this attribute only contains experiments whose configuration has already dumped to the file!

fix_paths(d, root=None, project=None)[source]¶

Fix the paths in the given dictionary to get absolute paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

items() → a set-like object providing a view on D's items[source]¶

Notes

Reimplemented to not load all experiments under python2.7

iteritems()[source]¶

Notes

Reimplemented to not load all experiments under python2.7

itervalues()[source]¶

Notes

Reimplemented to not load all experiments under python2.7

load()[source]¶: Load all experiments in this dictionary into memory

paths = ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']¶: list of str. The keys describing paths for the model. Note that these keys here are replaced by the keys in the paths attribute of the specific model_organization.ModelOrganizer instance

project_map¶: A mapping from project name to experiments

rel_paths(d, root=None, project=None)[source]¶

Fix the paths in the given dictionary to get relative paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

remove(experiment)[source]¶: Remove the configuration of an experiment

save()[source]¶

Save the experiment configuration

This method stores the configuration of each of the experiments in a file '<project-dir>/.project/<experiment>.yml', where '<project-dir>' corresponds to the project directory of the specific '<experiment>'. Furthermore it dumps all experiments to the exp_file configuration file.

values() → an object providing a view on D's values[source]¶

Notes

Reimplemented to not load all experiments under python2.7

class ProjectsConfig(conf_dir, d=None)[source]¶

Bases: collections.OrderedDict

The project configuration

This class stores the configuration from the projects, where each key corresponds to the name of one project and the value to the corresponding configuration.

Instances of this class are initialized by a file 'projects.yml' in the configuration directory (see the all_projects attribute) that stores a mapping from project name to project directory path. The configuration for each individual project is then loaded from the '<project-dir>/.project/.project.yml' file

Notes

Attributes

`all_projects`	The name of the configuration file
`conf_dir`	The path to the configuration directory
`paths`	list of str. The keys describing paths for the model. Note that these

Methods

`fix_paths`(d[, root, project])	Fix the paths in the given dictionary to get absolute paths
`rel_paths`(d[, root, project])	Fix the paths in the given dictionary to get relative paths
`save`()	Save the project configuration

If you move one project has been moved to another directory, make sure to update the 'projects.yml' file (the rest is updated when loading the configuration)

Parameters

conf_dir (str) – The path to the configuration directory containing a file called 'projects.yml'
d (dict) – A dictionary to use to setup this configuration instead of loading them from the disk

all_projects¶: The name of the configuration file

conf_dir = None¶: The path to the configuration directory

fix_paths(d, root=None, project=None)[source]¶

Fix the paths in the given dictionary to get absolute paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

paths = ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']¶: list of str. The keys describing paths for the model. Note that these keys here are replaced by the keys in the paths attribute of the specific model_organization.ModelOrganizer instance

rel_paths(d, root=None, project=None)[source]¶

Fix the paths in the given dictionary to get relative paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

save()[source]¶

Save the project configuration

This method dumps the configuration for each project and the project paths (see the all_projects attribute) to the hard drive

get_configdir(name)[source]¶

Return the string representing the configuration directory.

The directory is chosen as follows:

If the name.upper() + CONFIGDIR environment variable is supplied, choose that.

2a. On Linux, choose $HOME/.config.

2b. On other platforms, choose $HOME/.matplotlib.

If the chosen directory exists, use that as the configuration directory.
A directory: return None.

Notes

This function is taken from the matplotlib [1] module

References

[1]: http://matplotlib.org/api/

ordered_yaml_dump(data, stream=None, Dumper=None, **kwds)[source]¶

Dumps the stream from an OrderedDict. Taken from

http://stackoverflow.com/questions/5121931/in-python-how-can-you-load-yaml- mappings-as-ordereddicts

ordered_yaml_load(stream, Loader=None, object_pairs_hook=<class 'collections.OrderedDict'>)[source]¶

Loads the stream into an OrderedDict. Taken from

http://stackoverflow.com/questions/5121931/in-python-how-can-you-load-yaml- mappings-as-ordereddicts

safe_dump(d, fname, *args, **kwargs)[source]¶

Savely dump d to fname using yaml

This method creates a copy of fname called fname + '~' before saving d to fname using ordered_yaml_dump()

Parameters

d (object) – The object to dump
fname (str) – The path where to dump d

Other Parameters

``*args, **kwargs`` – Will be forwarded to the ordered_yaml_dump() function

safe_load(fname)[source]¶

Load the file fname and make sure it can be done in parallel

Parameters: fname (str) – The path name

setup_logging(default_path=None, default_level=20, env_key=None)[source]¶

Setup logging configuration

Parameters

default_path (str) – Default path of the yaml logging configuration file. If None, it defaults to the ‘logging.yaml’ file in the config directory
default_level (int) – Default: logging.INFO. Default level if default_path does not exist
env_key (str) – environment variable specifying a different logging file than default_path (Default: ‘LOG_CFG’)

Returns

path – Path to the logging configuration file

Return type

str

Notes

Function taken from http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python