model_organization.config module

Classes

Archive Just a dummy string subclass to identify archived experiments
Config(name) Configuration class for one model organizer
ExperimentsConfig(projects[, d, project_map]) The configuration of the experiments
ProjectsConfig(conf_dir[, d]) The project configuration

Functions

get_configdir(name) Return the string representing the configuration directory.
ordered_yaml_dump(data[, stream, Dumper]) Dumps the stream from an OrderedDict.
ordered_yaml_load(stream[, Loader, …]) Loads the stream into an OrderedDict.
safe_dump(d, fname, \*args, \*\*kwargs) Savely dump d to fname using yaml
safe_load(fname) Load the file fname and make sure it can be done in parallel
setup_logging([default_path, default_level, …]) Setup logging configuration
class Archive[source]

Bases: str

Just a dummy string subclass to identify archived experiments

Attributes

project The name of the project inside this archive
time The time when this project has been archived
project = None

The name of the project inside this archive

time = None

The time when this project has been archived

class Config(name)[source]

Bases: object

Configuration class for one model organizer

Attributes

experiments ExperimentConfig. The configuration of the experiments
global_config OrderedDict. The global configuration that applies to all
projects ProjectsConfig. The configuration of the projects

Methods

remove_experiment(experiment)
save() Save the entire configuration files
experiments = {}

ExperimentConfig. The configuration of the experiments

global_config = {}

OrderedDict. The global configuration that applies to all projects

projects = {}

ProjectsConfig. The configuration of the projects

remove_experiment(experiment)[source]
save()[source]

Save the entire configuration files

class ExperimentsConfig(projects, d=None, project_map=None)[source]

Bases: collections.OrderedDict

The configuration of the experiments

This class acts like a collections.OrderedDict but loads the experiment configuration only when you access the specific item (i.e. via d['exp_id'])

Parameters:
  • projects (ProjectConfig) – The project configuration
  • d (dict) – An alternative dictionary to initialize from. If not given, the experiments are loaded on the fly from the exp_files attribute
  • project_map (dict) – A mapping from project to experiments. If not given, it is created when accessing the project_map experiment

Methods

as_ordereddict() Convenience method to convert this object into an OrderedDict
fix_paths(d[, root, project]) Fix the paths in the given dictionary to get absolute paths
items()

Notes
iteritems()

Notes
itervalues()

Notes
load() Load all experiments in this dictionary into memory
rel_paths(d[, root, project]) Fix the paths in the given dictionary to get relative paths
remove(experiment) Remove the configuration of an experiment
save() Save the experiment configuration
values()

Notes

Attributes

exp_file The path to the file containing all experiments in the configuration
exp_files A mapping from experiment to experiment configuration file
paths list of str. The keys describing paths for the model. Note that these
project_map A mapping from project name to experiments
as_ordereddict()[source]

Convenience method to convert this object into an OrderedDict

exp_file

The path to the file containing all experiments in the configuration

exp_files

A mapping from experiment to experiment configuration file

Note that this attribute only contains experiments whose configuration has already dumped to the file!

fix_paths(d, root=None, project=None)[source]

Fix the paths in the given dictionary to get absolute paths

Parameters:
  • d (dict) – One experiment configuration dictionary
  • root (str) – The root path of the project
  • project (str) – The project name
Returns:

The modified d
Return type:

dict

Notes

d is modified in place!

items() → a set-like object providing a view on D's items[source]

Notes

Reimplemented to not load all experiments under python2.7

iteritems()[source]

Notes

Reimplemented to not load all experiments under python2.7

itervalues()[source]

Notes

Reimplemented to not load all experiments under python2.7

load()[source]

Load all experiments in this dictionary into memory

paths = ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']

list of str. The keys describing paths for the model. Note that these keys here are replaced by the keys in the paths attribute of the specific model_organization.ModelOrganizer instance

project_map

A mapping from project name to experiments

rel_paths(d, root=None, project=None)[source]

Fix the paths in the given dictionary to get relative paths

Parameters:
  • d (dict) – One experiment configuration dictionary
  • root (str) – The root path of the project
  • project (str) – The project name
Returns:

The modified d
Return type:

dict

Notes

d is modified in place!

remove(experiment)[source]

Remove the configuration of an experiment

save()[source]

Save the experiment configuration

This method stores the configuration of each of the experiments in a file '<project-dir>/.project/<experiment>.yml', where '<project-dir>' corresponds to the project directory of the specific '<experiment>'. Furthermore it dumps all experiments to the exp_file configuration file.

values() → an object providing a view on D's values[source]

Notes

Reimplemented to not load all experiments under python2.7

class ProjectsConfig(conf_dir, d=None)[source]

Bases: collections.OrderedDict

The project configuration

This class stores the configuration from the projects, where each key corresponds to the name of one project and the value to the corresponding configuration.

Instances of this class are initialized by a file 'projects.yml' in the configuration directory (see the all_projects attribute) that stores a mapping from project name to project directory path. The configuration for each individual project is then loaded from the '<project-dir>/.project/.project.yml' file

Notes

Attributes

all_projects The name of the configuration file
conf_dir The path to the configuration directory
paths list of str. The keys describing paths for the model. Note that these

Methods

fix_paths(d[, root, project]) Fix the paths in the given dictionary to get absolute paths
rel_paths(d[, root, project]) Fix the paths in the given dictionary to get relative paths
save() Save the project configuration

If you move one project has been moved to another directory, make sure to update the 'projects.yml' file (the rest is updated when loading the configuration)

Parameters:
  • conf_dir (str) – The path to the configuration directory containing a file called 'projects.yml'
  • d (dict) – A dictionary to use to setup this configuration instead of loading them from the disk
all_projects

The name of the configuration file

conf_dir = None

The path to the configuration directory

fix_paths(d, root=None, project=None)[source]

Fix the paths in the given dictionary to get absolute paths

Parameters:
  • d (dict) – One experiment configuration dictionary
  • root (str) – The root path of the project
  • project (str) – The project name
Returns:

The modified d
Return type:

dict

Notes

d is modified in place!

paths = ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']

list of str. The keys describing paths for the model. Note that these keys here are replaced by the keys in the paths attribute of the specific model_organization.ModelOrganizer instance

rel_paths(d, root=None, project=None)[source]

Fix the paths in the given dictionary to get relative paths

Parameters:
  • d (dict) – One experiment configuration dictionary
  • root (str) – The root path of the project
  • project (str) – The project name
Returns:

The modified d
Return type:

dict

Notes

d is modified in place!

save()[source]

Save the project configuration

This method dumps the configuration for each project and the project paths (see the all_projects attribute) to the hard drive

get_configdir(name)[source]

Return the string representing the configuration directory.

The directory is chosen as follows:

  1. If the name.upper() + CONFIGDIR environment variable is supplied, choose that.

2a. On Linux, choose $HOME/.config.

2b. On other platforms, choose $HOME/.matplotlib.

  1. If the chosen directory exists, use that as the configuration directory.
  2. A directory: return None.

Notes

This function is taken from the matplotlib [1] module

References

[1]: http://matplotlib.org/api/

ordered_yaml_dump(data, stream=None, Dumper=None, **kwds)[source]

Dumps the stream from an OrderedDict. Taken from

http://stackoverflow.com/questions/5121931/in-python-how-can-you-load-yaml- mappings-as-ordereddicts

ordered_yaml_load(stream, Loader=None, object_pairs_hook=<class 'collections.OrderedDict'>)[source]

Loads the stream into an OrderedDict. Taken from

http://stackoverflow.com/questions/5121931/in-python-how-can-you-load-yaml- mappings-as-ordereddicts

safe_dump(d, fname, *args, **kwargs)[source]

Savely dump d to fname using yaml

This method creates a copy of fname called fname + '~' before saving d to fname using ordered_yaml_dump()

Parameters:
  • d (object) – The object to dump
  • fname (str) – The path where to dump d
Other Parameters:
 

``*args, **kwargs`` – Will be forwarded to the ordered_yaml_dump() function
safe_load(fname)[source]

Load the file fname and make sure it can be done in parallel

Parameters:fname (str) – The path name
setup_logging(default_path=None, default_level=20, env_key=None)[source]

Setup logging configuration

Parameters:
  • default_path (str) – Default path of the yaml logging configuration file. If None, it defaults to the ‘logging.yaml’ file in the config directory
  • default_level (int) – Default: logging.INFO. Default level if default_path does not exist
  • env_key (str) – environment variable specifying a different logging file than default_path (Default: ‘LOG_CFG’)
Returns:

path – Path to the logging configuration file
Return type:

str

Notes

Function taken from http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python