API Reference¶

Classes

ModelOrganizer([config])

A class for organizing a model

class ModelOrganizer(config=None)[source]¶

Bases: object

A class for organizing a model

This class is indended to hold the basic methods for organizing a model and should be adapted to your specific needs. The important steps are

specify the name of your model in the name attribute
use the existing commands, e.g. the setup() method or init() method if you need to adapt them to your specific needs
when you need your own commands,
1. create a new method
2. insert the name of the method in the commands attribute
3. If necessary, you can use another name for the command for the command line parser and specify this one in the parser_commands attribute
4. Create a _modify_<command> method (where <command> should be replaced by the name of the command) which accepts a funcargparse.FuncArgParser as an argument to update the parameters for the command line parsing
5. In your new command, call the app_main() method which sets the correct experiment, etc.. You can also use the get_app_main_kwargs() method to filter out the right keyword arguments
When storing paths in the experiment configuration, we store them relative the project directory. In that way, we only have to modify the project configuration if we move our files to another place. Hence, if you want to use this possibility, you should include the configuration keys that represent file names in the paths attribute.

Path management

`abspath`(path[, project, root])	Returns the path from the current working directory
`fix_paths`(args, *kwargs)	Fix the paths in the given dictionary to get absolute paths
`paths`	list of str. The keys describing paths for the model
`rel_paths`(args, *kwargs)	Fix the paths in the given dictionary to get relative paths
`relpath`(path[, project, root])	Returns the relative path from the root directory of the project

Main functionality

`app_main`([experiment, last, new, verbose, …])	The main function for parsing global arguments
`experiment`	The identifier or the experiment that is currently processed
`is_archived`(experiment[, ignore_missing])	Convenience function to determine whether the given experiment has been
`main`([args])	Run the organizer from the command line
`parse_args`([args])	Parse the arguments from the command line (or directly) to the parser
`projectname`	The name of the project that is currently processed
`start`(**kwargs)	Start the commands of this organizer

Infrastructural methods

`archive`([odir, aname, fmt, projectname, …])	Archive one or more experiments or a project instance
`init`([projectname, description])	Initialize a new experiment
`remove`([projectname, complete, yes, …])	Delete an existing experiment and/or projectname
`setup`(root_dir[, projectname, link])	Perform the initial setup for the project
`unarchive`([experiments, archive, complete, …])	Extract archived experiments

Attributes

`commands`	commands that should be accessible from the command line. Each command
`exp_config`	The configuration settings of the current experiment
`global_config`	The global configuration settings
`logger`	The logger of this organizer
`name`	the name of the application/model
`no_modification`	bool(x) -> bool
`project_config`	The configuration settings of the current project of the

Configuration methods

`configure`([global_config, project_config, …])	Configure the project and experiments
`set_value`(items[, complete, on_projects, …])	Set a value in the configuration

Info methods

`del_value`(keys[, complete, on_projects, …])	Delete a value in the configuration
`get_value`(keys[, exp_path, project_path, …])	Get one or more values in the configuration
`info`([exp_path, project_path, global_path, …])	Print information on the experiments

Methods

get_app_main_kwargs(kwargs[, keep])

Extract the keyword arguments for the app_main() method

Parser management

`get_parser`()	Function returning the command line parser for this class
`parser`	The `funcargparse.FuncArgParser` to use for controlling the model from the command line.
`parser_commands`	mapping from the name of the parser command to the method name
`setup_parser`([parser, subparsers])	Create the argument parser for this instance

Parameters: config (model_organization.config.Config) – The configuration of the organizer

abspath(path, project=None, root=None)[source]¶

Returns the path from the current working directory

We only store the paths relative to the root directory of the project. This method fixes those path to be applicable from the working directory

Parameters

path (str) – The original path as it is stored in the configuration
project (str) – The project to use. If None, the projectname attribute is used
root (str) – If not None, the root directory of the project

Returns

The path as it is accessible from the current working directory

Return type

str

app_main(experiment=None, last=False, new=False, verbose=False, verbosity_level=None, no_modification=False, match=False)[source]¶

The main function for parsing global arguments

Parameters

experiment (str) – The id of the experiment to use
last (bool) – If True, the last experiment is used
new (bool) – If True, a new experiment is created
verbose (bool) – Increase the verbosity level to DEBUG. See also verbosity_level for a more specific determination of the verbosity
verbosity_level (str or int) – The verbosity level to use. Either one of 'DEBUG', 'INFO', 'WARNING', 'ERROR' or the corresponding integer (see pythons logging module)
no_modification (bool) – If True/set, no modifications in the configuration files will be done
match (bool) – If True/set, interprete experiment as a regular expression (regex) und use the matching experiment

archive(odir=None, aname=None, fmt=None, projectname=None, experiments=None, current_project=False, no_append=False, no_project_paths=False, exclude=None, keep_exp=False, rm_project=False, dry_run=False, dereference=False, **kwargs)[source]¶

Archive one or more experiments or a project instance

This method may be used to archive experiments in order to minimize the amount of necessary configuration files

Parameters

odir (str) – The path where to store the archive
aname (str) – The name of the archive (minus any format-specific extension). If None, defaults to the projectname
fmt ({ 'gztar' | 'bztar' | 'tar' | 'zip' }) – The format of the archive. If None, it is tested whether an archived with the name specified by aname already exists and if yes, the format is inferred, otherwise 'tar' is used
projectname (str) – If provided, the entire project is archived
experiments (str) – If provided, the given experiments are archived. Note that an error is raised if they belong to multiple project instances
current_project (bool) – If True, projectname is set to the current project
no_append (bool) – It True and the archive already exists, it is deleted
no_project_paths (bool) – If True, paths outside the experiment directories are neglected
exclude (list of str) – Filename patterns to ignore (see glob.fnmatch.fnmatch())
keep_exp (bool) – If True, the experiment directories are not removed and no modification is made in the configuration
rm_project (bool) – If True, remove all the project files
dry_run (bool) – If True, set, do not actually make anything
dereference (bool) – If set, dereference symbolic links. Note: This is automatically set for fmt=='zip'

commands = ['setup', 'init', 'set_value', 'get_value', 'del_value', 'info', 'unarchive', 'configure', 'archive', 'remove']¶: commands that should be accessible from the command line. Each command corresponds to one method of this class

configure(global_config=False, project_config=False, ifile=None, forcing=None, serial=False, nprocs=None, update_from=None, **kwargs)[source]¶

Configure the project and experiments

Parameters

global_config (bool) – If True/set, the configuration are applied globally (already existing and configured experiments are not impacted)
project_config (bool) – Apply the configuration on the entire project instance instead of only the single experiment (already existing and configured experiments are not impacted)
ifile (str) – The input file for the project. Must be a netCDF file with population data
forcing (str) – The input file for the project containing variables with population evolution information. Possible variables in the netCDF file are movement containing the number of people to move and change containing the population change (positive or negative)
serial (bool) – Do the parameterization always serial (i.e. not in parallel on multiple processors). Does automatically impact global settings
nprocs (int or 'all') – Maximum number of processes to when making the parameterization in parallel. Does automatically impact global settings and disables serial
update_from (str) – Path to a yaml configuration file to update the specified configuration with it
**kwargs – Other keywords for the app_main() method

del_value(keys, complete=False, on_projects=False, on_globals=False, projectname=None, base='', dtype=None, **kwargs)[source]¶

Delete a value in the configuration

Parameters

keys (list of str) – A list of keys to be deleted. If the key goes some levels deeper, keys may be separated by a '.' (e.g. 'namelists.weathergen'). Hence, to insert a ',', it must be escaped by a preceeding ''.
complete (bool) – If True/set, the information on all experiments are printed
on_projects (bool) – If set, show information on the projects rather than the experiment
on_globals (bool) – If set, show the global configuration settings
projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time

exp_config¶: The configuration settings of the current experiment

experiment¶: The identifier or the experiment that is currently processed

fix_paths(*args, **kwargs)[source]¶

Fix the paths in the given dictionary to get absolute paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

get_app_main_kwargs(kwargs, keep=False)[source]¶

Extract the keyword arguments for the app_main() method

Parameters

kwargs (dict) – A mapping containing keyword arguments for the app_main() method
keep (bool) – If True, the keywords are kept in the kwargs. Otherwise, they are removed

Returns

The keyword arguments for the app_main() method

Return type

dict

Notes

The returned keyword arguments are deleted from kwargs

classmethod get_parser()[source]¶: Function returning the command line parser for this class

get_value(keys, exp_path=False, project_path=False, complete=False, on_projects=False, on_globals=False, projectname=None, no_fix=False, only_keys=False, base='', return_list=False, archives=False, **kwargs)[source]¶

Get one or more values in the configuration

Parameters

keys (list of str) – A list of keys to get the values of. If the key goes some levels deeper, keys may be separated by a '.' (e.g. 'namelists.weathergen'). Hence, to insert a ',', it must be escaped by a preceeding ''.
exp_path (bool) – If True/set, print the filename of the experiment configuration
project_path (bool) – If True/set, print the filename on the project configuration
complete (bool) – If True/set, the information on all experiments are printed
on_projects (bool) – If set, show information on the projects rather than the experiment
on_globals (bool) – If set, show the global configuration settings
projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
no_fix (bool) – If set, paths are given relative to the root directory of the project
only_keys (bool) – If True, only the keys of the given dictionary are printed
archives (bool) – If True, print the archives and the corresponding experiments for the specified project
base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time
return_list (bool) – If True, the list of values corresponding to keys is returned, otherwise they are printed separated by a new line to the standard output

global_config¶: The global configuration settings

info(exp_path=False, project_path=False, global_path=False, config_path=False, complete=False, no_fix=False, on_projects=False, on_globals=False, projectname=None, return_dict=False, insert_id=True, only_keys=False, archives=False, **kwargs)[source]¶

Print information on the experiments

Parameters

exp_path (bool) – If True/set, print the filename of the experiment configuration
project_path (bool) – If True/set, print the filename on the project configuration
global_path (bool) – If True/set, print the filename on the global configuration
config_path (bool) – If True/set, print the path to the configuration directory
complete (bool) – If True/set, the information on all experiments are printed
no_fix (bool) – If set, paths are given relative to the root directory of the project
on_projects (bool) – If set, show information on the projects rather than the experiment
on_globals (bool) – If set, show the global configuration settings
projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
return_dict (bool) – If True, the dictionary is returned instead of printed
insert_id (bool) – If True and neither on_projects, nor on_globals, nor projectname is given, the experiment id is inserted in the dictionary
only_keys (bool) – If True, only the keys of the given dictionary are printed
archives (bool) – If True, print the archives and the corresponding experiments for the specified project

init(projectname=None, description=None, **kwargs)[source]¶

Initialize a new experiment

Parameters

projectname (str) – The name of the project that shall be used. If None, the last one created will be used
description (str) – A short summary of the experiment
**kwargs – Keyword arguments passed to the app_main() method

Notes

If the experiment is None, a new experiment will be created

is_archived(experiment, ignore_missing=True)[source]¶

Convenience function to determine whether the given experiment has been archived already

Parameters: experiment (str) – The experiment to check
Returns: The path to the archive if it has been archived, otherwise None
Return type: str or None

logger¶: The logger of this organizer

classmethod main(args=None)[source]¶

Run the organizer from the command line

Parameters

name (str) – The name of the program
args (list) – The arguments that are parsed to the argument parser

name = 'model_organizer'¶: the name of the application/model

no_modification = False¶

parse_args(args=None)[source]¶

Parse the arguments from the command line (or directly) to the parser of this organizer

Parameters: args (list) – A list of arguments to parse. If None, the sys.argv argument is used
Returns: The namespace with the commands as given in **kwargs and the return values of the corresponding method
Return type: argparse.Namespace

parser = None¶: The funcargparse.FuncArgParser to use for controlling the model from the command line. This attribute is set by the setup_parser() method and used by the start method

parser_commands = {}¶: mapping from the name of the parser command to the method name

paths = ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']¶: list of str. The keys describing paths for the model

print_ = None¶

project_config¶: The configuration settings of the current project of the experiment

projectname¶: The name of the project that is currently processed

rel_paths(*args, **kwargs)[source]¶

Fix the paths in the given dictionary to get relative paths

Parameters

d (dict) – One experiment configuration dictionary
root (str) – The root path of the project
project (str) – The project name

Returns

The modified d

Return type

dict

Notes

d is modified in place!

relpath(path, project=None, root=None)[source]¶

Returns the relative path from the root directory of the project

We only store the paths relative to the root directory of the project. This method gives you this path from a path that is accessible from the current working directory

Parameters

path (str) – The original path accessible from the current working directory
project (str) – The project to use. If None, the projectname attribute is used
root (str) – If not None, the root directory of the project

Returns

The path relative from the root directory

Return type

str

remove(projectname=None, complete=False, yes=False, all_projects=False, **kwargs)[source]¶

Delete an existing experiment and/or projectname

Parameters

projectname (str) – The name for which the data shall be removed. If True, the project will be determined by the experiment. If not None, all experiments for the given project will be removed.
complete (bool) – If set, delete not only the experiments and config files, but also all the project files
yes (bool) – If True/set, do not ask for confirmation
all_projects (bool) – If True/set, all projects are removed

Warning

This will remove the entire folder and all the related informations in the configurations!

set_value(items, complete=False, on_projects=False, on_globals=False, projectname=None, base='', dtype=None, **kwargs)[source]¶

Set a value in the configuration

Parameters

items (dict) – A dictionary whose keys correspond to the item in the configuration and whose values are what shall be inserted. If the key goes some levels deeper, keys may be separated by a '.' (e.g. 'namelists.weathergen'). Hence, to insert a ',', it must be escaped by a preceeding ''.
complete (bool) – If True/set, the information on all experiments are printed
on_projects (bool) – If set, show information on the projects rather than the experiment
on_globals (bool) – If set, show the global configuration settings
projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time
dtype (str) – The name of the data type or a data type to cast the value to

setup(root_dir, projectname=None, link=False, **kwargs)[source]¶

Perform the initial setup for the project

Parameters

root_dir (str) – The path to the root directory where the experiments, etc. will be stored
projectname (str) – The name of the project that shall be initialized at root_dir. A new directory will be created namely root_dir + '/' + projectname
link (bool) – If set, the source files are linked to the original ones instead of copied

Other Parameters

``**kwargs`` – Are passed to the app_main() method

setup_parser(parser=None, subparsers=None)[source]¶

Create the argument parser for this instance

This method uses the functions defined in the commands attribute to create a command line utility via the FuncArgParser class. Each command in the commands attribute is interpreted as on subparser and setup initially via the FuncArgParser.setup_args() method. You can modify the parser for each command cmd by including a _modify_cmd method that accepts the subparser as an argument

Parameters

parser (FuncArgParser) – The parser to use. If None, a new one will be created
subparsers (argparse._SubParsersAction) – The subparsers to use. If None, the add_subparser method from parser will be called

Returns

FuncArgParser – The created command line parser or the given parser
argparse._SubParsersAction – The created subparsers action or the given subparsers
dict – A mapping from command name in the commands attribute to the corresponding command line parser

API Reference¶

Submodules¶