API Reference¶
Classes
ModelOrganizer ([config]) |
A class for organizing a model |
-
class
ModelOrganizer
(config=None)[source]¶ Bases:
object
A class for organizing a model
This class is indended to hold the basic methods for organizing a model and should be adapted to your specific needs. The important steps are
- specify the name of your model in the
name
attribute - use the existing commands, e.g. the
setup()
method orinit()
method if you need to adapt them to your specific needs - when you need your own commands,
- create a new method
- insert the name of the method in the
commands
attribute - If necessary, you can use another name for the command for the
command line parser and specify this one in the
parser_commands
attribute - Create a
_modify_<command>
method (where<command>
should be replaced by the name of the command) which accepts afuncargparse.FuncArgParser
as an argument to update the parameters for the command line parsing - In your new command, call the
app_main()
method which sets the correct experiment, etc.. You can also use theget_app_main_kwargs()
method to filter out the right keyword arguments
- When storing paths in the experiment configuration, we store them
relative the project directory. In that way, we only have to modify the
project configuration if we move our files to another place.
Hence, if you want to use this possibility, you should include the
configuration keys that represent file names in the
paths
attribute.
Path management
abspath
(path[, project, root])Returns the path from the current working directory fix_paths
(\*args, \*\*kwargs)Fix the paths in the given dictionary to get absolute paths paths
list of str. The keys describing paths for the model rel_paths
(\*args, \*\*kwargs)Fix the paths in the given dictionary to get relative paths relpath
(path[, project, root])Returns the relative path from the root directory of the project Main functionality
app_main
([experiment, last, new, verbose, …])The main function for parsing global arguments experiment
The identifier or the experiment that is currently processed is_archived
(experiment[, ignore_missing])Convenience function to determine whether the given experiment has been main
([args])Run the organizer from the command line parse_args
([args])Parse the arguments from the command line (or directly) to the parser projectname
The name of the project that is currently processed start
(\*\*kwargs)Start the commands of this organizer Infrastructural methods
archive
([odir, aname, fmt, projectname, …])Archive one or more experiments or a project instance init
([projectname, description])Initialize a new experiment remove
([projectname, complete, yes, …])Delete an existing experiment and/or projectname setup
(root_dir[, projectname, link])Perform the initial setup for the project unarchive
([experiments, archive, complete, …])Extract archived experiments Attributes
commands
commands that should be accessible from the command line. Each command exp_config
The configuration settings of the current experiment global_config
The global configuration settings logger
The logger of this organizer name
the name of the application/model no_modification
bool(x) -> bool project_config
The configuration settings of the current project of the Configuration methods
configure
([global_config, project_config, …])Configure the project and experiments set_value
(items[, complete, on_projects, …])Set a value in the configuration Info methods
del_value
(keys[, complete, on_projects, …])Delete a value in the configuration get_value
(keys[, exp_path, project_path, …])Get one or more values in the configuration info
([exp_path, project_path, global_path, …])Print information on the experiments Methods
get_app_main_kwargs
(kwargs[, keep])Extract the keyword arguments for the app_main()
methodParser management
get_parser
()Function returning the command line parser for this class parser
The funcargparse.FuncArgParser
to use for controlling the model from the command line.parser_commands
mapping from the name of the parser command to the method name setup_parser
([parser, subparsers])Create the argument parser for this instance Parameters: config (model_organization.config.Config) – The configuration of the organizer -
abspath
(path, project=None, root=None)[source]¶ Returns the path from the current working directory
We only store the paths relative to the root directory of the project. This method fixes those path to be applicable from the working directory
Parameters: - path (str) – The original path as it is stored in the configuration
- project (str) – The project to use. If None, the
projectname
attribute is used - root (str) – If not None, the root directory of the project
Returns: The path as it is accessible from the current working directory
Return type:
-
app_main
(experiment=None, last=False, new=False, verbose=False, verbosity_level=None, no_modification=False, match=False)[source]¶ The main function for parsing global arguments
Parameters: - experiment (str) – The id of the experiment to use
- last (bool) – If True, the last experiment is used
- new (bool) – If True, a new experiment is created
- verbose (bool) – Increase the verbosity level to DEBUG. See also verbosity_level for a more specific determination of the verbosity
- verbosity_level (str or int) – The verbosity level to use. Either one of
'DEBUG', 'INFO', 'WARNING', 'ERROR'
or the corresponding integer (see pythons logging module) - no_modification (bool) – If True/set, no modifications in the configuration files will be done
- match (bool) – If True/set, interprete experiment as a regular expression (regex) und use the matching experiment
-
archive
(odir=None, aname=None, fmt=None, projectname=None, experiments=None, current_project=False, no_append=False, no_project_paths=False, exclude=None, keep_exp=False, rm_project=False, dry_run=False, dereference=False, **kwargs)[source]¶ Archive one or more experiments or a project instance
This method may be used to archive experiments in order to minimize the amount of necessary configuration files
Parameters: - odir (str) – The path where to store the archive
- aname (str) – The name of the archive (minus any format-specific extension). If None, defaults to the projectname
- fmt ({ 'gztar' | 'bztar' | 'tar' | 'zip' }) – The format of the archive. If None, it is tested whether an
archived with the name specified by aname already exists and if
yes, the format is inferred, otherwise
'tar'
is used - projectname (str) – If provided, the entire project is archived
- experiments (str) – If provided, the given experiments are archived. Note that an error is raised if they belong to multiple project instances
- current_project (bool) – If True, projectname is set to the current project
- no_append (bool) – It True and the archive already exists, it is deleted
- no_project_paths (bool) – If True, paths outside the experiment directories are neglected
- exclude (list of str) – Filename patterns to ignore (see
glob.fnmatch.fnmatch()
) - keep_exp (bool) – If True, the experiment directories are not removed and no modification is made in the configuration
- rm_project (bool) – If True, remove all the project files
- dry_run (bool) – If True, set, do not actually make anything
- dereference (bool) – If set, dereference symbolic links. Note: This is automatically set
for
fmt=='zip'
-
commands
= ['setup', 'init', 'set_value', 'get_value', 'del_value', 'info', 'unarchive', 'configure', 'archive', 'remove']¶ commands that should be accessible from the command line. Each command corresponds to one method of this class
-
configure
(global_config=False, project_config=False, ifile=None, forcing=None, serial=False, nprocs=None, update_from=None, **kwargs)[source]¶ Configure the project and experiments
Parameters: - global_config (bool) – If True/set, the configuration are applied globally (already existing and configured experiments are not impacted)
- project_config (bool) – Apply the configuration on the entire project instance instead of only the single experiment (already existing and configured experiments are not impacted)
- ifile (str) – The input file for the project. Must be a netCDF file with population data
- forcing (str) – The input file for the project containing variables with population evolution information. Possible variables in the netCDF file are movement containing the number of people to move and change containing the population change (positive or negative)
- serial (bool) – Do the parameterization always serial (i.e. not in parallel on multiple processors). Does automatically impact global settings
- nprocs (int or 'all') – Maximum number of processes to when making the parameterization in parallel. Does automatically impact global settings and disables serial
- update_from (str) – Path to a yaml configuration file to update the specified configuration with it
- **kwargs – Other keywords for the
app_main()
method
-
del_value
(keys, complete=False, on_projects=False, on_globals=False, projectname=None, base='', dtype=None, **kwargs)[source]¶ Delete a value in the configuration
Parameters: - keys (list of str) – A list of keys to be deleted.
If the key goes some
levels deeper, keys may be separated by a
'.'
(e.g.'namelists.weathergen'
). Hence, to insert a','
, it must be escaped by a preceeding''
. - complete (bool) – If True/set, the information on all experiments are printed
- on_projects (bool) – If set, show information on the projects rather than the experiment
- on_globals (bool) – If set, show the global configuration settings
- projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
- base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time
- keys (list of str) – A list of keys to be deleted.
If the key goes some
levels deeper, keys may be separated by a
-
exp_config
¶ The configuration settings of the current experiment
-
experiment
¶ The identifier or the experiment that is currently processed
-
fix_paths
(*args, **kwargs)[source]¶ Fix the paths in the given dictionary to get absolute paths
Parameters: Returns: The modified d
Return type: Notes
d is modified in place!
-
get_app_main_kwargs
(kwargs, keep=False)[source]¶ Extract the keyword arguments for the
app_main()
methodParameters: - kwargs (dict) – A mapping containing keyword arguments for the
app_main()
method - keep (bool) – If True, the keywords are kept in the kwargs. Otherwise, they are removed
Returns: The keyword arguments for the
app_main()
methodReturn type: Notes
The returned keyword arguments are deleted from kwargs
- kwargs (dict) – A mapping containing keyword arguments for the
-
get_value
(keys, exp_path=False, project_path=False, complete=False, on_projects=False, on_globals=False, projectname=None, no_fix=False, only_keys=False, base='', return_list=False, archives=False, **kwargs)[source]¶ Get one or more values in the configuration
Parameters: - keys (list of str) – A list of keys to get the values of.
If the key goes some
levels deeper, keys may be separated by a
'.'
(e.g.'namelists.weathergen'
). Hence, to insert a','
, it must be escaped by a preceeding''
. - exp_path (bool) – If True/set, print the filename of the experiment configuration
- project_path (bool) – If True/set, print the filename on the project configuration
- complete (bool) – If True/set, the information on all experiments are printed
- on_projects (bool) – If set, show information on the projects rather than the experiment
- on_globals (bool) – If set, show the global configuration settings
- projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
- no_fix (bool) – If set, paths are given relative to the root directory of the project
- only_keys (bool) – If True, only the keys of the given dictionary are printed
- archives (bool) – If True, print the archives and the corresponding experiments for the specified project
- base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time
- return_list (bool) – If True, the list of values corresponding to keys is returned, otherwise they are printed separated by a new line to the standard output
- keys (list of str) – A list of keys to get the values of.
If the key goes some
levels deeper, keys may be separated by a
-
global_config
¶ The global configuration settings
-
info
(exp_path=False, project_path=False, global_path=False, config_path=False, complete=False, no_fix=False, on_projects=False, on_globals=False, projectname=None, return_dict=False, insert_id=True, only_keys=False, archives=False, **kwargs)[source]¶ Print information on the experiments
Parameters: - exp_path (bool) – If True/set, print the filename of the experiment configuration
- project_path (bool) – If True/set, print the filename on the project configuration
- global_path (bool) – If True/set, print the filename on the global configuration
- config_path (bool) – If True/set, print the path to the configuration directory
- complete (bool) – If True/set, the information on all experiments are printed
- no_fix (bool) – If set, paths are given relative to the root directory of the project
- on_projects (bool) – If set, show information on the projects rather than the experiment
- on_globals (bool) – If set, show the global configuration settings
- projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
- return_dict (bool) – If True, the dictionary is returned instead of printed
- insert_id (bool) – If True and neither on_projects, nor on_globals, nor projectname is given, the experiment id is inserted in the dictionary
- only_keys (bool) – If True, only the keys of the given dictionary are printed
- archives (bool) – If True, print the archives and the corresponding experiments for the specified project
-
init
(projectname=None, description=None, **kwargs)[source]¶ Initialize a new experiment
Parameters: - projectname (str) – The name of the project that shall be used. If None, the last one created will be used
- description (str) – A short summary of the experiment
- **kwargs – Keyword arguments passed to the
app_main()
method
Notes
If the experiment is None, a new experiment will be created
-
is_archived
(experiment, ignore_missing=True)[source]¶ Convenience function to determine whether the given experiment has been archived already
Parameters: experiment (str) – The experiment to check Returns: The path to the archive if it has been archived, otherwise None Return type: str or None
-
logger
¶ The logger of this organizer
-
name
= 'model_organizer'¶ the name of the application/model
-
no_modification
= False¶
-
parse_args
(args=None)[source]¶ Parse the arguments from the command line (or directly) to the parser of this organizer
Parameters: args (list) – A list of arguments to parse. If None, the sys.argv
argument is usedReturns: The namespace with the commands as given in **kwargs
and the return values of the corresponding methodReturn type: argparse.Namespace
-
parser
= None¶ The
funcargparse.FuncArgParser
to use for controlling the model from the command line. This attribute is set by thesetup_parser()
method and used by the start method
-
parser_commands
= {}¶ mapping from the name of the parser command to the method name
-
paths
= ['expdir', 'src', 'data', 'input', 'outdata', 'outdir', 'plot_output', 'project_output', 'forcing']¶ list of str. The keys describing paths for the model
-
print_
= None¶
-
project_config
¶ The configuration settings of the current project of the experiment
-
projectname
¶ The name of the project that is currently processed
-
rel_paths
(*args, **kwargs)[source]¶ Fix the paths in the given dictionary to get relative paths
Parameters: Returns: The modified d
Return type: Notes
d is modified in place!
-
relpath
(path, project=None, root=None)[source]¶ Returns the relative path from the root directory of the project
We only store the paths relative to the root directory of the project. This method gives you this path from a path that is accessible from the current working directory
Parameters: - path (str) – The original path accessible from the current working directory
- project (str) – The project to use. If None, the
projectname
attribute is used - root (str) – If not None, the root directory of the project
Returns: The path relative from the root directory
Return type:
-
remove
(projectname=None, complete=False, yes=False, all_projects=False, **kwargs)[source]¶ Delete an existing experiment and/or projectname
Parameters: - projectname (str) – The name for which the data shall be removed. If True, the project will be determined by the experiment. If not None, all experiments for the given project will be removed.
- complete (bool) – If set, delete not only the experiments and config files, but also all the project files
- yes (bool) – If True/set, do not ask for confirmation
- all_projects (bool) – If True/set, all projects are removed
Warning
This will remove the entire folder and all the related informations in the configurations!
-
set_value
(items, complete=False, on_projects=False, on_globals=False, projectname=None, base='', dtype=None, **kwargs)[source]¶ Set a value in the configuration
Parameters: - items (dict) – A dictionary whose keys correspond to the item in the configuration
and whose values are what shall be inserted.
If the key goes some
levels deeper, keys may be separated by a
'.'
(e.g.'namelists.weathergen'
). Hence, to insert a','
, it must be escaped by a preceeding''
. - complete (bool) – If True/set, the information on all experiments are printed
- on_projects (bool) – If set, show information on the projects rather than the experiment
- on_globals (bool) – If set, show the global configuration settings
- projectname (str) – The name of the project that shall be used. If provided and on_projects is not True, the information on all experiments for this project will be shown
- base (str) – A base string that shall be put in front of each key in values to avoid typing it all the time
- dtype (str) – The name of the data type or a data type to cast the value to
- items (dict) – A dictionary whose keys correspond to the item in the configuration
and whose values are what shall be inserted.
If the key goes some
levels deeper, keys may be separated by a
-
setup
(root_dir, projectname=None, link=False, **kwargs)[source]¶ Perform the initial setup for the project
Parameters: - root_dir (str) – The path to the root directory where the experiments, etc. will be stored
- projectname (str) – The name of the project that shall be initialized at root_dir. A
new directory will be created namely
root_dir + '/' + projectname
- link (bool) – If set, the source files are linked to the original ones instead of copied
Other Parameters: ``**kwargs`` – Are passed to the
app_main()
method
-
setup_parser
(parser=None, subparsers=None)[source]¶ Create the argument parser for this instance
This method uses the functions defined in the
commands
attribute to create a command line utility via theFuncArgParser
class. Each command in thecommands
attribute is interpreted as on subparser and setup initially via theFuncArgParser.setup_args()
method. You can modify the parser for each command cmd by including a_modify_cmd
method that accepts the subparser as an argumentParameters: - parser (FuncArgParser) – The parser to use. If None, a new one will be created
- subparsers (argparse._SubParsersAction) – The subparsers to use. If None, the
add_subparser
method from parser will be called
Returns: - FuncArgParser – The created command line parser or the given parser
- argparse._SubParsersAction – The created subparsers action or the given subparsers
- dict – A mapping from command name in the
commands
attribute to the corresponding command line parser
See also
-
start
(**kwargs)[source]¶ Start the commands of this organizer
Parameters: **kwargs – Any keyword from the commands
orparser_commands
attributeReturns: The namespace with the commands as given in **kwargs
and the return values of the corresponding methodReturn type: argparse.Namespace
-
unarchive
(experiments=None, archive=None, complete=False, project_data=False, replace_project_config=False, root=None, projectname=None, fmt=None, force=False, **kwargs)[source]¶ Extract archived experiments
Parameters: - experiments (list of str) – The experiments to extract. If None the current experiment is used
- archive (str) – The path to an archive to extract the experiments from. If None,
we assume that the path to the archive has been stored in the
configuration when using the
archive()
command - complete (bool) – If True, archives are extracted completely, not only the experiment
(implies
project_data = True
) - project_data (bool) – If True, the data for the project is extracted as well
- replace_project_config (bool) – If True and the project does already exist in the configuration, it is updated with what is stored in the archive
- root (str) –
An alternative root directory to use. Otherwise the experiment will be exctracted to
- the root directory specified in the configuration files (if the project exists in it) and replace_project_config is False
- the root directory as stored in the archive
- projectname (str) – The projectname to use. If None, use the one specified in the archive
- fmt ({ 'gztar' | 'bztar' | 'tar' | 'zip' }) – The format of the archive. If None, it is inferred
- force (bool) – If True, force to overwrite the configuration of all experiments from what is stored in archive. Otherwise, the configuration of the experiments in archive are only used if missing in the current configuration
- specify the name of your model in the