Config management with `xoa.cfgm`

Introduction

The xoa.cfgm module simplifies and extends the functionality provided by the excellent configobj configuration file reader and validator package. It makes global usage a little easier and adds support for plurals forms and for exporting the specifications to commandline options and rst declarations.

Loading and validating

Let’s define the configuration specifications:

In [1]: cfgspecs_lines = ["[logger] # logging system", "level=option(debug,info,error,default=info) # logging level", "name=string(default=root) # logger name", "[domain]", "xlim=floats(n=2, default=list(-20,0)) # min and max longitudes", "ylim=floats(n=2,default=list(40,50))"]

In [2]: print("\n".join(cfgspecs_lines))
[logger] # logging system
level=option(debug,info,error,default=info) # logging level
name=string(default=root) # logger name
[domain]
xlim=floats(n=2, default=list(-20,0)) # min and max longitudes
ylim=floats(n=2,default=list(40,50))

Note

Note that the built-in float validation function has been used here in its plurals form.

Let’s set the user configuration:

In [3]: cfg_lines = ["[logger]", "name=xoa", "[domain]", "xminmax=-10,0"]

In [4]: print("\n".join(cfg_lines))
[logger]
name=xoa
[domain]
xminmax=-10,0

Now, initialize the config manager:

In [5]: from xoa import cfgm

In [6]: CFGM = cfgm.ConfigManager(cfgspecs_lines)

Note

To configure a library, the cfgspecs_lines argument is typically the name of a static file placed in the same directory as the module, which will initialize an instance of the ConfigManager class.

And finally, load and validate the user configuration:

In [7]: cfg = CFGM.load(cfg_lines)

In [8]: from pprint import pprint

In [9]: pprint(cfg)
ConfigObj({'logger': {'name': 'xoa', 'level': 'info'}, 'domain': {'xminmax': ['-10', '0'], 'xlim': [-20.0, 0.0], 'ylim': [40.0, 50.0]}})

Get the default values:

In [10]: pprint(CFGM.defaults.dict())
{'domain': {'xlim': [-20.0, 0.0], 'ylim': [40.0, 50.0]},
 'logger': {'level': 'info', 'name': 'root'}}

Extending the validation capabilities

One can add user validation fonctions to the default ones.

Use the print_validation_fonctions() function to print the list of validation functions. In the following example, only those matching *datetime* are printed:

In [11]: cfgm.print_validation_functions("*time*") # use a pattern to restrict search
cdtime(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtime_list(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtime_list_list(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtime_lists(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtimes(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtimes_list(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
cdtimess(value, min=None, max=None, default=None)
    Validation function of a date (compatible with :func:`cdtime.s2c`)
datetime(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetime64(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64_list(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64_list_list(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64_lists(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64s(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64s_list(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime64ss(value, default=None)
    Validation function to create a :class:`numpy.datetime64`
datetime_list(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetime_list_list(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetime_lists(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetimes(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetimes_list(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
datetimess(value, default=None)
    Validation function to magically create a :class:`datetime.datetime`
pydatetime(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetime_list(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetime_list_list(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetime_lists(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetimes(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetimes_list(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
pydatetimess(value, default=None, fmt='%Y-%m-%dT%H:%M:%S')
    Parse value as a :class:`datetime.datetime` object
timestamp(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestamp_list(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestamp_list_list(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestamp_lists(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestamps(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestamps_list(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`
timestampss(value, default=None)
    Validation function of date as parsable by :func:`pandas.Timestamp`

To define a new validation function, use the register_validation_fonctions() function. Here we define and register a validation function that converts an entry to an angle in degrees within [0, 360):

# Define
In [12]: from validate import VdtTypeError

In [13]: def is_angle(value, radians=False):
   ....:     """Validate an angle with optional convertion to radians"""
   ....:     try:
   ....:         value = float(value) % 360.
   ....:     except Exception:
   ....:         raise VdtTypeError("Invalid angle")
   ....:     value = float(value)
   ....:     if radians:
   ....:         value = np.radians(value)
   ....:     return value
   ....: 

# Register
In [14]: cfgm.register_validation_functions(angle=is_angle)

# Check that it is registered
In [15]: cfgm.print_validation_functions("angle")
angle(value, radians=False)
    Validate an angle with optional convertion to radians

# Check that it works
In [16]: validator = cfgm.get_validator()

In [17]: print(validator.check("angle(radians=True)", 180+360))
3.141592653589793

Using the `argparse` capabilities

The ConfigManager has the capability to generate commandline options from the configuration specifications thanks to the arg_parse method. The goal is to add more control to the configuration for the user that use it from an executable script that parses the commandline:

Default values are internally defined in the config specifications.
The user optionally alter these value with its configuration file.
The user optionally alter its own configuration using the commandline options, which are set to None by default.

In other word, the commandline arguments take precedence over the user configuration, which takes precedence over the default internal configuration.

Taking advantage of the example above, we create a parser and add arguments that reflect the configuration specifications:

# Write the user config file
In [18]: with open("config.cfg", "w") as f:
   ....:     f.write("\n".join(cfg_lines))
   ....: 

# Define our commandline options
In [19]: argv = ["--logger-level", "error", "myfile.nc"]

# Init the parser
In [20]: from argparse import ArgumentParser

In [21]: parser = ArgumentParser(description="My script")

In [22]: parser.add_argument("ncfile", help="netcdf file")
Out[22]: _StoreAction(option_strings=[], dest='ncfile', nargs=None, const=None, default=None, type=None, choices=None, required=True, help='netcdf file', metavar=None)

# Add options and parse
In [23]: cfg, args = CFGM.arg_parse(
   ....:     parser=parser, getargs=True, args=argv, cfgfile="config.cfg")
   ....: 

# Args
In [24]: print(args)
Namespace(ncfile='myfile.nc', cfgfile='config.cfg', logger_level='error', logger_name=None, domain_xlim=None, domain_ylim=None)

# Cfg
In [25]: pprint(cfg)
ConfigObj({'logger': {'level': 'error', 'name': 'xoa'}, 'domain': {'xlim': [-20.0, 0.0], 'ylim': [40.0, 50.0], 'xminmax': ['-10', '0']}})

# See the help like with the -h option
In [26]: parser.print_help()
usage: __main__.py [-h] [--long-help] [--short-help] [--cfgfile CFGFILE]
                   [--logger-level LOGGER_LEVEL] [--logger-name LOGGER_NAME]
                   [--domain-xlim DOMAIN_XLIM] [--domain-ylim DOMAIN_YLIM]
                   ncfile

My script

positional arguments:
  ncfile                netcdf file

options:
  -h, --help            show a reduced help and exit
  --long-help           show an extended help and exit
  --short-help          show a very reduced help and exit
  --cfgfile CFGFILE     user configuration file that overrides defauts

logger:
  logging system

  --logger-level LOGGER_LEVEL
                        logging level. [default: info]
  --logger-name LOGGER_NAME
                        logger name. [default: root]

domain:
  --domain-xlim DOMAIN_XLIM
                        min and max longitudes. [default: -20.0,0.0]
  --domain-ylim DOMAIN_YLIM
                        Undocumented [default: 40.0,50.0]

# Long help
In [27]: parser.parse_args(["--long-help"])
usage: __main__.py [-h] [--long-help] [--short-help] [--cfgfile CFGFILE]
                   [--logger-level LOGGER_LEVEL] [--logger-name LOGGER_NAME]
                   [--domain-xlim DOMAIN_XLIM] [--domain-ylim DOMAIN_YLIM]
                   ncfile

My script

positional arguments:
  ncfile                netcdf file

options:
  -h, --help            show a reduced help and exit
  --long-help           show an extended help and exit
  --short-help          show a very reduced help and exit
  --cfgfile CFGFILE     user configuration file that overrides defauts

logger:
  logging system

  --logger-level LOGGER_LEVEL
                        logging level. [default: info]
  --logger-name LOGGER_NAME
                        logger name. [default: root]

domain:
  --domain-xlim DOMAIN_XLIM
                        min and max longitudes. [default: -20.0,0.0]
  --domain-ylim DOMAIN_YLIM
                        Undocumented [default: 40.0,50.0]
An exception has occurred, use %tb to see the full traceback.

SystemExit: 0


# Very short help
In [28]: parser.parse_args(["--short-help"])
usage: __main__.py [-h] [--long-help] [--short-help] [--cfgfile CFGFILE]
                   [other-options]
                   ncfile

My script

positional arguments:
  ncfile  netcdf file
An exception has occurred, use %tb to see the full traceback.

SystemExit: 0

The arg_parse method makes also available the --short-help and --long--help in addition to the --help option.

Note

The previous example can be compacted using the cfgargparse() function:

cfg, args = cfgargparse(cfgspecsfile, parser=parser, getargs=True)

Converting to `.rst` format

The config manager instance is exportable to rst declaration for documentation purpose.

In [29]: print(CFGM.get_rst())
.. confsec:: [logger]

    Logging system

    .. confopt:: [logger] level
    
        | default: ``info``
        | possible choices: ``debug, info, error``
        
        Logging level
    
    .. confopt:: [logger] name
    
        | default: ``root``
        | type: :func:`string <configobj.validate.is_string>`
        
        Logger name
    
.. confsec:: [domain]

    .. confopt:: [domain] xlim
    
        | default: ``-20, 0``
        | type: :func:`floats <configobj.validate.is_float>`
        | n: ``2``
        
        Min and max longitudes
    
    .. confopt:: [domain] ylim
    
        | default: ``40, 50``
        | type: :func:`floats <configobj.validate.is_float>`
        | n: ``2``

The result is the following.

[logger]

Logging system

[logger] level: default: info

type: option

args: debug, info, error

Logging level

[logger] name: default: root

type: string

Logger name

[domain]

[domain] xlim: default: -20, 0

type: floats

n: 2

Min and max longitudes

[domain] ylim: default: 40, 50

type: floats

n: 2

These above sphinx declarations need two objet types to be declared:

app.add_object_type('confopt', 'confopt',
                    objname='configuration option',
                    indextemplate='pair: %s; configuration option')
app.add_object_type('confsec', 'confsec',
                    objname='configuration section',
                    indextemplate='pair: %s; configuration section')

The name of these types are parameters of the cfg2rst() function (and xoa.cfgm.ConfigManager.get_rst() method).

Fortunately, the xoa.cfgm comes also as a Sphinx extension:

Add "xoa.cfgm" to the list of Sphinx extensions that are declared in the conf.py file.
Declare the following Sphinx config variables in the conf.py:

cfgm_get_cfgm_func

Function that returns a xoa.cfgm.ConfigManager instance.

cfgm_rst_file

Name of the outfile file in which rst declarations are written.

Config management with xoa.cfgm

Introduction

Loading and validating

Extending the validation capabilities

Using the argparse capabilities

Converting to .rst format

Config management with `xoa.cfgm`

Using the `argparse` capabilities

Converting to `.rst` format