7. Testing

You can test pconfigs using pytest or python. Consider the following project.

7.1. Setup a project.

my_project/
    source/
        __init__.py
        modules/                   # Modules define pconfig types, classes, and defaults.
            __init__.py
            trainer.py
            model.py

        pconfigs/                  # Config directories define system configurations.
            __pconfigs__.py        # A sentinel file marks a directory of pconfigs.
            experiment_1.py        # Files only contain config instances.
            experiment_2.py

            subdir/
                __pconfigs__.py    # Optional (see below).
                sub_expr_1.py
                sub_expr_2.py

    docs/                          # Etc.

Config files like pconfigs/experiment_1.py specify config instances. For example,

from source.modules.trainer import TrainerConfig

config = TrainerConfig(
    message="Hello, World!",
) 

And this config might run your first experiment,

$ python -m pconfigs.run source.pconfigs.experiment_1.config

7.2. Test pconfig files.

Config files usually contain computed @pproperty fields that should not break as the configs are developed. You can test configs as follows.

~/my_project$ ptest

which is installed with the pconfigs package. This command calls pytest with arguments that minimize the output. Alternativelly, pytest can be run manually with

~/my_project$ pytest --pyargs pconfigs.test

Both invocations will run the same tests: all .py files in directories that contain a __pconfigs__.py sentinel file will be evaluated. This validates that all @pproperties run without error. The search for sentinel files starts in the current directory and covers all of its subdirectories except hidden ones (names starting with ., such as .git or .venv).

7.3. Test all subdirectories.

You can test all configs in a subdirectory tree as follows.

Modify __pconfigs__.py:

from pconfigs import TestSubdirs

By importing the TestSubdirs sentinel, pconfigs.test will test all .py config files in all subdirectories except hidden ones, regardless of whether the subdirectory contains a __pconfigs__.py file.

7.4. Test specific pconfigs.

The pconfig testing system is designed to be low maintenance: as new configs are added, or config filenames are changed, running pconfigs.test will continue to test all configs that are marked with the __pconfigs__.py sentinel file; no development work is required to update tests. Nonetheless, if you find that it is necessary to specify that only specific configs in a directory are tested, you can do so as follows.

To test specific configs, modify __pconfigs__.py:

from pconfigs import TestSubdirs, TestManually

TestManually(
    "experiment_1",                    # List only the modules that you want to test.
)

The above file will test only experiment_1.py in the current directory. Subdirectory configs sub_expr_1.py and sub_expr_2.py will also be tested because TestSubdirs is imported.