H - A Time-Dependent Experiment

Here, we run an experiment that access, containerize and execute multiple time-dependent models, and then proceeds to evaluate the forecasts once they are created.

TL; DR

In a terminal, navigate to floatcsep/tutorials/case_h and type:

$ floatcsep run config.yml

After the calculation is complete, the results will be summarized in results/report.md.

Experiment Components 

The experiment input files are:

case_h
    ├── catalog.csv
    ├── config.yml
    ├── tests.yml
    └── models.yml

The models.yml contains the instructions to clone and build the source codes from software repositories (e.g., gitlab, Github), and how to interface them with floatCSEP. Once downloaded and built, the experiment structure should look like:

case_h
    ├── models
        ├── etas
            └── ...     (ETAS source code)
        ├── pymock_poisson
            └── ...     (Poisson-PyMock source code)
        └── pymock_nb
            └── ...     (Negative-Binomial-PyMock source code)
    ├── catalog.csv
    ├── config.yml
    ├── tests.yml
    ├── custom_report.py
    └── models.yml

Configuration 

Models 

As in Tutorial G, each Model requires to build and execute a source code to generate forecasts. The instructions for each model are located within models.yml, which we further explain here:

Note

The models.yml will define how to interface floatCSEP to each Model, implying that a Model should be developed, or adapted to ensure the interface requirements specified below.

The repository URL of each model and their specific versions (e.g., commit hash, tag, release) are specified as:

tutorials/case_h/models.yml

- etas:
    giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/vetas.git
    repo_hash: v3.2
- Poisson Mock:
    giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/pymock.git
    repo_hash: v0.1
- Negbinom Mock:
    giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/pymock.git
    repo_hash: v0.1

A path needs to be indicated for each model, to both download the repository contents therein and from where the source code will be executed.
tutorials/case_h/models.yml
1- etas: 2 giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/vetas.git 3 repo_hash: v3.2 4 path: models/etas
Important

The implies that the inputs (catalog and argument file) should be read by the model from a {path}/input folder, and the forecast outputs stored into a folder {path}/forecasts.

There is some flexibility to interface floatCSEP with a model. For instance, a different filepath can be set for the argument file:
tutorials/case_h/models.yml
5 args_file: input/args.json
Note

floatCSEP can read .txt, .json and .yml format-styled argument files. In all cases, the minimum required arguments, should be formatted as:
#.txt start_date = {DATESTRING} end_date = {DATESTRING}
#.yml start_date: {DATESTRING} end_date: {DATESTRING}
//.json "start_date": "{DATESTRING}", "end_date": "{DATESTRING}"
floatcsep will dynamically modify the start_date and start_date for each time window run, and any extra arguments will just be added for all time-windows.

The func entry indicates how the models are invoked from a shell terminal.
tutorials/case_h/models.yml
- etas: func: etas-run - Poisson Mock: func: pymock - Negbinom Mock: func: pymock
Important

Please refer to Tutorial G for example of how to set up func for the model and interface it to floatCSEP.
A prefix for the resulting forecast filepaths can be specified beforehand for each model. Without specifying this, the default is {model_name} (e.g., etas, Poisson Mock, Negbinom Mock).
tutorials/case_h/models.yml
- Negbinom Mock: prefix: pymock
The experiment will read the forecasts as:
{model_path}/{forecasts}/{prefix}_{start}_{end}.csv
where start and end follow either the %Y-%m-%dT%H:%M:%S.%f - ISO861 FORMAT, or the short date version %Y-%m-%d if the windows are set by midnight.
Additional function arguments can be passed to the model with the entry func_kwargs. We perhaps noted that both Poisson Mock and Negbinom Mock use the same source code. With func_kwargs a different subclass can be defined for the same source code (in this case, a Negative-Binomial number distribution instead of Poisson).
tutorials/case_h/models.yml
- Poisson Mock: func_kwargs: n_sims: 100 mag_min: 3.5 seed: 23 - Negbinom Mock: func_kwargs: n_sims: 100 mag_min: 3.5 seed: 23 distribution: negbinom

Time 

The configuration is identical to time-independent models, with the exception that now a horizon can be defined instead of intervals, which is the forecast time-window length. The experiment’s class should now be explicited as exp_class: td
tutorials/case_h/config.yml
time_config:
  start_date: 2016-8-25T00:00:00
  end_date: 2016-8-26T00:00:00
  horizon: 1days
  exp_class: td

Catalog 

The catalog was obtained previous to the experiment using query_bsi, but it was filtered from 2006 onwards, so it has enough data for the model calibration.

Tests 

With time-dependent models, now catalog evaluations found in csep.core.catalog_evaluations can be used.

tutorials/case_h/tests.yml

- Catalog_N-test:
    func: catalog_evaluations.number_test
    func_kwargs:
        verbose: False
    plot_func:
        - plot_number_test:
            plot_args:
                title: Number consistency test
                bins: 30
        - plot_consistency_test:
            plot_args:
                title: Number consistency test

Note

It is possible to assign two plotting functions to a test, whose plot_args and plot_kwargs can be placed indented beneath

Custom Post-Process 

A custom reporting function can be set within the postprocess configuration to replace the generate_report():
tutorials/case_h/config.yml
postprocess:
  report: custom_report.py:main
This option provides hook for a python script and a function within as:
{python_sript}:{function_name}
The requirements are that the script to be located within the same directory as the configuration file, whereas the function must receive a floatcsep.experiment.Experiment as argument
def main(experiment):
    """

    Args:
        experiment: a floatcsep.experiment.Experiment class

    """
In this way, the report function use all the Experiment attributes/methods to access catalogs, forecasts and test results. The script tutorials/case_h/custom_report.py can also be viewed directly on GitHub, where it is exemplified how to access the experiment artifacts.

Running the experiment 

The experiment can be run by simply navigating to the tutorials/case_h folder in the terminal and typing.
$ floatcsep run config.yml
This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in results/report.md.

GitHub | CSEP Website | pyCSEP | Download PDF