H - A Time-Dependent Experiment

Here, we run an experiment that accesses, containerizes and executes multiple time-dependent models, and then proceeds to evaluate the forecasts once they are created.

TL; DR

In a terminal, navigate to floatcsep/tutorials/case_h and type:

$ floatcsep run config.yml

After the calculation is complete, the results will be summarized in results/report.md. The experiment region, catalog, forecasts and results can be viewed in the Experiment Dashboard with:

$ floatcsep view config.yml

For troubleshooting, run the experiment with:

$ floatcsep run config.yml --debug

Experiment Components

The experiment input files are:

case_h
    ├── catalog.csv
    ├── config.yml
    ├── tests.yml
    └── models.yml

  • The models.yml contains the instructions to clone and build the source codes from software repositories (e.g., gitlab, Github), and how to interface them with floatCSEP. Once downloaded and built, the experiment structure should look like this:

case_h
    ├── models
        ├── etas
            └── ...     (ETAS source code)
        ├── pymock_poisson
            └── ...     (Poisson-PyMock source code)
        └── pymock_nb
            └── ...     (Negative-Binomial-PyMock source code)
    ├── catalog.csv
    ├── config.yml
    ├── tests.yml
    ├── custom_report.py
    └── models.yml

Configuration

Models

As in Tutorial G, each Model requires to build and execute a source code to generate forecasts. The instructions for each model are located within models.yml, which we further explain here:

Note

The models.yml will define how to interface floatCSEP to each Model, implying that a Model should be developed, or adapted to ensure the interface requirements specified below.

  1. The repository URL of each model and their specific versions if needed (e.g., commit hash, tag, release) are specified as:

    tutorials/case_h/models.yml
    - etas:
    giturl: https://github.com/swiss-seismological-service/etas.git
    repo_hash: csep/v1.0.0
- Poisson Mock:
    giturl: https://github.com/pabloitu/pymock.git
- Negbinom Mock:
    giturl: https://github.com/pabloitu/pymock.git

  2. A path needs to be indicated for each model, to both download the repository contents therein and from where the source code will be executed.

    tutorials/case_h/models.yml
    1- etas:
2    giturl: https://github.com/swiss-seismological-service/etas.git
3    repo_hash: csep/v1.0.0
4    path: models/etas

    Important

    The implies that the inputs (catalog and argument file) should be read by the model from a {path}/input folder, and the forecast outputs stored into a folder {path}/forecasts.

  1. There is some flexibility to interface floatCSEP with a model. For instance, a different filepath can be set for the argument file:

    tutorials/case_h/models.yml
    1- etas:
2    giturl: https://github.com/swiss-seismological-service/etas.git
3    repo_hash: csep/v1.0.0
4    path: models/etas
5    build: conda
6    args_file: args.json

    Note

    floatCSEP can read .txt, .json and .yml format-styled argument files. In all cases, the minimum required arguments, should be formatted as:

    #.txt
start_date = {DATESTRING}
end_date = {DATESTRING}

    #.yml
start_date: {DATESTRING}
end_date: {DATESTRING}

    //.json
"start_date": "{DATESTRING}",
"end_date": "{DATESTRING}"

    floatcsep will dynamically modify the start_date and start_date for each time window run, and any extra arguments will just be added for all time-windows.

  1. The func entry indicates how the models are invoked from a shell terminal.

    tutorials/case_h/models.yml
    - etas:
    func: etas-run
- Poisson Mock:
    func: pymock
- Negbinom Mock:
    func: pymock

    Important

    Please refer to Tutorial G for example of how to set up func for the model and interface it to floatCSEP.

  2. A prefix for the resulting forecast filepaths can be specified beforehand for each model. Without specifying this, the default is {model_name} (e.g., etas, Poisson Mock, Negbinom Mock).

    tutorials/case_h/models.yml
    - Poisson Mock:
    prefix: pymock
- Negbinom Mock:
    prefix: pymock

    The experiment will read the forecasts as:

    {model_path}/forecasts/{prefix}_{start}_{end}.csv

    where start and end follow either the %Y-%m-%dT%H:%M:%S.%f - ISO861 FORMAT, or the short date version %Y-%m-%d if the windows are set by midnight.

  3. Additional function arguments can be passed to the model with the entry func_kwargs. Both Poisson Mock and Negbinom Mock use the same source code, but a different subclass can be defined with func_kwargs (in this case, a Negative-Binomial number distribution instead of Poisson).

    tutorials/case_h/models.yml
    - Poisson Mock:
    func_kwargs:
      n_sims: 1000
      mag_min: 3.5
      seed: 23
- Negbinom Mock:
    func_kwargs:
      n_sims: 1000
      mag_min: 3.5
      seed: 23
      distribution: negbinom
      apply_mc_to_lambda: True

Note

For further details on how to configure time-dependent models, see Models Configuration

Time

The configuration is identical to time-independent models, with the exception that now a horizon can be defined instead of intervals, which is the forecast time-window length. The experiment’s class should now be explicited as exp_class: td

tutorials/case_h/config.yml
time_config:
  start_date: 2016-8-25T00:00:00
  end_date: 2016-8-26T00:00:00
  horizon: 1days
  exp_class: td

Catalog

The catalog was obtained prior to the experiment using query bsi, but it was filtered from 2006 onwards, so it has enough data for the model calibration.

Tests

Catalog-based evaluations found in csep.core.catalog_evaluations can be used.

tutorials/case_h/tests.yml
- Catalog_N-test:
    func: catalog_evaluations.number_test
    func_kwargs:
        verbose: False
    plot_func:
        - plot_test_distribution:
            plot_args:
                title: Number consistency test
                bins: 30
        - plot_consistency_test:
            plot_args:
                title: Number consistency test

Note

It is possible to assign two plotting functions to a test, whose plot_args and plot_kwargs can be placed indented beneath.

Custom Post-Process

A custom reporting function can be set within the postprocess configuration to replace the generate_report():

tutorials/case_h/config.yml
postprocess:
  report: custom_report.py:main

This option provides hook for a Python script and a function within as:

{python_sript}:{function_name}

The script must be located within the same directory as the configuration file, whereas the function must receive a floatcsep.experiment.Experiment as argument:

def main(experiment):
    """

    Args:
        experiment: a floatcsep.experiment.Experiment class

In this way, the report function use all the Experiment attributes/methods to access catalogs, forecasts and test results. The script tutorials/case_h/custom_report.py can also be viewed directly in the GitHub repository, where it is exemplified how to access the experiment artifacts.

Note

For further details on how to configure post-procesing, see:

Running the experiment

The experiment can be run by simply navigating to the tutorials/case_h folder in the terminal and typing:

$ floatcsep run config.yml

This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in results/report.md.

To view the results in a dashboard, type:

$ floatcsep view config.yml

pyCSEP under the hood

Classes and functions used in this tutorial

Where to learn pyCSEP further: