F - Testing Catalog-Based Forecasts

This example shows how set up an experiment with a time-dependent model, whose forecast files already exist.

TL; DR

In a terminal, navigate to floatcsep/tutorials/case_f and type:

$ floatcsep run config.yml

After the calculation is complete, the results will be summarized in results/report.md.

Experiment Components

The source files can be found in the tutorials/case_e folder or in GitHub. The experiment structure is as follows:

case_f
    └──  etas
        ├── forecasts
            ├── etas_2016-11-14_2016-11-15.csv  (forecast files)
            ...
            └── etas_2016-11-20_2016-11-21.csv
    ├── catalog.csv
    ├── config.yml
    ├── models.yml
    └── tests.yml

  • The model to be evaluated (etas) is a collection of daily forecasts from 2016-11-14 until 2016-11-21.

Important

The forecasts must be located in a folder forecasts inside the model folder. This is meant for consistency with models based on source codes (see subsequent tutorials).

Model

The time-dependency of a model is manifested here by the provision of different forecasts, i.e., statistical descriptions of seismicity, for different time-windows. In this example, the forecasts were created from an external model https://github.com/lmizrahi/etas (doi:10.1785/0220200231), with which the experiment has no interface. This means that we use only the forecast files and no source code. We leave the handling of a model source code for subsequent tutorials.

Configuration

Time

The configuration is analogous to time-independent models with multiple time-windows (e.g., case C) with the exception that a horizon could be defined instead of intervals, which is the forecast time-window length. The experiment’s class should now be explicited as exp_class: td.

tutorials/case_f/config.yml
time_config:
  start_date: 2016-11-14T00:00:00
  end_date: 2016-11-21T00:00:00
  horizon: 1day
  exp_class: td

Note

floatCSEP is flexible with the definition of time windows/deltas. Alternative string inputs for horizon can be 1-day, 1 day, 1d, etc.

Catalog

The catalog catalog.json was obtained previously by using query_geonet and it was filtered to the testing period. However, it can be re-queried by changing its definition to:

catalog: query_geonet

Models

Some additional arguments should be passed to a time-dependent model, such as its class (‘td’ for time-dependent) and the number of simulations.

tutorials/case_f/config.yml
- etas:
    path: etas/
    class: td
    n_sims: 100

Note

For consistency with time-dependent models that will create forecasts from a source code, the path should point to the folder of the model, which itself should contain a sub-folder named {path}/forecasts where the files are located.

Important

Note that for catalog-based forecasts, the model should explicit the number of simulations. This is meant for forecast files that contain synthetic catalogs with zero-event simulations, and therefore do not contain the total number of synthetic catalogs used.

Tests

With time-dependent models, now catalog evaluations found in csep.core.catalog_evaluations can be used.

- Catalog_N-test:
    func: catalog_evaluations.number_test
    plot_func:
      - plot_number_test:
          plot_args:
            title: Test distribution
      - plot_consistency_test:
          plot_kwargs:
            one_sided_lower: True
- Catalog_S-test:
    func: catalog_evaluations.spatial_test
    plot_func:
      - plot_consistency_test:
          plot_kwargs:
            one_sided_lower: True

Note

It is possible to assign two plotting functions to a test, whose plot_args and plot_kwargs can be placed indented beneath

Running the experiment

The experiment can be run by simply navigating to the tutorials/case_h folder in the terminal and typing.

$ floatcsep run config.yml

This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in results/report.md.