A - Testing a Simple Model

The following example shows the definition of a testing experiment of a single time-independent forecast against a catalog.

TL; DR

In a terminal, navigate to floatcsep/tutorials/case_a and type:

$ floatcsep run config.yml

After the calculation is complete, the results will be summarized in results/report.md. The experiment region, catalog, forecasts and results can be viewed in the Experiment Dashboard with:

$ floatcsep view config.yml

Experiment Components

The source code can be found in the tutorials/case_a folder or in GitHub. The directory structure of the experiment is:

case_a
    ├── region.txt
    ├── catalog.csep
    ├── best_model.dat
    └── config.yml

  • The testing region region.txt consists of a grid with two 1ºx1º bins, defined by its bottom-left nodes. (see Regions in pyCSEP*)/ The grid spacing is obtained automatically. The nodes are:

    tutorials/case_a/region.txt
    # lon lat
0 0
1 0

  • The testing catalog catalog.csep contains only one event and is formatted in the csep_ascii() style (see Catalogs in pyCSEP*). Catalog formats are detected automatically

    tutorials/case_a/catalog.csep
    lon,lat,mag,time_string,depth,cat_id,event_id
0.5,0.5,6.0,2020-06-01T00:00:00,30,1,1
1.5,0.5,7.9,2020-06-02T00:00:00,30,1,2

  • The forecast best_model.dat to be evaluated is written in the .dat format (see Forecasts in pyCSEP). Forecast formats are detected automatically (see floatcsep.utils.file_io.GriddedForecastParsers)

    tutorials/case_a/best_model.dat
    # lon_min lon_max lat_min lat_max depth_min depth_max mag_min mag_max rate
0 1 0 1 0 70 6 8 1.5 1
1 2 0 1 0 70 6 8 0.5 1

Configuration

The experiment is defined by a time-, region-, model- and test-configurations, as well as a catalog and a region. In this example, they are written together in the config.yml file.

Warning

Every file path (e.g., of a catalog) specified in the config.yml file should be relative to the directory containing the configuration file.

Time

The time configuration is manifested in the time_config inset. The simplest definition is to set only the start and end dates of the experiment. These are always UTC date-times in isoformat (%Y-%m-%dT%H:%M:%S.%f - ISO861):

tutorials/case_a/config.yml
time_config:
  start_date: 2020-1-1T00:00:00
  end_date: 2021-1-1T00:00:00

Note

In case the time window are bounded by their midnights, the start_date and end_date can be in the format %Y-%m-%d.

The results of the experiment run will be associated with this time window, whose identifier will be its bounds: 2020-01-01_2021-01-01

Region

The region - a file path or a pycsep function, such as italy_csep_region (check the available regions in csep.core.regions) -, the depth limits and magnitude discretization are defined in the region_config inset.

tutorials/case_a/config.yml
region_config:
  region: region.txt
  mag_min: 6.0
  mag_max: 8.0
  mag_bin: 1.0
  depth_min: 0
  depth_max: 70

Catalog

It is defined in the catalog inset. This should only make reference to a catalog file or a catalog query function (see catalog loaders in csep). floatCSEP will automatically filter the catalog to the experiment time, spatial and magnitude frames:

tutorials/case_a/config.yml
catalog: catalog.csep

Models

The model configuration is set in the models inset with a list of model names, which specify their file paths (and other attributes). Here, we just set the path as best_model.dat, whose format is automatically detected (see Working with conventional gridded forecasts in pyCSEP) .

tutorials/case_a/config.yml
  - best_model:
      path: best_model.dat

Note

A time-independent forecast model has default units of [eq/year] per cell. A forecast defined for a different number of years can be specified with the forecast_unit: {years} attribute.

Evaluations

The experiment’s evaluations are defined in the tests inset. It should be a list of test names making reference to their function and plotting function. These can be either from pyCSEP (see Evaluations) or defined manually. Here, we use the Poisson consistency N-test: its function is poisson_evaluations.number_test with a plotting function plot_poisson_consistency_test

tutorials/case_a/config.yml
  - Poisson N-test:
      func: poisson_evaluations.number_test
      plot_func: plot_consistency_test

Important

See here all available Evaluation Functions, along with their corresponding Plotting Functions.

Note

For further details on how to configure an experiment, models and evaluations, see:

Running the experiment

Run command

The experiment can be run by simply navigating to the tutorials/case_a folder in the terminal and typing.

$ floatcsep run config.yml

This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in results/report.md.

Note

The command floatcsep run {config_file} can be called from any working directory, as long as the specified file paths (e.g. region, models) are relative to the config.yml file.

Results

The run command creates the result path tree for each time window analyzed.

  • The testing catalog of the window is stored in results/{window}/catalog in json format. This is a subset of the global testing catalog.

  • Human-readable results are found in results/{window}/evaluations

  • Catalog, forecasts and evaluation results figures in results/{window}/figures.

  • The complete results are summarized in results/report.md

pyCSEP under the hood

This tutorial uses floatCSEP as the orchestrator, but relies on pyCSEP for functions and objects.

Classes and functions used in this tutorial

Where to learn pyCSEP further: