B - Multiple Models and Tests


In a terminal, navigate to floatcsep/examples/case_b and type:

$ floatcsep run config.yml

After the calculation is complete, the results will be summarized in results/report.md.


The following example is an experiment including multiple forecasts and evaluations. The input structure of the experiment is:

    └──  models
        ├── model_a.csv
        ├── model_b.csv
        ├── model_c.csv
        └── model_d.csv
    ├── config.yml
    ├── catalog.json
    ├── models.yml
    ├── tests.yml
    └── region.txt

The testing catalog is now defined in json format, which is the default catalog used by floatcsep, as it allows the storage of metadata.


An user-defined catalog can be saved as json with CSEPCatalog.write_json() using pycsep


In this example, the time, region and catalog specifications are written in the config.yml file.

  start_date: 2010-1-1T00:00:00
  end_date: 2020-1-1T00:00:00

  region: region.txt
  mag_min: 4.0
  mag_max: 8.0
  mag_bin: 0.1
  depth_min: 0
  depth_max: 70

catalog: catalog.json

whereas the models’ and tests’ configurations are referred to external files for readability

models: models.yml
test_config: tests.yml


The model configuration is now set in the models.yml file, where a list of model names specify their file paths.

- Model A:
    path: models/model_a.csv
- Model B:
    path: models/model_b.csv
- Model C:
    path: models/model_c.csv
- Model D:
    path: models/model_d.csv


The evaluations are defined in the tests.yml file as a list of evaluation names, with their functions and plots (see Evaluations). In this example, we use the N-, M-, S- and CL-consistency tests, along with the comparison T-test.

- N-test:
    func: poisson_evaluations.number_test
    plot_func: plot_poisson_consistency_test
- S-test:
    func: poisson_evaluations.spatial_test
    plot_func: plot_poisson_consistency_test
      one_sided_lower: True
- M-test:
    func: poisson_evaluations.magnitude_test
    plot_func: plot_poisson_consistency_test
      one_sided_lower: True
- CL-test:
    func: poisson_evaluations.conditional_likelihood_test
    plot_func: plot_poisson_consistency_test
      one_sided_lower: True
- T-test:
    func: poisson_evaluations.paired_t_test
    ref_model: Model A
    plot_func: plot_comparison_test


Plotting keyword arguments can be set in the plot_kwargs option - see plot_poisson_consistency_test() and plot_comparison_test() -.


Comparison tests (such as the paired_t_test) requires a reference model, whose name should be set in ref_model at the given test configuration.

Running the experiment

The experiment can be run by simply navigating to the examples/case_b folder in the terminal an type.

floatcsep run config.yml

This will automatically set all the file paths of the calculation (testing catalogs, evaluation results, figures) and will display a summarized report in results/report.md.