Evaluations Definition

floatCSEP evaluate forecasts using the routines defined in pyCSEP (See Testing Theory). Depending on the forecast types (e.g., GriddedForecasts or CatalogForecasts), different evaluation functions can be used.

Each evaluation specifies a func parameter, representing the evaluation function to be applied, a configuration of the function with func_kwargs (e.g., number of simulations, confidence intervals) and a plot_func parameter for visualizing the results. Evaluations for GriddedForecasts typically use functions from csep.core.poisson_evaluations or csep.core.binomial_evaluations, while evaluations for CatalogForecasts use functions from csep.core.catalog_evaluations.

Important

An evaluation in test_config points to a pyCSEP evaluation function, valid for the forecast class.

Example Configuration:

test_config.yml

 - S-test:
     func: poisson_evaluations.spatial_test
     plot_func: plot_poisson_consistency_test
     plot_kwargs:
       one_sided_lower: True
 - T-test:
     func: poisson_evaluations.paired_t_test
     ref_model: Model A
     plot_func: plot_comparison_test

Evaluation Parameters

Each evaluation listed in test_config accepts the following parameters:

Parameter	Description
func (required)	Specify which evaluation/test function to run. Must be a pyCSEP `{module}.{function}` suite (e.g., `poisson_evaluations.number_test`) or floatCSEP function.
func_kwargs	Any keyword argument to control the specific func. For example, `poisson_evaluations.spatial_test` may be configured with `num_simulations: 2000`.
plot_func (required)	The function to plot the evaluation results, from either the `csep.utils.plots` module (e.g., `plot_poisson_consistency_test`) or floatCSEP `helpers` module.
plot_args	Arguments passed to customize the plot function. Can be titles, labels, colors, font size, etc. Review the documentation of the respective function.
plot_kwargs	Keyword arguments to customize the plot function. Review the documentation of the respective function.
ref_model	A reference model against which the current model is compared in comparative tests (e.g., Model A).
markdown	A description of the test to be used as caption when reporting results

Evaluations Functions

floatCSEP supports the following evaluations:

Evaluations for GriddedForecasts

Function	Evaluates:
`poisson_evaluations.number_test`	the total number of forecasted events compared to the observed events using a Poisson distribution.
`poisson_evaluations.spatial_test`	the forecasted spatial distribution relative to the observed events using a Poisson distribution.
`poisson_evaluations.magnitude_test`	the forecasted magnitude distribution relative to the observed events using a Poisson distribution.
`poisson_evaluations.conditional_likelihood_test`	the likelihood of the observed events given the forecasted rates, conditioned on the total earthquake occurrences, assuming a Poisson distribution.
`poisson_evaluations.paired_t_test`	the information gain between one forecast to a reference (`ref_model`), and test for a significant difference by using a paired T-test.
`binomial_evaluations.binary_spatial_test`	the forecasted spatial distribution relative to the observed events, assuming a Binary/Bernoulli process.
`binomial_evaluations.binary_likelihood_test`	the likelihood of the observed events given the forecasted rates, assuming a Binary distribution.
`binomial_evaluations.negative_binomial_number_test`	the total number of forecasted events compared to the observed events using a Negative Binomial distribution.
`brier_score`	the forecast skill using a quadratic metric rather than logarithmic. Does not penalize false-negatives as much as log-likelihood metrics.
`vector_poisson_t_w_test`	a forecast skill compared to multiple forecasts, by carrying out the paired_t_test and w_test jointly.
`sequential_likelihood`	the temporal evolution of log-likelihoods scores.
`sequential_information_gain`	the temporal evolution of the information gain in time, compared to a `ref_model`.

Evaluations for CatalogForecasts

Function	Evaluates:
`catalog_evaluations.number_test`	the total number of forecasted events compared to observed events in an earthquake catalog.
`catalog_evaluations.spatial_test`	the spatial distribution of forecasted vs. observed earthquake events in an earthquake catalog.
`catalog_evaluations.magnitude_test`	the magnitude distribution of forecasted events to those observed in the earthquake catalog.
`catalog_evaluations.pseudolikelihood_test`	the pseudolikelihood of the observed events, given the forecasted synthetic catalogs
`catalog_evaluations.calibration_test`	the consistency of multiple test-quantiles in time with the expected uniform distribution using a Kolmogorov-Smirnov test.

Note

Check each function’s docstring to see which func_kwargs are compatible with it.

Plotting Functions

floatCSEP supports the following:

Plotting functions

Plotting function	Compatible with:
`plot_poisson_consistency_test`	`poisson_evaluations.number_test`, `poisson_evaluations.spatial_test`, `poisson_evaluations.magnitude_test`, `poisson_evaluations.conditional_likelihood_test`.
`plot_consistency_test`	`binomial_evaluations.negative_binomial_number_test`, `binomial_evaluations.binary_likelihood_test`, `binomial_evaluations.binary_spatial_test`, `brier_score`, `catalog_evaluations.number_test`, `catalog_evaluations.magnitude_test`, `catalog_evaluations.spatial_test`, `catalog_evaluations.pseudolikelihood_test`
`plot_comparison_test`	`poisson_evaluations.paired_t_test`
`plot_number_test`	`catalog_evaluations.number_test`
`plot_magnitude_test`	`catalog_evaluations.magnitude_test`
`plot_spatial_test`	`catalog_evaluations.spatial_test`
`plot_likelihood_test`	`catalog_evaluations.pseudolikelihood_test`
`plot_calibration_test`	`catalog_evaluations.calibration_test`
`plot_sequential_likelihood>`	`sequential_likelihood`, `sequential_information_gain`
`plot_matrix_comparative_test`	`vector_poisson_t_w_test`

Note

Check each plot functions’s docstring to see which plot_args and plot_kwargs are compatible with it.

It is also possible to assign two or more plotting functions to a test, the plot_args and plot_kwargs of which can be placed as dictionaries indented beneath the functions:

Example:

test_config.yml

- Number Test:
   func: catalog_evaluations.number_test
   plot_func:
      - plot_number_test:
            plot_args:
               title: Number test distribution
      - plot_consistency_test:
            plot_args:
               linewidth: 2
            plot_kwargs:
               one_sided_lower: True

GitHub | CSEP Website | pyCSEP | Download PDF