verification: Proper verification rules

> library(verification)

Loading required package: fields

Loading required package: spam

Loading required package: grid


Attaching package: 'grid'

The following object is masked from 'package:timetools':

    unit

Spam version 1.3-0 (2015-10-24) is loaded.
Type 'help( Spam)' or 'demo( spam)' for a short introduction 
and overview of this package.
Help for individual functions is also obtained by adding the
suffix '.spam' to the function name, e.g. 'help( chol.spam)'.


Attaching package: 'spam'

The following objects are masked from 'package:base':

    backsolve, forwardsolve

Loading required package: maps


 # maps v3.1: updated 'world': all lakes moved to separate new #
 # 'lakes' database. Type '?world' or 'news(package="maps")'.  #


Attaching package: 'maps'

The following object is masked from 'package:fma':

    ozone

Loading required package: boot

Loading required package: CircStats

Loading required package: MASS


Attaching package: 'MASS'

The following objects are masked from 'package:fma':

    cement, housing, petrol

Loading required package: dtw

Loading required package: proxy


Attaching package: 'proxy'

The following object is masked from 'package:spam':

    as.matrix

The following objects are masked from 'package:stats':

    as.dist, dist

The following object is masked from 'package:base':

    as.matrix

Loaded dtw v1.18-1. See ?dtw for help, citation("dtw") for use in publication.


Attaching package: 'verification'

The following object is masked from 'package:tseries':

    value

> data("pop")

バージョン: 1.42

関数名 概略
attribute Attribute plot
brier Brier Score
check.func check loss function
conditional.quantile Conditional Quantile Plot
crps Continuous Ranked Probability Score
crpsDecomposition Decompostion of Continuous Ranked Probability Score
disc.dat Discrimination plot dataset.
discrimination.plot Discrimination plot
fss Fractional Skill Score
leps Linear Error in Probability Space (LEPS)
lines.roc Add lines to ROC or attribute diagrams
measurement.error Skill score with measurement error.
multi.cont Multiple Contingency Table Statistics
observation.error Observation Error
performance.diagram Performance Diagram
pop Probability of precipitation (pop) data.
precip.ensemble An ensemble of precipitation forecasts
predcomp.test Time Series Prediction Comparison Test
prob.frcs.dat Probablisitic Forecast Dataset.
probcont2disc Converts continuous probability values into binned discrete probability forecasts.
qrel.plot Quantile Reliability Plot
quantile2disc Convert Continuous Forecast Values to Discrete Forecast Values.
quantileScore Quantile Score
rcrv Reduced centered random variable
reliability.plot Reliability Plot
roc.area Area under curve (AUC) calculation for Response Operating Characteristic curve.
roc.plot Relative operating characteristic curve.
rps Ranked Probability Score
table.stats Verification statistics for a 2 by 2 Contingency Table
table.stats.boot Percentile bootstrap for 2 by 2 table
value Forecast Value Function
verify Verification function

verify

Arguments

  • obs
  • pred
  • p
  • baseline
  • frcst.type... prob, binary, norm.dist, car, cont, quantile
  • obs.type
  • thresholds
  • show
  • bins
  • fudge
  • ...
> obs<- c(28, 72, 23, 2680)
> A <- verify(obs, pred = NULL, frcst.type = "binary", obs.type = "binary")

[1] " Assume data entered as c(n11, n01, n10, n00) Obs*Forecast"

> summary(A)


The forecasts are binary, the observations are binary.
The contingency table for the forecast 
     [,1] [,2]
[1,]   28   72
[2,]   23 2680

PODy =  0.5489 
Std. Err. for POD =  0.06967 
TS   =  0.2276 
Std. Err. for TS =  0.03278 
ETS  =  0.216 
Std. Err. for ETS =  0.03411 
FAR  =  0.7199 
Std. Err. for FAR =  0.03469 
HSS  =  0.3553 
Std. Err. for HSS =  0.04614 
PC   =  0.9661 
Std. Err. for PC =  0.003245 
BIAS =  1.961 
Odds Ratio =  45.31 
Log Odds Ratio =  3.814 
Std. Err. for log Odds Ratio =  0.3057 
Odds Ratio Skill Score =  0.9568 
Std. Err. for Odds Ratio Skill Score =   
Extreme Dependency Score (EDS) =  0.7396 
Std. Err. for EDS =  0.04794 
Symmetric Extreme Dependency Score (SEDS) =  0.5935 
Std. Err. for SEDS =  0.04391 
Extremal Dependence Index (EDI) =  0.7173 
Std. Err. for EDI =  0.06167 
Symmetric Extremal Dependence Index (SEDI) =  0.7527 
Std. Err. for SEDI =  0.06043

> obs <- round(runif(100, 1,5))
> pred <- round(runif(100, 1,5))
> 
> A <- verify(obs, pred, frcst.type = "cat", obs.type = "cat" )
> summary(A)


The forecasts are categorical, the observations are categorical.
Percent Correct      =  0.15 
Heidke Skill Score   =  -0.0835 
Pierce Skill Score   =  -0.0862 
Gerrity Score        =  -0.0184 

 Statistics considering each category in turn. 

Threat Score              0.0435 0.05 0.0769 0.128 0.087 
Bias by cat.               
Percent correct by cat.   0.78 0.62 0.52 0.59 0.79 
Hit Rate (POD) by cat.    0.111 0.08 0.121 0.273 0.182 
False Alarm Rate by cat.  0.154  0.2 0.284 0.321 0.135 
False Alarm Ratio by cat. 0.933 0.882 0.826 0.806 0.857