Public Member Functions | Protected Member Functions

CCrossValidation Class Reference


Detailed Description

base class for cross-validation evaluation. Given a learning machine, a splitting strategy, an evaluation criterium, features and correspnding labels, this provides an interface for cross-validation. Results may be retrieved using the evaluate method. A number of repetitions may be specified for obtaining more accurate results. The arithmetic mean of different runs is returned along with confidence intervals, if a p-value is specified. Default number of runs is one, confidence interval combutation is disabled.

This class calculates an evaluation criterium of every fold and then calculates the arithmetic mean of all folds. This is for example suitable for the AUC or for Accuracy. However, for example F1-measure may not be merged this way (result will be biased). To solve this, different sub-classes may average results of each cross validation fold differently by overwriting the evaluate_one_run method.

See [Forman, G. and Scholz, M. (2009). Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. Technical report, HP Laboratories.] for details on this subject.

Definition at line 77 of file CrossValidation.h.

Inheritance diagram for CCrossValidation:
Inheritance graph
[legend]

List of all members.

Public Member Functions

 CCrossValidation ()
 CCrossValidation (CMachine *machine, CFeatures *features, CLabels *labels, CSplittingStrategy *splitting_strategy, CEvaluation *evaluation_criterium)
virtual ~CCrossValidation ()
EEvaluationDirection get_evaluation_direction ()
CrossValidationResult evaluate ()
CMachineget_machine () const
void set_num_runs (int32_t num_runs)
void set_conf_int_alpha (float64_t m_conf_int_alpha)
virtual const char * get_name () const

Protected Member Functions

virtual float64_t evaluate_one_run ()

Constructor & Destructor Documentation

constructor

Definition at line 20 of file CrossValidation.cpp.

CCrossValidation ( CMachine machine,
CFeatures features,
CLabels labels,
CSplittingStrategy splitting_strategy,
CEvaluation evaluation_criterium 
)

constructor

Parameters:
machine learning machine to use
features features to use for cross-validation
labels labels that correspond to the features
splitting_strategy splitting strategy to use
evaluation_criterium evaluation criterium to use

Definition at line 34 of file CrossValidation.cpp.

~CCrossValidation (  )  [virtual]

destructor

Definition at line 25 of file CrossValidation.cpp.


Member Function Documentation

CrossValidationResult evaluate (  ) 

method for evaluation. Performs cross-validation. Is repeated m_num_runs. If this number is larger than one, a confidence interval is calculated if m_conf_int_alpha is (0<p<1). By default m_num_runs=1 and m_conf_int_alpha=0

Returns:
result of evaluation

Definition at line 82 of file CrossValidation.cpp.

float64_t evaluate_one_run (  )  [protected, virtual]

Evaluates one single cross-validation run. Current implementation evaluates each fold separately and then calculates arithmetic mean. Suitable for accuracy and AUC for example. NOT for F1-measure. Has to be overridden by sub-classes if results have to be merged differently

Returns:
evaluation result of one cross-validation run

Definition at line 131 of file CrossValidation.cpp.

EEvaluationDirection get_evaluation_direction (  ) 
Returns:
in which direction is the best evaluation value?

Definition at line 98 of file CrossValidation.h.

CMachine * get_machine (  )  const
Returns:
underlying learning machine

Definition at line 76 of file CrossValidation.cpp.

virtual const char* get_name ( void   )  const [virtual]
Returns:
name of the SGSerializable

Implements CSGObject.

Definition at line 122 of file CrossValidation.h.

void set_conf_int_alpha ( float64_t  m_conf_int_alpha  ) 

setter for the number of runs to use for evaluation

Definition at line 112 of file CrossValidation.cpp.

void set_num_runs ( int32_t  num_runs  ) 

setter for the number of runs to use for evaluation

Definition at line 123 of file CrossValidation.cpp.


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines

SHOGUN Machine Learning Toolbox - Documentation