Detailed Description

The class wraps the Shogun's C-style LBFGS minimizer.

Definition at line 49 of file LBFGSMinimizer.h.

Inheritance diagram for LBFGSMinimizer:

Public Member Functions
	LBFGSMinimizer ()

	LBFGSMinimizer (FirstOrderCostFunction *fun)

virtual	~LBFGSMinimizer ()

virtual float64_t	minimize ()

virtual bool	supports_batch_update () const

virtual void	set_lbfgs_parameters (int m=100, int max_linesearch=1000, ELBFGSLineSearch linesearch=BACKTRACKING_STRONG_WOLFE, int max_iterations=1000, float64_t delta=0.0, int past=0, float64_t epsilon=1e-5, float64_t min_step=1e-20, float64_t max_step=1e+20, float64_t ftol=1e-4, float64_t wolfe=0.9, float64_t gtol=0.9, float64_t xtol=1e-16, float64_t orthantwise_c=0.0, int orthantwise_start=0, int orthantwise_end=1)

virtual CMinimizerContext *	save_to_context ()

virtual void	load_from_context (CMinimizerContext *context)

virtual void	set_cost_function (FirstOrderCostFunction *fun)

virtual void	set_penalty_weight (float64_t penalty_weight)

virtual void	set_penalty_type (Penalty *penalty_type)

Protected Member Functions
virtual void	init_minimization ()

virtual void	update_context (CMinimizerContext *context)

virtual float64_t	get_penalty (SGVector< float64_t > var)

virtual void	update_gradient (SGVector< float64_t > gradient, SGVector< float64_t > var)

Protected Attributes
int	m_m

int	m_max_linesearch

int	m_linesearch

int	m_max_iterations

float64_t	m_delta

int	m_past

float64_t	m_epsilon

float64_t	m_min_step

float64_t	m_max_step

float64_t	m_ftol

float64_t	m_wolfe

float64_t	m_gtol

float64_t	m_xtol

float64_t	m_orthantwise_c

int	m_orthantwise_start

int	m_orthantwise_end

SGVector< float64_t >	m_target_variable

FirstOrderCostFunction *	m_fun

Penalty *	m_penalty_type

float64_t	m_penalty_weight

Constructor & Destructor Documentation

LBFGSMinimizer ( )

Definition at line 37 of file LBFGSMinimizer.cpp.

LBFGSMinimizer ( FirstOrderCostFunction * fun )

Constructor

Parameters

fun	cost function (user have to manully delete the pointer)

Definition at line 47 of file LBFGSMinimizer.cpp.

~LBFGSMinimizer ( )

virtual

Definition at line 43 of file LBFGSMinimizer.cpp.

Member Function Documentation

virtual float64_t get_penalty ( SGVector< float64_t > var )

protectedvirtualinherited

Get the penalty given target variables For L2 penalty, the target variable is \(w\) and the value of penalty is \(\lambda \frac{w^t w}{2}\), where \(\lambda\) is the weight of penalty

Parameters

var	the variable used in regularization

Definition at line 164 of file FirstOrderMinimizer.h.

void init_minimization ( )

protectedvirtual

Init before minimization

Definition at line 94 of file LBFGSMinimizer.cpp.

virtual void load_from_context ( CMinimizerContext * context )

virtual

Load the given context object to restores mutable variables Usually it is used in deserialization.

Parameters

context a context object

Reimplemented from FirstOrderMinimizer.

Definition at line 142 of file LBFGSMinimizer.h.

float64_t minimize ( )

virtual

Do minimization and get the optimal value

Returns: optimal value

Implements FirstOrderMinimizer.

Definition at line 101 of file LBFGSMinimizer.cpp.

virtual CMinimizerContext* save_to_context ( )

virtual

Return a context object which stores mutable variables Usually it is used in serialization.

Returns: a context object

Reimplemented from FirstOrderMinimizer.

Definition at line 135 of file LBFGSMinimizer.h.

virtual void set_cost_function ( FirstOrderCostFunction * fun )

virtualinherited

Set cost function used in the minimizer

Parameters

fun	the cost function

Definition at line 92 of file FirstOrderMinimizer.h.

void set_lbfgs_parameters	(	int	m = `100`,
		int	max_linesearch = `1000`,
		ELBFGSLineSearch	linesearch = `BACKTRACKING_STRONG_WOLFE`,
		int	max_iterations = `1000`,
		float64_t	delta = `0.0`,
		int	past = `0`,
		float64_t	epsilon = `1e-5`,
		float64_t	min_step = `1e-20`,
		float64_t	max_step = `1e+20`,
		float64_t	ftol = `1e-4`,
		float64_t	wolfe = `0.9`,
		float64_t	gtol = `0.9`,
		float64_t	xtol = `1e-16`,
		float64_t	orthantwise_c = `0.0`,
		int	orthantwise_start = `0`,
		int	orthantwise_end = `1`
	)

virtual

set L-BFGS parameters For details please see shogun/optimization/lbfgs/lbfgs.h

Parameters

m	The number of corrections to approximate the inverse hessian matrix. Default value is 100.
max_linesearch	The maximum number of trials to do line search for each L-BFGS update. Default value is 1000.
linesearch	The line search algorithm. Default value is using the backtracking with the strong Wolfe condition line search
max_iterations	The maximum number of iterations for L-BFGS update. Default value is 1000.
delta	Delta for convergence test based on the change of function value. Default value is 0.
past	Distance for delta-based convergence test. Default value is 0.
epsilon	Epsilon for convergence test based on the change of gradient. Default value is 1e-5
min_step	The minimum step of the line search. The default value is 1e-20
max_step	The maximum step of the line search. The default value is 1e+20
ftol	A parameter used in Armijo condition. Default value is 1e-4
wolfe	A parameter used in curvature condition. Default value is 0.9
gtol	A parameter used in Morethuente linesearch to control the accuracy. Default value is 0.9
xtol	The machine precision for floating-point values. Default value is 1e-16.
orthantwise_c	Coeefficient for the L1 norm of variables. This parameter should be set to zero for standard minimization problems. Setting this parameter to a positive value activates Orthant-Wise Limited-memory Quasi-Newton (OWL-QN) method. Default value is 0.
orthantwise_start	Start index for computing L1 norm of the variables. This parameter is valid only for OWL-QN method. Default value is 0.
orthantwise_end	End index for computing L1 norm of the variables. Default value is 1.

Definition at line 58 of file LBFGSMinimizer.cpp.

virtual void set_penalty_type ( Penalty * penalty_type )

virtualinherited

Set the type of penalty For example, L2 penalty

Parameters

penalty_type the type of penalty. If NULL is given, regularization is not enabled.

Definition at line 137 of file FirstOrderMinimizer.h.

virtual void set_penalty_weight ( float64_t penalty_weight )

virtualinherited

Set the weight of penalty

Parameters

penalty_weight the weight of penalty, which is positive

Definition at line 126 of file FirstOrderMinimizer.h.

virtual bool supports_batch_update ( ) const

virtual

Does minimizer support batch update?

Returns: whether minimizer supports batch update

Implements FirstOrderMinimizer.

Definition at line 73 of file LBFGSMinimizer.h.

virtual void update_context ( CMinimizerContext * context )

protectedvirtualinherited

Update a context object to store mutable variables

Parameters

context a context object

Reimplemented in FirstOrderStochasticMinimizer, SMDMinimizer, and SMIDASMinimizer.

Definition at line 147 of file FirstOrderMinimizer.h.

virtual void update_gradient	(	SGVector< float64_t >	gradient,
		SGVector< float64_t >	var
	)

protectedvirtualinherited

Add gradient of the penalty wrt target variables to unpenalized gradient For least sqaure with L2 penalty,

\[ L2f(w)=f(w) + L2(w) \]

where \( f(w)=\sum_i{(y_i-w^T x_i)^2}\) is the least sqaure cost function and \(L2(w)=\lambda \frac{w^t w}{2}\) is the L2 penalty

Target variables is \(w\) Unpenalized gradient is \(\frac{\partial f(w) }{\partial w}\) Gradient of the penalty wrt target variables is \(\frac{\partial L2(w) }{\partial w}\)

Parameters

gradient	unpenalized gradient wrt its target variable
var	the target variable

Definition at line 190 of file FirstOrderMinimizer.h.

Member Data Documentation

float64_t m_delta

protected

Delta for convergence test based on the change of function value.

Definition at line 174 of file LBFGSMinimizer.h.

float64_t m_epsilon

protected

Epsilon for convergence test based on the change of gradient.

Definition at line 180 of file LBFGSMinimizer.h.

float64_t m_ftol

protected

A parameter used in Armijo condition.

Definition at line 189 of file LBFGSMinimizer.h.

FirstOrderCostFunction* m_fun

protectedinherited

Cost function

Definition at line 205 of file FirstOrderMinimizer.h.

float64_t m_gtol

protected

A parameter used in Morethuente linesearch to control the accuracy.

Definition at line 195 of file LBFGSMinimizer.h.

int m_linesearch

protected

The line search algorithm.

Definition at line 168 of file LBFGSMinimizer.h.

int m_m

protected

The number of corrections to approximate the inverse hessian matrix.

Definition at line 162 of file LBFGSMinimizer.h.

int m_max_iterations

protected

The maximum number of iterations for L-BFGS update.

Definition at line 171 of file LBFGSMinimizer.h.

int m_max_linesearch

protected

The maximum number of trials to do line search for each L-BFGS update.

Definition at line 165 of file LBFGSMinimizer.h.

float64_t m_max_step

protected

The maximum step of the line search.

Definition at line 186 of file LBFGSMinimizer.h.

float64_t m_min_step

protected

The minimum step of the line search.

Definition at line 183 of file LBFGSMinimizer.h.

float64_t m_orthantwise_c

protected

Coeefficient for the L1 norm of variables.

Definition at line 201 of file LBFGSMinimizer.h.

int m_orthantwise_end

protected

End index for computing L1 norm of the variables.

Definition at line 207 of file LBFGSMinimizer.h.

int m_orthantwise_start

protected

Start index for computing L1 norm of the variables.

Definition at line 204 of file LBFGSMinimizer.h.

int m_past

protected

Distance for delta-based convergence test.

Definition at line 177 of file LBFGSMinimizer.h.

Penalty* m_penalty_type

protectedinherited

the type of penalty

Definition at line 208 of file FirstOrderMinimizer.h.

float64_t m_penalty_weight

protectedinherited

the weight of penalty

Definition at line 211 of file FirstOrderMinimizer.h.

SGVector<float64_t> m_target_variable

protected

Target variable

Definition at line 210 of file LBFGSMinimizer.h.

float64_t m_wolfe

protected

A parameter used in curvature condition.

Definition at line 192 of file LBFGSMinimizer.h.

float64_t m_xtol

protected

The machine precision for floating-point values.

Definition at line 198 of file LBFGSMinimizer.h.

The documentation for this class was generated from the following files:

Detailed Description

Public Member Functions

Protected Member Functions

Protected Attributes

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation