详细描述

The class implements the stochastic variance reduced gradient (SVRG) minimizer.

Reference: Johnson, Rie, and Tong Zhang. "Accelerating stochastic gradient descent using predictive variance reduction." Advances in Neural Information Processing Systems. 2013.

在文件 SVRGMinimizer.h 第 47 行定义.

类 SVRGMinimizer 继承关系图:

[图例]

Public 成员函数
	SVRGMinimizer ()

	SVRGMinimizer (FirstOrderSAGCostFunction *fun)

virtual	~SVRGMinimizer ()

virtual float64_t	minimize ()

virtual void	set_sgd_number_passes (int32_t sgd_passes)

virtual void	set_average_update_interval (int32_t interval)

virtual bool	supports_batch_update () const

virtual void	set_gradient_updater (DescendUpdater *gradient_updater)

virtual void	set_number_passes (int32_t num_passes)

virtual void	load_from_context (CMinimizerContext *context)

virtual void	set_learning_rate (LearningRate *learning_rate)

virtual int32_t	get_iteration_counter ()

virtual void	set_cost_function (FirstOrderCostFunction *fun)

virtual CMinimizerContext *	save_to_context ()

virtual void	set_penalty_weight (float64_t penalty_weight)

virtual void	set_penalty_type (Penalty *penalty_type)

Protected 成员函数
virtual void	init_minimization ()

virtual void	do_proximal_operation (SGVector< float64_t >variable_reference)

virtual void	update_context (CMinimizerContext *context)

virtual float64_t	get_penalty (SGVector< float64_t > var)

virtual void	update_gradient (SGVector< float64_t > gradient, SGVector< float64_t > var)

Protected 属性
int32_t	m_num_sgd_passes

int32_t	m_svrg_interval

SGVector< float64_t >	m_average_gradient

SGVector< float64_t >	m_previous_variable

DescendUpdater *	m_gradient_updater

int32_t	m_num_passes

int32_t	m_cur_passes

int32_t	m_iter_counter

LearningRate *	m_learning_rate

FirstOrderCostFunction *	m_fun

Penalty *	m_penalty_type

float64_t	m_penalty_weight

构造及析构函数说明

SVRGMinimizer ( )

Default constructor

在文件 SVRGMinimizer.cpp 第 37 行定义.

SVRGMinimizer ( FirstOrderSAGCostFunction * fun )

Constructor

参数

fun	stochastic cost function

在文件 SVRGMinimizer.cpp 第 47 行定义.

~SVRGMinimizer ( )

virtual

Destructor

在文件 SVRGMinimizer.cpp 第 43 行定义.

成员函数说明

virtual void do_proximal_operation ( SGVector< float64_t > variable_reference )

protectedvirtualinherited

Do proximal update in place

参数

variable_reference variable_reference to be updated

在文件 FirstOrderStochasticMinimizer.h 第 167 行定义.

virtual int32_t get_iteration_counter ( )

virtualinherited

How many samples/mini-batch does the minimizer use?

返回: the number of samples/mini-batches used in optimization

在文件 FirstOrderStochasticMinimizer.h 第 160 行定义.

virtual float64_t get_penalty ( SGVector< float64_t > var )

protectedvirtualinherited

Get the penalty given target variables For L2 penalty, the target variable is \(w\) and the value of penalty is \(\lambda \frac{w^t w}{2}\), where \(\lambda\) is the weight of penalty

参数

var	the variable used in regularization

在文件 FirstOrderMinimizer.h 第 164 行定义.

void init_minimization ( )

protectedvirtual

init the minimization process

重载 FirstOrderStochasticMinimizer .

在文件 SVRGMinimizer.cpp 第 61 行定义.

virtual void load_from_context ( CMinimizerContext * context )

virtualinherited

Load the given context object to restores mutable variables Usually it is used in deserialization.

参数

context a context object

重载 FirstOrderMinimizer .

被 SMDMinimizer , 以及 SMIDASMinimizer 重载.

在文件 FirstOrderStochasticMinimizer.h 第 136 行定义.

float64_t minimize ( )

virtual

Do minimization and get the optimal value

返回: optimal value

实现了 FirstOrderStochasticMinimizer.

在文件 SVRGMinimizer.cpp 第 81 行定义.

virtual CMinimizerContext* save_to_context ( )

virtualinherited

Return a context object which stores mutable variables Usually it is used in serialization.

返回: a context object

被 LBFGSMinimizer , 以及 NLOPTMinimizer 重载.

在文件 FirstOrderMinimizer.h 第 103 行定义.

virtual void set_average_update_interval ( int32_t interval )

virtual

Set the number of interval to average stochastic sample gradients

If we have \((n-g)\) passes to go through data and the interval is \(k\), we will average stochastic sample gradients at the 0-th, k-th, 2k-th, 3k-th, ... pass

Note that \(n\) is the total number to go through data and \(g\) is the number of using SGDMinimizer to initialize variables,

参数

interval how often to average stochastic sample gradients

在文件 SVRGMinimizer.h 第 94 行定义.

virtual void set_cost_function ( FirstOrderCostFunction * fun )

virtualinherited

Set cost function used in the minimizer

参数

fun	the cost function

在文件 FirstOrderMinimizer.h 第 92 行定义.

virtual void set_gradient_updater ( DescendUpdater * gradient_updater )

virtualinherited

Set a gradient updater

参数

gradient_updater the gradient_updater

在文件 FirstOrderStochasticMinimizer.h 第 104 行定义.

virtual void set_learning_rate ( LearningRate * learning_rate )

virtualinherited

Set the learning rate of a minimizer

参数

learning_rate learn rate or step size

在文件 FirstOrderStochasticMinimizer.h 第 151 行定义.

virtual void set_number_passes ( int32_t num_passes )

virtualinherited

Set the number of times to go through all data points (samples) For example, num_passes=1 means go through all data points once.

Recall that a stochastic cost function \(f(w)\) can be written as \(\sum_i{ f_i(w) }\), where \(f_i(w)\) is the differentiable function for the i-th sample.

参数

num_passes the number of times

在文件 FirstOrderStochasticMinimizer.h 第 125 行定义.

virtual void set_penalty_type ( Penalty * penalty_type )

virtualinherited

Set the type of penalty For example, L2 penalty

参数

penalty_type the type of penalty. If NULL is given, regularization is not enabled.

在文件 FirstOrderMinimizer.h 第 137 行定义.

virtual void set_penalty_weight ( float64_t penalty_weight )

virtualinherited

Set the weight of penalty

参数

penalty_weight the weight of penalty, which is positive

在文件 FirstOrderMinimizer.h 第 126 行定义.

virtual void set_sgd_number_passes ( int32_t sgd_passes )

virtual

Set the number to go through data using SGDMinimizer to initialize variables before SVRG minimization

参数

sgd_passes the number to go through data using SGDMinimizer

在文件 SVRGMinimizer.h 第 72 行定义.

virtual bool supports_batch_update ( ) const

virtualinherited

Does minimizer support batch update

返回: whether minimizer supports batch update

实现了 FirstOrderMinimizer.

在文件 FirstOrderStochasticMinimizer.h 第 98 行定义.

virtual void update_context ( CMinimizerContext * context )

protectedvirtualinherited

Update a context object to store mutable variables

参数

context a context object

重载 FirstOrderMinimizer .

被 SMDMinimizer , 以及 SMIDASMinimizer 重载.

在文件 FirstOrderStochasticMinimizer.h 第 187 行定义.

virtual void update_gradient	(	SGVector< float64_t >	gradient,
		SGVector< float64_t >	var
	)

protectedvirtualinherited

Add gradient of the penalty wrt target variables to unpenalized gradient For least sqaure with L2 penalty,

\[ L2f(w)=f(w) + L2(w) \]

where \( f(w)=\sum_i{(y_i-w^T x_i)^2}\) is the least sqaure cost function and \(L2(w)=\lambda \frac{w^t w}{2}\) is the L2 penalty

Target variables is \(w\) Unpenalized gradient is \(\frac{\partial f(w) }{\partial w}\) Gradient of the penalty wrt target variables is \(\frac{\partial L2(w) }{\partial w}\)

参数

gradient	unpenalized gradient wrt its target variable
var	the target variable

在文件 FirstOrderMinimizer.h 第 190 行定义.

类成员变量说明

SGVector<float64_t> m_average_gradient

protected

used to store average gradient

在文件 SVRGMinimizer.h 第 119 行定义.

int32_t m_cur_passes

protectedinherited

current iteration to go through data

在文件 FirstOrderStochasticMinimizer.h 第 215 行定义.

FirstOrderCostFunction* m_fun

protectedinherited

Cost function

在文件 FirstOrderMinimizer.h 第 205 行定义.

DescendUpdater* m_gradient_updater

protectedinherited

the gradient update step

在文件 FirstOrderStochasticMinimizer.h 第 200 行定义.

int32_t m_iter_counter

protectedinherited

number of used samples/mini-batches

在文件 FirstOrderStochasticMinimizer.h 第 218 行定义.

LearningRate* m_learning_rate

protectedinherited

learning_rate object

在文件 FirstOrderStochasticMinimizer.h 第 221 行定义.

int32_t m_num_passes

protectedinherited

iteration to go through data

在文件 FirstOrderStochasticMinimizer.h 第 212 行定义.

int32_t m_num_sgd_passes

protected

the number to go through data using SGD before SVRG update

在文件 SVRGMinimizer.h 第 113 行定义.

Penalty* m_penalty_type

protectedinherited

the type of penalty

在文件 FirstOrderMinimizer.h 第 208 行定义.

float64_t m_penalty_weight

protectedinherited

the weight of penalty

在文件 FirstOrderMinimizer.h 第 211 行定义.

SGVector<float64_t> m_previous_variable

protected

used to store previous result

在文件 SVRGMinimizer.h 第 122 行定义.

int32_t m_svrg_interval

protected

interval to average gradient

在文件 SVRGMinimizer.h 第 116 行定义.

该类的文档由以下文件生成:

详细描述

Public 成员函数

Protected 成员函数

Protected 属性

构造及析构函数说明

成员函数说明

类成员变量说明