L1PenaltyForTG Class Reference

Detailed Description

The is the base class for L1 penalty/regularization within the FirstOrderMinimizer framework.

For L1 penalty, \(L1(w)\)

\[ L1(w)=\| w \|_1 = \sum_i \| w_i \| \]

This class implements the truncated gradient method.

Reference: Stochastic Gradient Descent Training for L1-regularized Log-linear Models with Cumulative Penalty

Definition at line 53 of file L1PenaltyForTG.h.

Inheritance diagram for L1PenaltyForTG:

Public Member Functions
	L1PenaltyForTG ()
virtual	~L1PenaltyForTG ()
virtual void	update_variable_for_proximity (SGVector< float64_t > variable, float64_t proximal_weight)
virtual void	update_context (CMinimizerContext *context)
virtual void	load_from_context (CMinimizerContext *context)
virtual float64_t	get_penalty (float64_t variable)
virtual float64_t	get_penalty_gradient (float64_t variable, float64_t gradient_of_variable)
virtual void	set_rounding_epsilon (float64_t epsilon)
virtual float64_t	get_sparse_variable (float64_t variable, float64_t penalty_weight)

Protected Attributes
float64_t	m_u
SGVector< float64_t >	m_q
float64_t	m_rounding_epsilon

Constructor & Destructor Documentation

L1PenaltyForTG

(

)

Definition at line 57 of file L1PenaltyForTG.h.

virtual ~L1PenaltyForTG ( )

virtual

Definition at line 60 of file L1PenaltyForTG.h.

Member Function Documentation

virtual float64_t get_penalty ( float64_t  variable )

virtualinherited

Given the value of a target variable, this method returns the penalty of the variable

Parameters

variable

value of the variable

Returns

penalty of the variable

Implements Penalty.

Definition at line 66 of file L1Penalty.h.

virtual float64_t get_penalty_gradient ( float64_t  variable, float64_t  gradient_of_variable  )

virtualinherited

Return the gradient of the penalty wrt a target variable

Note that for L1 penalty we do not compute the gradient/sub-gradient in our implementation. Instead, we do a proximal projection.

Parameters

variable	value of a target variable
gradient_of_variable	unregularized/unpenalized gradient of the variable

Returns

the gradient of the penalty wrt the variable

Implements Penalty.

Definition at line 79 of file L1Penalty.h.

virtual float64_t get_sparse_variable ( float64_t  variable, float64_t  penalty_weight  )

virtualinherited

Get the sparse variable

Parameters

variable	the raw variable
penalty_weight	weight of the penalty

Returns

sparse value of the variable

Implements SparsePenalty.

Definition at line 128 of file L1Penalty.h.

virtual void load_from_context ( CMinimizerContext *  context )

virtual

Load the given context object to restore mutable variables

Parameters

context

a context object

Reimplemented from L1Penalty.

Definition at line 115 of file L1PenaltyForTG.h.

virtual void set_rounding_epsilon ( float64_t  epsilon )

virtualinherited

Set the rounding epsilon

Parameters

epsilon

rounding epsilon

Definition at line 87 of file L1Penalty.h.

virtual void update_context ( CMinimizerContext *  context )

virtual

Update a context object to store mutable variables used in learning rate

Parameters

context

a context object

Reimplemented from L1Penalty.

Definition at line 96 of file L1PenaltyForTG.h.

virtual void update_variable_for_proximity ( SGVector< float64_t >  variable, float64_t  proximal_weight  )

virtual

Do proximal projection/operation in place

Parameters

variable	the raw variable
proximal_weight	weight of the penalty

Reimplemented from L1Penalty.

Definition at line 66 of file L1PenaltyForTG.h.

Member Data Documentation

SGVector<float64_t> m_q

protected

q is defined in Figure 2 of the reference

Definition at line 132 of file L1PenaltyForTG.h.

float64_t m_rounding_epsilon

protectedinherited

rounding epsilon

Definition at line 148 of file L1Penalty.h.

float64_t m_u

protected

u is defined in Figure 2 of the reference

Definition at line 130 of file L1PenaltyForTG.h.

---

The documentation for this class was generated from the following file:

• L1PenaltyForTG.h