This class implements streaming features with sparse feature vectors. The vector is represented as an SGSparseVector<T>. Each entry is of type SGSparseVectorEntry<T> with members `feat_index' and `entry'.
This class expects the input from the StreamingFile object to be zero-based, i.e., a feature entered as 1:6.5 would have feat_index=0 and entry=6.5.
The current example is stored as a combination of current_vector and current_label. current_num_features stores the highest dimensionality of examples encountered upto the point of the function call. For example, if the first example is '1:6.5 7:10.0', then current_num_features would be 7 after the first function call.
Since the dimensionality of the feature space is not immediately known initially, current_num_features may increase as more examples are processed and larger dimensions are seen. For this purpose, `expand_if_required()' is provided which when called with a dynamically allocated float or double array and the length, reallocates that array to the new dimensionality (if necessary), setting the newer dimensions to zero, and updates the length parameter to equal the new length of the array.
Definition at line 43 of file StreamingSparseFeatures.h.
Public Member Functions | |
CStreamingSparseFeatures () | |
CStreamingSparseFeatures (CStreamingFile *file, bool is_labelled, int32_t size) | |
~CStreamingSparseFeatures () | |
virtual void | set_vector_reader () |
virtual void | set_vector_and_label_reader () |
virtual void | start_parser () |
virtual void | end_parser () |
virtual bool | get_next_example () |
T | get_feature (int32_t index) |
SGSparseVector< T > | get_vector () |
virtual float64_t | get_label () |
virtual void | release_example () |
virtual void | reset_stream () |
int32_t | set_num_features (int32_t num) |
virtual int32_t | get_dim_feature_space () const |
virtual void | expand_if_required (float32_t *&vec, int32_t &len) |
virtual void | expand_if_required (float64_t *&vec, int32_t &len) |
virtual float32_t | dot (CStreamingDotFeatures *df) |
T | dense_dot (T alpha, T *vec, int32_t dim, T b) |
virtual float64_t | dense_dot (const float64_t *vec2, int32_t vec2_len) |
virtual float32_t | dense_dot (const float32_t *vec2, int32_t vec2_len) |
virtual void | add_to_dense_vec (float64_t alpha, float64_t *vec2, int32_t vec2_len, bool abs_val=false) |
virtual void | add_to_dense_vec (float32_t alpha, float32_t *vec2, int32_t vec2_len, bool abs_val=false) |
int64_t | get_num_nonzero_entries () |
float32_t | compute_squared () |
void | sort_features () |
virtual int32_t | get_num_features () |
virtual int32_t | get_nnz_features_for_vector () |
virtual EFeatureType | get_feature_type () |
virtual EFeatureClass | get_feature_class () |
virtual CFeatures * | duplicate () const |
virtual const char * | get_name () const |
virtual int32_t | get_num_vectors () const |
virtual int32_t | get_size () |
Static Public Member Functions | |
static T | sparse_dot (T alpha, SGSparseVectorEntry< T > *avec, int32_t alen, SGSparseVectorEntry< T > *bvec, int32_t blen) |
Protected Attributes | |
CInputParser < SGSparseVectorEntry< T > > | parser |
The parser object, which reads from input and returns parsed example objects. | |
CStreamingFile * | working_file |
The StreamingFile object to read from. | |
SGSparseVector< T > | current_sgvector |
The current example's feature vector as an SGVector<T> | |
SGSparseVectorEntry< T > * | current_vector |
The current example's feature vector as an SGSparseVectorEntry<T>*. | |
index_t | current_vec_index |
The current vector index. | |
float64_t | current_label |
The current example's label. | |
int32_t | current_length |
Number of set indices in current example. | |
int32_t | current_num_features |
Number of features in current vector (as seen so far upto the current vector). |
Default constructor.
Sets the reading functions to be CStreamingFile::get_*_vector and get_*_vector_and_label depending on the type T.
Definition at line 6 of file StreamingSparseFeatures.cpp.
CStreamingSparseFeatures | ( | CStreamingFile * | file, | |
bool | is_labelled, | |||
int32_t | size | |||
) |
Constructor taking args. Initializes the parser with the given args.
file | StreamingFile object, input file. | |
is_labelled | Whether examples are labelled or not. | |
size | Number of example objects to be stored in the parser at a time. |
Definition at line 13 of file StreamingSparseFeatures.cpp.
Destructor.
Ends the parsing thread. (Waits for pthread_join to complete)
Definition at line 23 of file StreamingSparseFeatures.cpp.
void add_to_dense_vec | ( | float64_t | alpha, | |
float64_t * | vec2, | |||
int32_t | vec2_len, | |||
bool | abs_val = false | |||
) | [virtual] |
Add alpha*current_vector to another float64_t type dense vector. Takes the absolute value of current_vector if specified.
alpha | alpha | |
vec2 | vector to add to, float64_t* | |
vec2_len | length of vector | |
abs_val | true if abs of current_vector should be taken |
Definition at line 192 of file StreamingSparseFeatures.cpp.
void add_to_dense_vec | ( | float32_t | alpha, | |
float32_t * | vec2, | |||
int32_t | vec2_len, | |||
bool | abs_val = false | |||
) | [virtual] |
Add alpha*current_vector to another dense vector. Takes the absolute value of current_vector if specified.
alpha | alpha | |
vec2 | vector to add to | |
vec2_len | length of vector | |
abs_val | true if abs of current_vector should be taken |
Implements CStreamingDotFeatures.
Definition at line 220 of file StreamingSparseFeatures.cpp.
float32_t compute_squared | ( | ) |
Compute sum of squares of features on current vector.
Definition at line 254 of file StreamingSparseFeatures.cpp.
Dot product with another dense vector.
vec2 | The dense vector with which to take the dot product. | |
vec2_len | length of vector |
Implements CStreamingDotFeatures.
Definition at line 172 of file StreamingSparseFeatures.cpp.
Dot product with another float64_t type dense vector.
vec2 | The dense vector with which to take the dot product. | |
vec2_len | length of vector |
Definition at line 152 of file StreamingSparseFeatures.cpp.
T dense_dot | ( | T | alpha, | |
T * | vec, | |||
int32_t | dim, | |||
T | b | |||
) |
compute the dot product between dense weights and a sparse feature vector alpha * sparse^T * w + b
alpha | scalar to multiply with | |
vec | dense vector to compute dot product with | |
dim | length of the dense vector | |
b | bias |
Definition at line 133 of file StreamingSparseFeatures.cpp.
float32_t dot | ( | CStreamingDotFeatures * | df | ) | [virtual] |
Dot product taken with another StreamingDotFeatures object.
Currently only works if it is a CStreamingSparseFeatures object. It takes the dot product of the current_vectors of both objects.
df | CStreamingDotFeatures object. |
Implements CStreamingDotFeatures.
Definition at line 443 of file StreamingSparseFeatures.cpp.
CFeatures * duplicate | ( | ) | const [virtual] |
Duplicate the object.
Implements CFeatures.
Definition at line 304 of file StreamingSparseFeatures.cpp.
void end_parser | ( | ) | [virtual] |
Ends the parsing thread.
Waits for the thread to join.
Implements CStreamingFeatures.
Definition at line 385 of file StreamingSparseFeatures.cpp.
void expand_if_required | ( | float32_t *& | vec, | |
int32_t & | len | |||
) | [virtual] |
Expand the vector passed so that it its length is equal to the dimensionality of the features. The previous values are kept intact through realloc, and the new ones are set to zero.
vec | float32_t* vector | |
len | length of the vector |
Reimplemented from CStreamingDotFeatures.
Definition at line 60 of file StreamingSparseFeatures.cpp.
void expand_if_required | ( | float64_t *& | vec, | |
int32_t & | len | |||
) | [virtual] |
Expand the vector passed so that it its length is equal to the dimensionality of the features. The previous values are kept intact through realloc, and the new ones are set to zero.
vec | float64_t* vector | |
len | length of the vector |
Reimplemented from CStreamingDotFeatures.
Definition at line 72 of file StreamingSparseFeatures.cpp.
int32_t get_dim_feature_space | ( | ) | const [virtual] |
obtain the dimensionality of the feature space
(not mix this up with the dimensionality of the input space, usually obtained via get_num_features())
Implements CStreamingDotFeatures.
Definition at line 437 of file StreamingSparseFeatures.cpp.
T get_feature | ( | int32_t | index | ) |
get a single feature
index | index of feature in this vector |
Definition at line 29 of file StreamingSparseFeatures.cpp.
EFeatureClass get_feature_class | ( | ) | [virtual] |
Return the feature class
Implements CFeatures.
Definition at line 462 of file StreamingSparseFeatures.cpp.
virtual EFeatureType get_feature_type | ( | ) | [virtual] |
float64_t get_label | ( | ) | [virtual] |
Return the label of the current example as a float.
Examples must be labelled, otherwise an error occurs.
Implements CStreamingFeatures.
Definition at line 423 of file StreamingSparseFeatures.cpp.
virtual const char* get_name | ( | ) | const [virtual] |
Return the name.
Implements CSGObject.
Definition at line 339 of file StreamingSparseFeatures.h.
bool get_next_example | ( | ) | [virtual] |
Instructs the parser to return the next example.
This example is stored as the current_example in this object.
Implements CStreamingFeatures.
Definition at line 391 of file StreamingSparseFeatures.cpp.
int32_t get_nnz_features_for_vector | ( | ) | [virtual] |
Return the number of non-zero features in vector
Reimplemented from CStreamingDotFeatures.
Definition at line 456 of file StreamingSparseFeatures.cpp.
int32_t get_num_features | ( | ) | [virtual] |
Return the number of features in the current example.
Implements CStreamingFeatures.
Definition at line 450 of file StreamingSparseFeatures.cpp.
int64_t get_num_nonzero_entries | ( | ) |
Get number of non-zero entries in current sparse vector
Definition at line 248 of file StreamingSparseFeatures.cpp.
int32_t get_num_vectors | ( | ) | const [virtual] |
Return the number of vectors stored in this object.
Implements CFeatures.
Definition at line 310 of file StreamingSparseFeatures.cpp.
int32_t get_size | ( | ) | [virtual] |
Return the size of one T object.
Implements CFeatures.
Definition at line 318 of file StreamingSparseFeatures.cpp.
SGSparseVector< T > get_vector | ( | ) |
Return the current feature vector as an SGSparseVector<T>.
Definition at line 413 of file StreamingSparseFeatures.cpp.
void release_example | ( | ) | [virtual] |
Release the current example, indicating to the parser that it has been processed by the learning algorithm.
The parser is then free to throw away that example.
Implements CStreamingFeatures.
Definition at line 431 of file StreamingSparseFeatures.cpp.
void reset_stream | ( | ) | [virtual] |
Reset the file back to the first example if possible.
Reimplemented from CStreamingFeatures.
Definition at line 46 of file StreamingSparseFeatures.cpp.
int32_t set_num_features | ( | int32_t | num | ) |
set number of features
Sometimes when loading sparse features not all possible dimensions are used. This may pose a problem to classifiers when being applied to higher dimensional test-data. This function allows to artificially explode the feature space
num | the number of features, must be larger than the current number of features |
Definition at line 51 of file StreamingSparseFeatures.cpp.
void set_vector_and_label_reader | ( | ) | [virtual] |
Sets the read function (in case the examples are labelled) to get_*_vector_and_label from CStreamingFile.
The exact function depends on type T.
The parser uses the function set by this while reading labelled examples.
Implements CStreamingFeatures.
Definition at line 328 of file StreamingSparseFeatures.cpp.
void set_vector_reader | ( | ) | [virtual] |
Sets the read function (in case the examples are unlabelled) to get_*_vector() from CStreamingFile.
The exact function depends on type T.
The parser uses the function set by this while reading unlabelled examples.
Implements CStreamingFeatures.
Definition at line 323 of file StreamingSparseFeatures.cpp.
void sort_features | ( | ) |
Ensure features of the current vector are in ascending order. It modifies the current_vector in-place, though a temporary vector is created and later freed.
Definition at line 267 of file StreamingSparseFeatures.cpp.
T sparse_dot | ( | T | alpha, | |
SGSparseVectorEntry< T > * | avec, | |||
int32_t | alen, | |||
SGSparseVectorEntry< T > * | bvec, | |||
int32_t | blen | |||
) | [static] |
compute the dot product between two sparse feature vectors alpha * vec^T * vec
alpha | scalar to multiply with | |
avec | first sparse feature vector | |
alen | avec's length | |
bvec | second sparse feature vector | |
blen | bvec's length |
Definition at line 84 of file StreamingSparseFeatures.cpp.
void start_parser | ( | ) | [virtual] |
Starts the parsing thread.
To be called before trying to use any feature vectors from this object.
Implements CStreamingFeatures.
Definition at line 378 of file StreamingSparseFeatures.cpp.
float64_t current_label [protected] |
The current example's label.
Definition at line 388 of file StreamingSparseFeatures.h.
int32_t current_length [protected] |
Number of set indices in current example.
Definition at line 391 of file StreamingSparseFeatures.h.
int32_t current_num_features [protected] |
Number of features in current vector (as seen so far upto the current vector).
Definition at line 394 of file StreamingSparseFeatures.h.
SGSparseVector<T> current_sgvector [protected] |
The current example's feature vector as an SGVector<T>
Definition at line 379 of file StreamingSparseFeatures.h.
index_t current_vec_index [protected] |
The current vector index.
Definition at line 385 of file StreamingSparseFeatures.h.
SGSparseVectorEntry<T>* current_vector [protected] |
The current example's feature vector as an SGSparseVectorEntry<T>*.
Definition at line 382 of file StreamingSparseFeatures.h.
CInputParser< SGSparseVectorEntry<T> > parser [protected] |
The parser object, which reads from input and returns parsed example objects.
Definition at line 373 of file StreamingSparseFeatures.h.
CStreamingFile* working_file [protected] |
The StreamingFile object to read from.
Reimplemented from CStreamingFeatures.
Definition at line 376 of file StreamingSparseFeatures.h.