Detailed Description

A Streaming File access class.

Vectors are read as one vector per line
NOT YET IMPLEMENTED:
Matrices are written out as one column per line
Sparse Matrices are written out as one column per line (libsvm/svmlight style format)
Strings are written out as one string per line

Definition at line 38 of file StreamingFile.h.

Inheritance diagram for CStreamingFile:

Public Member Functions
	CStreamingFile ()
	CStreamingFile (char *fname, char rw='r')
virtual	~CStreamingFile ()
void	close ()
virtual bool	is_seekable ()
virtual void	reset_stream ()
virtual void	get_vector (VwExample *&ex, int32_t &len)
virtual void	get_vector_and_label (VwExample *&ex, int32_t &len, float64_t &label)
virtual const char *	get_name () const
Dense Vector Access Functions
Functions to access dense vectors of one of several base data types. These functions are used when reading one dense vector at a time from an input source and return the vector and length of the vector by reference.
virtual void	get_vector (bool *&vector, int32_t &len)
virtual void	get_vector (uint8_t *&vector, int32_t &len)
virtual void	get_vector (char *&vector, int32_t &len)
virtual void	get_vector (int32_t *&vector, int32_t &len)
virtual void	get_vector (float32_t *&vector, int32_t &len)
virtual void	get_vector (float64_t *&vector, int32_t &len)
virtual void	get_vector (int16_t *&vector, int32_t &len)
virtual void	get_vector (uint16_t *&vector, int32_t &len)
virtual void	get_vector (int8_t *&vector, int32_t &len)
virtual void	get_vector (uint32_t *&vector, int32_t &len)
virtual void	get_vector (int64_t *&vector, int32_t &len)
virtual void	get_vector (uint64_t *&vector, int32_t &len)
virtual void	get_vector (floatmax_t *&vector, int32_t &len)
Dense Vector And Label Access Functions
Functions to access dense vectors of one of several base data types. These functions are used when reading one dense vector at a time from an input source and return the vector, length and label of the vector by reference.
virtual void	get_vector_and_label (bool *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (uint8_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (char *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (int32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (float32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (float64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (int16_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (uint16_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (int8_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (uint32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (int64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (uint64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_vector_and_label (floatmax_t *&vector, int32_t &len, float64_t &label)
String Access Functions
Functions to access string of one of several base data types. These functions are used when reading one string vector at a time from an input source and return the vector and length of the vector by reference.
virtual void	get_string (bool *&vector, int32_t &len)
virtual void	get_string (uint8_t *&vector, int32_t &len)
virtual void	get_string (char *&vector, int32_t &len)
virtual void	get_string (int32_t *&vector, int32_t &len)
virtual void	get_string (float32_t *&vector, int32_t &len)
virtual void	get_string (float64_t *&vector, int32_t &len)
virtual void	get_string (int16_t *&vector, int32_t &len)
virtual void	get_string (uint16_t *&vector, int32_t &len)
virtual void	get_string (int8_t *&vector, int32_t &len)
virtual void	get_string (uint32_t *&vector, int32_t &len)
virtual void	get_string (int64_t *&vector, int32_t &len)
virtual void	get_string (uint64_t *&vector, int32_t &len)
virtual void	get_string (floatmax_t *&vector, int32_t &len)
String And Label Access Functions
Functions to access strings of one of several base data types. These functions are used when reading one string vector at a time from an input source and return the vector, length and label of the vector by reference.
virtual void	get_string_and_label (bool *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (uint8_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (char *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (int32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (float32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (float64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (int16_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (uint16_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (int8_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (uint32_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (int64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (uint64_t *&vector, int32_t &len, float64_t &label)
virtual void	get_string_and_label (floatmax_t *&vector, int32_t &len, float64_t &label)
Sparse Vector Access Functions
Functions to access sparse vectors of one of several base data types. These functions are used when reading one sparse vector at a time from an input source and return the vector and length of the vector by reference.
virtual void	get_sparse_vector (SGSparseVectorEntry< bool > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< uint8_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< char > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< int32_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< float32_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< float64_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< int16_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< uint16_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< int8_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< uint32_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< int64_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< uint64_t > *&vector, int32_t &len)
virtual void	get_sparse_vector (SGSparseVectorEntry< floatmax_t > *&vector, int32_t &len)
Sparse Vector And Label Access Functions
Functions to access sparse vectors of one of several base data types. These functions are used when reading one sparse vector at a time from an input source and return the vector, length and label of the vector by reference.
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< bool > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< uint8_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< char > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< int32_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< float32_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< float64_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< int16_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< uint16_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< int8_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< uint32_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< int64_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< uint64_t > *&vector, int32_t &len, float64_t &label)
virtual void	get_sparse_vector_and_label (SGSparseVectorEntry< floatmax_t > *&vector, int32_t &len, float64_t &label)
Protected Attributes
CIOBuffer *	buf
	Buffer to hold stuff in memory.
char	task
	Task.
char *	filename
	Name of the handled file.

Constructor & Destructor Documentation

CStreamingFile ( )

default constructor

Definition at line 195 of file StreamingFile.cpp.

CStreamingFile	(	char *	fname,
		char	rw = `'r'`
	)

constructor

Parameters:

	fname	filename to open
	rw	mode, 'r' or 'w'

Definition at line 201 of file StreamingFile.cpp.

~CStreamingFile ( ) [virtual]

default destructor

Definition at line 232 of file StreamingFile.cpp.

Member Function Documentation

void close ( )

Closes the file

Definition at line 57 of file StreamingFile.h.

virtual const char* get_name ( ) const [virtual]

Returns:: object name

Implements CSGObject.

Reimplemented in CStreamingAsciiFile, CStreamingFileFromFeatures, CStreamingFileFromSimpleFeatures< T >, CStreamingFileFromSparseFeatures< T >, CStreamingFileFromStringFeatures< T >, CStreamingVwCacheFile, and CStreamingVwFile.

Definition at line 289 of file StreamingFile.h.

virtual void get_sparse_vector	(	SGSparseVectorEntry< uint32_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< int64_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< uint64_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< floatmax_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< int8_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< bool > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< uint8_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< char > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< int32_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< float32_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< float64_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< int16_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector	(	SGSparseVectorEntry< uint16_t > *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< int64_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< bool > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< uint8_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< char > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< int32_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< float32_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< float64_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< uint16_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< int8_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< uint32_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< uint64_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< floatmax_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_sparse_vector_and_label	(	SGSparseVectorEntry< int16_t > *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string	(	bool *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	float32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	char *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	int32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	float64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	int16_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	int64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	uint64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	floatmax_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	int8_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	uint8_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	uint32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string	(	uint16_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_string_and_label	(	floatmax_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	int8_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	bool *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	uint8_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	char *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	int32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	float64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	int16_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	uint16_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	uint32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	int64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	uint64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_string_and_label	(	float32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector	(	int32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	uint32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	int64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	float32_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	bool *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	float64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	floatmax_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	char *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	int16_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	int8_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	uint16_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector	(	uint8_t *&	vector,
		int32_t &	len
	)			`[virtual]`

void get_vector	(	VwExample *&	ex,
		int32_t &	len
	)			`[virtual]`

Function to read VW examples without labels

Parameters:

	ex	example
	len	length of feature vector

Reimplemented in CStreamingVwCacheFile, and CStreamingVwFile.

Definition at line 182 of file StreamingFile.cpp.

virtual void get_vector	(	uint64_t *&	vector,
		int32_t &	len
	)			`[virtual]`

virtual void get_vector_and_label	(	int16_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	float32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	int32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	bool *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	uint32_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	uint16_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	float64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	char *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	floatmax_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

void get_vector_and_label	(	VwExample *&	ex,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

Function to read VW examples with labels

Parameters:

	ex	example
	len	length of feature vector
	label	label

Reimplemented in CStreamingVwCacheFile, and CStreamingVwFile.

Definition at line 187 of file StreamingFile.cpp.

virtual void get_vector_and_label	(	uint8_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	int8_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	uint64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual void get_vector_and_label	(	int64_t *&	vector,
		int32_t &	len,
		float64_t &	label
	)			`[virtual]`

virtual bool is_seekable ( ) [virtual]

Whether the stream is seekable/resettable

Returns:: false by default, unless overloaded

Reimplemented in CStreamingVwCacheFile, and CStreamingVwFile.

Definition at line 67 of file StreamingFile.h.

virtual void reset_stream ( ) [virtual]

Reset the stream, should be overloaded if possible

Reimplemented in CStreamingFileFromSimpleFeatures< T >, CStreamingFileFromSparseFeatures< T >, CStreamingFileFromStringFeatures< T >, and CStreamingVwCacheFile.

Definition at line 72 of file StreamingFile.h.

Member Data Documentation

CIOBuffer* buf [protected]

Buffer to hold stuff in memory.

Definition at line 294 of file StreamingFile.h.

char* filename [protected]

Name of the handled file.

Definition at line 298 of file StreamingFile.h.

char task [protected]

Task.

Definition at line 296 of file StreamingFile.h.

The documentation for this class was generated from the following files:

CStreamingFile Class Reference

Detailed Description

Public Member Functions

Protected Attributes

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation