Data Management

The DManager project propose classes and functions for managing the data. More...

Namespaces
namespace	STK::Import
	The import namespace contains the constants and global functions used in the DManager project when importing data in STK containers.

Classes
class	STK::KernelHandler
	implementation of the DataHandlerBase class for kernel mixture models. More...
class	STK::CellBase< Type >
	template Base class for the Cell in a list . More...
class	STK::CellVe< Type >
	template class for the Vertical Cell of a Vertical List. More...
class	STK::CellHo< Type >
	template class for the Horizontal Cell of a Horizontal List. More...
class	STK::Cell2D< Type >
	template class for the 2 Dimensional Cells. More...
class	STK::CvHandler
	CvHanler is an utility function for building the submatrix/subvectors needed when using k-folds cross-validation. More...
class	STK::DataFrame
	DataFrame is a List of Variable with the same number of rows. More...
class	STK::DataHandler
	implementation of the DataHandlerBase class using ReadWriteCsv and Array2D. More...
class	STK::DataHandlerBase< Derived >
	A class derived from a DataHandlerBase allows to store various data sets identified by an idData and an idModel. More...
class	STK::ExportToCsv
	Export data to a Csv. More...
class	STK::IDataBridge
	Interface class wrapping a data set. More...
class	STK::IDataHandler
	A DataHandler class allow to store various data set identified with an idData (a string). More...
class	STK::ImportFromCsv
	import data from a Csv File in a DataFrame. More...
class	STK::IPage
	A IPage is an interface base class for reading and/or writing a page of option in a file. More...
class	STK::IVariable
	IVariable is an Interface class for all Variables classes. More...
class	STK::List1D< Type_ >
	template One dimensional Horizontal List. More...
class	STK::Option
	the Option class allow to store the value of an option from a IPage. More...
class	STK::PartitionHandler
	CvHanler is an utility function for building the submatrix/subvectors needed when creating learning and test data sets. More...
class	STK::TReadWriteCsv< Type >
	the TReadWriteCsv class : allow to write and/or to read a csv file. More...
class	STK::ReadWritePages
	The ReadWritePages class allow to handle a file of option containing pages. More...
class	STK::Variable< Type_ >
	Variable is an implementation of the Base class IVariable using The Array1D class for storing the data. More...

Enumerations
enum	STK::DManager::TypeDataFile { STK::DManager::unknown_ =0 , STK::DManager::csv_ }
	type of the data file we handle. More...
enum	STK::Import::TypeImport {   STK::Import::unknown_ =0 , STK::Import::numeric_ , STK::Import::only_numeric_ , STK::Import::string_ ,   STK::Import::directed_ , STK::Import::intelligent_ }
	define the type of import we want to perform. More...
enum	STK::Option::TypeOption {   STK::Option::unknown_ =0 , STK::Option::string_ , STK::Option::real_ , STK::Option::integer_ ,   STK::Option::range_ , STK::Option::lstring_ , STK::Option::lreal_ , STK::Option::linteger_ ,   STK::Option::lrange_ , STK::Option::page_ }
	A TypeOption is the kind of options that can read or write a program in a file. More...
enum	STK::Csv::readflags { STK::Csv::RW_APPEND_DATA = 0x00000004 , STK::Csv::RW_REPLACE_DATA = 0x00000008 }
	TODO: These flags determine the behavior of the reading methods. More...

Functions
TypeDataFile	STK::DManager::stringToTypeDataFile (String const &type)
	convert a String to a TypeDataFile.
String	STK::DManager::TypeDataFileToString (TypeDataFile const &type)
	convert a TypeDataFile to a String.
bool	STK::DManager::checkStringToBoolean (String const &str)
	check if a string represent a boolean.
bool	STK::DManager::StringToBoolean (String const &str)
	convert a string to a boolean.
void	STK::DManager::removeCharBeforeAndAfter (String &str, Char c)
	remove all occurrences of the char c at the beginning and the end of the string str.
istream &	STK::DManager::getField (istream &is, String &value, Char delimiter)
	Get the current field from the input stream.
template<class TYPE >
void	STK::DManager::readList (String const &strBuffer, std::list< TYPE > &lst, Char sep=CHAR_SEP)
	Read a list of value of type TYPE stored in a line.
template<class TYPE >
void	STK::DManager::writeList (ostream &os, std::list< TYPE > const &lst, Char sep=CHAR_SEP)
	Write a list of value of type TYPE stored in a line.
int	STK::DManager::nbEndOfLine (istream &is)
template<class Vector >
void	STK::heapSort (Vector &T)
	Sort the container T in ascending order.
template<class Vector >
void	STK::heapSort (Vector const &T, Vector &Tsort)
	Sort the container T in ascending order and return the result in the container Tsort.
template<class Vector , class VectorInt >
void	STK::heapSort (VectorInt &I, Vector const &T)
	Sort the container T in ascending order using index array.
template<class Vector , class VectorInt >
void	STK::applySort1D (Vector &T, VectorInt const &I)
	Apply a sorting index array to the 1D container T.
template<class Array , class VectorInt >
void	STK::applySort2D (Array &T, VectorInt const &I)
	Apply a sorting index array to the 2D container T row by row.
TypeImport	STK::Import::stringToTypeImport (String const &type)
	Convert a String to a TypeImport.

Detailed Description

The DManager project propose classes and functions for managing the data.

The DManager project contains all the class related to data management. It provides

• an abstract base class for statistical variables,
• a template implementation for arbitrary data,
• a Dataframe (Table) class,
• classes for read and write csv and (TODO) dbf files,
• various classes for importing/exporting and converting data from different containers in different type,
• methods for sorting one dimensional containers and two-dimensionnal containers using one column as sorting key,
• classes for reading/writing options file.

Enumeration Type Documentation

readflags

enum STK::Csv::readflags

TODO: These flags determine the behavior of the reading methods.

Enumerator
RW_APPEND_DATA
RW_REPLACE_DATA

Definition at line 87 of file STK_ReadWriteCsv.h.

  {
    RW_APPEND_DATA    = 0x00000004,
    RW_REPLACE_DATA   = 0x00000008
  };

TypeDataFile

enum STK::DManager::TypeDataFile

type of the data file we handle.

Enumerator
unknown_	unknown reduction
csv_

Definition at line 82 of file STK_DManager_Util.h.

{
  unknown_ =0
  , csv_
};

TypeImport

enum STK::Import::TypeImport

define the type of import we want to perform.

Enumerator
unknown_	try to convert the columns in numeric values and let the others as string
numeric_	try to convert the columns in numeric values and let the others as string
only_numeric_	conserve only the columns with numeric values
string_	copy the columns
directed_	convert using an array of conversion.
intelligent_	try to convert with the most appropriate format the columns. NOT implemented

Definition at line 51 of file STK_Import_Util.h.

  {
    unknown_ =0,
    numeric_,
    only_numeric_,
    string_,
    directed_,
    intelligent_
  };

TypeOption

enum STK::Option::TypeOption

A TypeOption is the kind of options that can read or write a program in a file.

It can be:

• a String (e.g. toto)
• a Real (e.g. 3.14159)
• an int (e.g. 101)
• a Range of number (e.g. 2:10)
• a list of String
• a list of Real
• a list of int
• a list of Range
• a sub-page

Enumerator
unknown_
string_	characters
real_	floating
integer_	discrete number
range_	range of number
lstring_	list of strings separated by commas
lreal_	list of floating numbers separated by commas
linteger_	list of discrete numbers separated by commas
lrange_	list of range of number separated by commas
page_	a page option

Definition at line 71 of file STK_Option.h.

    {
      unknown_ =0,
      string_,      
      real_,        
      integer_,     
      range_,       
      lstring_,     
      lreal_,       
      linteger_,    
      lrange_,       
      page_         
    };

Function Documentation

applySort1D()

template<class Vector , class VectorInt >

void STK::applySort1D	(	Vector &	T,
		VectorInt const &	I
	)

Apply a sorting index array to the 1D container T.

Parameters

I	the index array sorting T
T	the container to sort

Definition at line 287 of file STK_HeapSort.h.

{
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(Vector);
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(VectorInt);
#ifdef STK_BOUNDS_CHECK
  if (I.range() != T.range())
  { STKRUNTIME_ERROR_2ARG(applySort1D,I.range(),T.range(),incompatible lengths);}
#endif
  Vector A(T.range());
  for (int i=I.begin(); i< I.end(); i++) { A[i] = T[I[i]];}
  T.move(A);
}

References STK::applySort1D(), STK::IArray2D< Derived >::move(), STK_STATIC_ASSERT_ONE_DIMENSION_ONLY, and STKRUNTIME_ERROR_2ARG.

Referenced by STK::applySort1D(), STK::applySort2D(), and STK::Stat::Univariate< TContainer1D, Real >::compOrderStatistics().

applySort2D()

template<class Array , class VectorInt >

void STK::applySort2D	(	Array &	T,
		VectorInt const &	I
	)

Apply a sorting index array to the 2D container T row by row.

Parameters

I	the index array sorting T
T	the container to sort

Definition at line 306 of file STK_HeapSort.h.

{
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(VectorInt);
#ifdef STK_BOUNDS_CHECK
  if (I.range() != T.rows())
  { STKRUNTIME_ERROR_2ARG(applySort1D,I.range(),T.rows(),incompatible lengths);}
#endif
  Array A(T.rows(), T.cols());
  for (int i=I.begin(); i< I.end(); i++) { A.row(i) = T.row(I[i]);}
  T.move(A);
}

References STK::applySort1D(), STK_STATIC_ASSERT_ONE_DIMENSION_ONLY, and STKRUNTIME_ERROR_2ARG.

checkStringToBoolean()

bool STK::DManager::checkStringToBoolean

(

String const &

str

)

check if a string represent a boolean.

A String is a boolean if it is written "TRUE" or "FALSE". There is no need to use upper case.

Parameters

str	the string to check

Returns

true if the String i a boolean, false otherwise.

Definition at line 78 of file STK_DManager_Util.cpp.

{
  // is it TRUE ?
  if (toUpperString(str) == _T("TRUE")) { return true;}
  // is it FALSE ?
  if (toUpperString(str) == _T("FALSE")) { return true;}
  // not a bolean string
  return false;
}

References _T, and STK::toUpperString().

getField()

istream & STK::DManager::getField	(	istream &	is,
		String &	value,
		Char	delimiter
	)

Get the current field from the input stream.

A field is between the current position and a delimiter or an end of line in the stream. All blank spaces and tabulations before and after the field are removed.

Parameters

is	the stream to treat
value	the value of the field
delimiter	the delimiter of the current field

Returns

the input stream without the value extracted

Definition at line 124 of file STK_DManager_Util.cpp.

{
  std::getline( is, value, delimiter);
  removeCharBeforeAndAfter(value, CHAR_BLANK);
  removeCharBeforeAndAfter(value, CHAR_TAB);
  return is;
}

References STK::DManager::removeCharBeforeAndAfter().

Referenced by STK::TReadWriteCsv< Type >::read(), STK::DManager::readList(), and STK::TReadWriteCsv< Type >::readNames().

heapSort() [1/3]

template<class Vector >

void STK::heapSort

(

Vector &

T

)

Sort the container T in ascending order.

Parameters

T	the container to sort

Definition at line 50 of file STK_HeapSort.h.

{
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(Vector);
  typedef typename hidden::Traits<Vector>::Type Type;
  // number of elements
  const int nb_elt = T.size();
  if (nb_elt < 2) return;
 
  // if the container is base one, shift0 = 0 and shift1 = 1
  int shift1 = T.begin(), shift0 = T.begin() - 1;
 
  // create heap
  for (int first = nb_elt/2; first > 0; -- first)
  {
    // the value value to insert in the heap
    Type value = T[shift0 + first];
    // organize the heap
    int i=first, j=2*first;
    while (j <= nb_elt)
    {
      // j+1 is greatest child
      if ( j < nb_elt && T[shift0 + j] < T[shift1 + j] ) j++;
      // we have find a child gt value
      if (value >= T[shift0 + j]) break;
      // else shift the inner value
      T[shift0 + i] = T[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    T[shift0 + i] = value;
  }
#ifdef STK_HEAPSORT_DEBUG
  std::cout << "T=\n" << T.asDerived() << _T("\n";);
#endif
  // sort T
  for (int last = nb_elt;;)
  { // the value to sort
    Type value = T[shift0 + last];
    // Put the top of the heap at the end
    T[shift0 + last] = T[shift1];
    // decrease last.  last==1 : we end the job
    if (--last == 1)
    { T[shift1] = value;
      break;
    }
    // organize the heap
    int i=1, j=2;
    while (j <= last)
    { // j+1 is greatest child
      if ( j < last && T[shift0 + j] < T[shift1 + j] ) j++;
      // we have find a child gt value
      if (value >= T[shift0 + j]) break;
      // else shift the inner value
      T[shift0 + i] = T[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    T[shift0 + i] = value;
  }
}

References _T, and STK_STATIC_ASSERT_ONE_DIMENSION_ONLY.

Referenced by STK::Stat::Univariate< TContainer1D, Real >::compOrderStatistics().

heapSort() [2/3]

template<class Vector >

void STK::heapSort	(	Vector const &	T,
		Vector &	Tsort
	)

Sort the container T in ascending order and return the result in the container Tsort.

Parameters

T	the container to sort
Tsort	the container with the result

Definition at line 122 of file STK_HeapSort.h.

{
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(Vector);
  typedef typename Vector::Type Type;
  // copy T in Tsort
  Tsort = T.asDerived();
  // number of elements
  const int nb_elt = Tsort.size();
  if (nb_elt < 2) return;
 
  // if the container is base one, shift0 = 0 and shift1 = 1
  int shift1 = Tsort.begin(), shift0 = Tsort.begin() - 1;
 
  // create heap
  for (int first = nb_elt/2; first > 0; -- first)
  {
    // the value value to insert in the heap
    Type value = Tsort[shift0 + first];
    // organize the heap
    int i=first, j=2*first;
    while (j <= nb_elt)
    {
      // j+1 is greatest child
      if ( j < nb_elt && Tsort[shift0 + j] < Tsort[shift1 + j] ) j++;
      // we have find a child gt value
      if (value >= Tsort[shift0 + j]) break;
      // else shift the inner value
      Tsort[shift0 + i] = Tsort[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    Tsort[shift0 + i] = value;
  }
#ifdef STK_HEAPSORT_DEBUG
  std::cout << "T=\n" << Tsort << _T("\n";);
#endif
  // sort T
  for (int last = nb_elt;;)
  { // the value to sort
    Type value = Tsort[shift0 + last];
    // Put the top of the heap at the end
    Tsort[shift0 + last] = Tsort[shift1];
    // decrease last.  last==1 : we end the job
    if (--last == 1)
    { Tsort[shift1] = value;
      break;
    }
    // organize the heap
    int i=1, j=2;
    while (j <= last)
    { // j+1 is greatest child
      if ( j < last && Tsort[shift0 + j] < Tsort[shift1 + j] ) j++;
      // we have find a child gt value
      if (value >= Tsort[shift0 + j]) break;
      // else shift the inner value
      Tsort[shift0 + i] = Tsort[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    Tsort[shift0 + i] = value;
  }
}

References _T, and STK_STATIC_ASSERT_ONE_DIMENSION_ONLY.

heapSort() [3/3]

template<class Vector , class VectorInt >

void STK::heapSort	(	VectorInt &	I,
		Vector const &	T
	)

Sort the container T in ascending order using index array.

T is not modified, I contain the indices of the elements of T in ascending order.

Parameters

I	the index array sorting T
T	the container to sort

Definition at line 197 of file STK_HeapSort.h.

{
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(Vector);
  STK_STATIC_ASSERT_ONE_DIMENSION_ONLY(VectorInt);
  typedef typename hidden::Traits<Vector>::Type Type;
 
  // number of elements
  int nb_elt = T.size();
 
  // create index array
  I.asDerived().resize(T.range());
  int first = I.begin(), last = I.lastIdx();
  for (int i=first; i<=last; i++)
  { I[i] = i;}
 
  if (nb_elt < 2) return;
 
  // if the container is base one, shift0 = 0 and shift1 = 1
  int shift1 = T.begin(), shift0 = T.begin() - 1;
 
  // create heap
  for (first = nb_elt/2; first > 0; --first)
  {
    // the value value to insert in the heap
    Type value = T[I[shift0 + first]];
    // organize the heap
    int i=first, j=2*first;
    while (j <= nb_elt)
    {
      // j+1 is greatest child
      if ( j < nb_elt && T[I[shift0 + j]] < T[I[shift1 + j]] ) j++;
      // we have find a child lt value
      if (value >= T[I[shift0 + j]]) break;
      // else shift the inner values
      I[shift0 + i] = I[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    I[shift0 + i] = shift0 + first;
  }
#ifdef STK_HEAPSORT_DEBUG
  std::cout << "I=\n" << I <<"\n";
#endif
  // sort T
  for (int last = nb_elt;;)
  {
    // the value to sort
    int ivalue = I[shift0 + last];
    Type value = T[ivalue];
    // Put the top of the heap at the end
    //T[shift0 + last] = T[shift1];
    I[shift0 + last] = I[shift1];
    // decrease last.  last==1 : we end the job
    if (--last == 1)
    { //T[shift1] = value;
      I[shift1] = ivalue;
      break;
    }
    // organize the heap
    int i=1, j=2;
    while (j <= last)
    { // j+1 is greatest child
      if ( j < last && T[I[shift0 + j]] < T[I[shift1 + j]] ) j++;
      // we have find a child gt value
      if (value >= T[I[shift0 + j]]) break;
      // else shift the inner value
      // T[shift0 + i] = T[shift0 + j];
      I[shift0 + i] = I[shift0 + j];
      // go down in the tree
      i = j;
      j*= 2;
    }
    // plug value in its final location
    // T[shift0 + i] = value;
    I[shift0 + i] = ivalue;
  }
#ifdef STK_HEAPSORT_DEBUG
  std::cout << "I=\n" << I <<"\n";
#endif
}

References STK_STATIC_ASSERT_ONE_DIMENSION_ONLY.

nbEndOfLine()

int STK::DManager::nbEndOfLine

(

istream &

is

)

Returns

the number of line in an istream

Parameters

is	the stream to parse.

Definition at line 136 of file STK_DManager_Util.cpp.

{
  char last = CHAR_BLANK;
  TestEndOfLineOp       test(&last);
  int nbLine = std::count_if( std::istreambuf_iterator<Char>( is )
                            , std::istreambuf_iterator<Char>(),
                              test
                            );
  if (last != CHAR_NL) nbLine++;
#ifdef STK_DEBUG
  stk_cout << _T("In nbEndOfLine, nbLine = ") << nbLine << _T("\n");
#endif
 return  nbLine;
}

References _T, and stk_cout.

Referenced by STK::TReadWriteCsv< Type >::initRead().

readList()

template<class TYPE >

void STK::DManager::readList	(	String const &	strBuffer,
		std::list< TYPE > &	lst,
		Char	sep = CHAR_SEP
	)

Read a list of value of type TYPE stored in a line.

Parameters

strBuffer	the string with the list of value
lst	the resulting list
sep	the separator character

Definition at line 153 of file STK_DManager_Util.h.

{
  // Declare an input string stream
  istringstream instream;
  // Use strBuffer as source of input.
  instream.str(strBuffer);
  // read the line
  do
  {
    // get field
    String strbuff;
    getField(instream, strbuff, sep);
    // check if it is a blank field
    if (strbuff.empty()) {  break;}
    // append Data to the list
    TYPE value;
    if (stringToType(value, strbuff))
      lst.push_back(value);
    // TODO: else emit warning or Exception
  }
  while(1);
}

References STK::DManager::getField(), and STK::stringToType().

Referenced by STK::Option::setValue().

removeCharBeforeAndAfter()

void STK::DManager::removeCharBeforeAndAfter	(	String &	str,
		Char	c
	)

remove all occurrences of the char c at the beginning and the end of the string str.

Parameters

str	the string to treat
c	the character to remove before and after

Definition at line 108 of file STK_DManager_Util.cpp.

{
  // erase first whitespaces
  str.erase(0, str.find_first_not_of(c));
  // erase remaining whitespaces
  size_t found =str.find_last_not_of(c);
  if (found != str.npos)
    str.erase(found+1);
  else
    str.clear(); // str is all whitespace
}

Referenced by STK::IPage::findKeyword(), STK::DManager::getField(), STK::IPage::processLine(), STK::IPage::read(), and STK::TReadWriteCsv< Type >::readCurrentLine().

StringToBoolean()

bool STK::DManager::StringToBoolean

(

String const &

str

)

convert a string to a boolean.

A String is a boolean if it is written "TRUE" or "FALSE". There is no need to use upper case.

Parameters

str	the string to convert

Returns

true if the String is "TRUE, false otherwise.

Definition at line 96 of file STK_DManager_Util.cpp.

{
  // is it TRUE ?
  if (toUpperString(str) == _T("TRUE")) { return true;}
  // if it's not true, it's false
  return false;
}

References _T, and STK::toUpperString().

stringToTypeDataFile()

TypeDataFile STK::DManager::stringToTypeDataFile

(

String const &

type

)

convert a String to a TypeDataFile.

Parameters

type	the String we want to convert

Returns

the TypeDataFile represented by the String type. if the string does not match any known name, the unknown_ type is returned.

Definition at line 52 of file STK_DManager_Util.cpp.

{
  if (toUpperString(type) == toUpperString(_T("csv")))  return csv_;
  return unknown_;
}

References _T, STK::DManager::csv_, STK::toUpperString(), and STK::DManager::unknown_.

stringToTypeImport()

TypeImport STK::Import::stringToTypeImport

(

String const &

type

)

Convert a String to a TypeImport.

Parameters

type	the String we want to convert

Returns

the TypeImport represented by the String type. if the string does not match any known name, the unknown_ type is returned.

Definition at line 50 of file STK_Import_Util.cpp.

{
  if (toUpperString(type) == toUpperString(_T("unknown"))) return unknown_;
  if (toUpperString(type) == toUpperString(_T("numeric")))  return numeric_;
  if (toUpperString(type) == toUpperString(_T("only_numeric"))) return only_numeric_;
  if (toUpperString(type) == toUpperString(_T("string"))) return string_;
  if (toUpperString(type) == toUpperString(_T("directed"))) return directed_;
  if (toUpperString(type) == toUpperString(_T("intelligent"))) return intelligent_;
  return unknown_;
}

References _T, STK::Import::directed_, STK::Import::intelligent_, STK::Import::numeric_, STK::Import::only_numeric_, STK::Import::string_, STK::toUpperString(), and STK::Import::unknown_.

TypeDataFileToString()

String STK::DManager::TypeDataFileToString

(

TypeDataFile const &

type

)

convert a TypeDataFile to a String.

Parameters

type	the type of data file we want to convert

Returns

the string associated to this type.

Definition at line 63 of file STK_DManager_Util.cpp.

{
  if (type == csv_)  return String(_T("csv"));
  return String(_T("unknown"));
}

References _T, and STK::DManager::csv_.

writeList()

template<class TYPE >

void STK::DManager::writeList	(	ostream &	os,
		std::list< TYPE > const &	lst,
		Char	sep = CHAR_SEP
	)

Write a list of value of type TYPE stored in a line.

Parameters

os	the output stream
lst	the list to write
sep	the separator character

Definition at line 183 of file STK_DManager_Util.h.

{
  if (lst.empty()) return;
  typename std::list<TYPE>::const_iterator it = lst.begin();
  os << *it;
  it++;
  for ( ; it != lst.end(); it++)
  { os << sep << CHAR_BLANK << *it;}
}

Referenced by STK::Option::write().