hicsv package

Module contents

I/O interface for hicsv (header-included comma-separated values) format.

Header-included comma-separated values (hicsv or hi-csv in short) is a text file format consisting of a header (metadata) and a tabulated values, both in a human-readable and editable form.

CSV is a great universal format that can store a table of values in a human-readable way. Many instruments and softwares can export data in CSV. We can then avoid dealing with the proprietary binary formats whose specifications are often not disclosed to consumers.

However, there is no commonly accepted method to store metadata along with a table of values. Metadata such as experimental conditions, author, etc. are often essential in science. People can easily lose track of these metadata when the CSV file only contains the table part.

There are some formats or conventions that (try to) treat a table and metadata nicely (e.g. Tabular Data Package https://specs.frictionlessdata.io/tabular-data-package/#introduction). But, to the best of the author’s knowledge, there is no text-based generic format that combines a table and metadata in a single file.

It is very important to combine the metadata and the table because it would be difficult to force everyone to always keep two separate files together.

Binary formats such as HDF5 can store metadata in addition to image or array data. But again, there is no text-based, human-readable format with embedded metadata.

hicsv format has been developed as a solution to the above problem. It is basically just a CSV table with a JSON header part attached at the top. The rough, tentative specification of hicsv is as follows.

  1. It should be a Unicode text file with the newline character \n.

  2. It starts with consequtive lines starting with #.

  3. The last one of the #-capped lines represents the “keys” of each column in the table.

  4. The rest of the #-capped lines are an indented JSON which represents the dictionary of metadata.

  5. The lines not starting with # are the CSV table. The CSV part follows the generally accepted conventions (mostly RFC 4180 but may not be strict) except for following additional restrictions.

  6. The columns in the CSV table should always be of the same length.

  7. Each column must consist of only one type.

The hicsv package is a minimal implementation of the I/O interface to the hicsv-formatted files. It, therefore, does not have (or aim for) any advanced functionalities. Please use numpy and/or other useful packages for anything beyond I/O.

Example

To read a hicsv-formatted text file,

>>> import hicsv
>>> d = hicsv.hicsv("foo.txt")

Or you can just create an empty object to which you can add columns and header info.

>>> e = hicsv.hicsv()

You can get one or more columns.

>>> c1, c2 = d.ga("column 1", "column 2")

The header info is stored as a dictionary. You can set and get information at any time:

>>> d.h["new header entry"] = "some value"
>>> print(d.h["new header entry"])

If you need to create a hicsv-formatted file from scratch:

>>> out = hicsv()

To add a column:

>>> out.append_column("column 1", arr)

Here arr should be a 1-d numpy array. Note that all columns must be of the same length, so adding an array with different length will cause error. Then you can write into a hicsv-formatted text file like:

>>> out.save("new hicsv file.txt", encoding="utf-8")

Note

Some users may find the following behavior of this package unexpected.

  1. The parser deletes the spaces at the beginning and at the end of each cell. This is true even if the cell content is double-quoted. For example, the two cells

    “spam”, “ spam “

    results in identical strings.

class hicsv.hicsv(fp: str | IO = '', **kwargs: Any)[source]

Bases: object

Container/writer class for a hicsv file.

Parameters:

  • fp – File-like object or path to the file. If an empty string, an empty object will be created. If a string or a file-like object is given, the file will be loaded as the object. Defaults to "" (empty string) which creates an empty object.

  • **kwargs – Arguments of built-in function open(). (New in 1.0.0.)

keys

column keys. a list of strings.

cols

columns. a list of 1-d numpy arrays.

h

header.

Example

>>> import numpy as np
>>> import hicsv
>>> d = hicsv.hicsv() # create empty object
>>> d.h["a new header entry"] = 1    # add an entry to the header
>>> print(d.h["a new header entry"]) # access the header entry
1
>>> d.append_column("the first column", np.array([0.0, 1.0, 2.0]))  # add column
>>> d.append_column("the second column", np.array(["a", "b", "c"])) # add another column
>>> print(d.ga("the first column")) # access the column
[0. 1. 2.]
>>> print(d.ga("the first column", "the second column")) # access two columns at once
[array([0., 1., 2.]), array(['a', 'b', 'c'], dtype='<U1')]
>>> d.save("spam.txt") # save to a text file
>>> d2 = hicsv.hicsv("spam.txt") # load from a file
append_column(key: str, arr: ndarray) → None[source]

Appends a single column to the table.

Parameters:

  • key – Identifier for the new column. Keys already in use are not allowed.

  • arr –

    A 1-d numpy array of the new column.

    If it’s the first column to be inserted, it can be of any length. However, if there is previously inserted column(s), the new column must be of the same length as the already-existing columns.

    The dtype of the column must be the sub-type of numpy.integer, numpy.floating, or numpy.str_.

Raises:

  • ValueError – If the array dimension is not 1 or if the array length does not match that of the already existing column(s).

  • KeyError – If the key is already in use.

  • TypeError – If dtype of arr is not the subtype of numpy.integer, numpy.floating, or numpy.str_.

Example

>>> d = hicsv.hicsv()
>>> d.append_column("first column", np.arange(5))
>>> d.append_column("second column", np.linspace(0.0, 1.0, 5))
>>> d.append_column("third column", np.array(["one", "two", "three", "four", "five"]))
>>> d.append_column("fourth column", np.array([0.0])) # this will fail due to length mismatch
fromfile(fp: str | IO, **kwargs: Any) → None[source]

Loads a hicsv file and updates the stored headers & columns.

Parameters:

  • fp – file-like object or path to the file.

  • **kwargs – Arguments of built-in open(). Used only when fp is a string. The mode, encoding, and newline parameters are overwritten by “r”, “utf-8”, and “n” by default. New in 1.1.0.

Note

It overwrites the header h and appends the loaded columns to the exsting cols by using append_column(). Therefore, if append_column() fails for any of the new columns, this will not work.

ga(*cols: int | str) → ndarray | List[ndarray][source]

Alias for get_columns_as_arrays().

Parameters:

*cols – column names or indices. str and int can be mixed.

Returns:

if one column name/index is given, a 1-d numpy array. if two or more names/indices are given, a list of 1-d numpy arrays.

Example

>>> a1 = d.ga("column 1")
>>> a2, a3 = d.ga("column two", "third column")
get_columns_as_arrays(*cols: int | str) → ndarray | List[ndarray][source]

Gets a single column or multiple columns by column names or column indices.

Parameters:

*cols – column names or indices. str and int can be mixed.

Returns:

if one column name/index is given, a 1-d numpy array. if two or more names/indices are given, a list of 1-d numpy arrays.

Example

>>> a1 = d.get_columns_as_arrays("column 1")
>>> a2, a3 = d.get_columns_as_arrays("column two", "third column")
insert_column(pos: int, key: str, arr: ndarray) → None[source]

Inserts a single column into the table.

Parameters:

  • pos – Position of the new column.

  • key – Identifier for the new column. Keys already in use are not allowed.

  • arr –

    A 1-d numpy array of the new column.

    If it’s the first column to be inserted, it can be of any length. However, if there is previously inserted column(s), the new column must be of the same length as the already-existing columns.

    The dtype of the column must be the sub-type of numpy.integer, numpy.floating, or numpy.str_.

Raises:

  • ValueError – If the array dimension is not 1 or if the array length does not match that of the already existing column(s).

  • KeyError – If the key is already in use.

  • TypeError – If dtype of arr is not the subtype of numpy.integer, numpy.floating, or numpy.str_.

remove_column(col: int | str) → None[source]

Removes a column from the object.

Parameters:

col – column name or index.

replace_column(col: int | str, arr: ndarray) → None[source]

Replaces the content of an existing column.

Parameters:

  • col – column name or index.

  • arr –

    A 1-d numpy array of the new column.

    It must be of the same length as the already-existing columns.

    The dtype of the column must be the sub-type of numpy.integer, numpy.floating, or numpy.str_.

Example

>>> d = hicsv.hicsv()
>>> d.append_column("first column", np.arange(5))
>>> d.append_column("second column", np.linspace(0.0, 1.0, 5))
>>> print(d.ga("first column"))
[0 1 2 3 4]
>>> d.replace_column("first column", np.array(["one", "two", "three", "four", "five"]))
>>> print(d.ga("first column"))
['one' 'two' 'three' 'four' 'five']
save(fp: str | IO, prettify: bool = True, add_version_info: bool = True, **kwargs: Any) → None[source]

Saves the object content to a hicsv-formatted text file.

Parameters:

  • fp – file-like object of the destination.

  • prettify – True to turn on pretty formatting.

  • add_version_info – If True, automatically adds the current versions of the module and the hicsv format specification.

  • **kwargs – Arguments of built-in open(). Used only when fp is a string. The mode, encoding, and newline parameters are overwritten by “w”, “utf-8”, and “n” by default. New in 1.0.0.

Example

>>> d = hicsv.hicsv()
>>> d.append_column("first column", np.arange(5))
>>> d.append_column("second column", np.linspace(0.0, 1.0, 5))
>>> d.append_column("third column", np.array(["one", "two", "three", "four", "five"]))
>>> d.save("spam.txt")

Or,

>>> with open("spam.txt", "w") as fp:
>>>     d.save(fp)
hicsv.txt2hicsv(fp: IO | str, sep: str = ',', ignore_lines: List[int] = [], key_line: int | None = None, keys: List[str] = [], **kwargs: Any) → hicsv[source]

Reads a generic delimited text file and converts it into a hicsv object.

This is a convenient function to import simple text data file into a hicsv object. It is also an example of the usage of hicsv class.

Parameters:

  • fp – string or file-like object of the input file.

  • sep – Delimiter character of the input file. Defaults to comma.

  • ignore_lines – List of line numbers to ignore. Defaults to empty list

  • key_line – Line number of the column key. If None, keys will be automatically named as “0”, “1”, “2”, etc.

  • keys – List of column keys. It’s useful if you know the table structure exactly. This must match the number of columns in the input file. When keys is specified, key_line will be ignored.

  • **kwargs – Arguments of built-in open(). Used only when fp is a string. (New in 1.0.0.)