csvkit 0.5.0 (beta)

About

csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats.

It is inspired by pdftk, gdal and the original csvcut utility by Joe Germuska and Aaron Bycoffe.

Important links:

Principles

csvkit is to tabular data what the standard Unix text processing suite (grep, sed, cut, sort) is to text. As such, csvkit adheres to the Unix philosophy.

  1. Small is beautiful.
  2. Make each program do one thing well.
  3. Build a prototype as soon as possible.
  4. Choose portability over efficiency.
  5. Store data in flat text files.
  6. Use software leverage to your advantage.
  7. Use shell scripts to increase leverage and portability.
  8. Avoid captive user interfaces.
  9. Make every program a filter.

As there is no formally defined CSV format, csvkit encourages well-known formatting standards:

  • Output favors compatability with the widest range of applications. This means that quoting is done with double-quotes and only when necessary, columns are separated with commas, and lines are terminated with unix style line endings (“\n”).
  • Data that is modified or generated will prefer consistency over brevity. Floats always include at least one decimal place, even if they are round. Dates and times are written in ISO8601 format.

Installation

For users:

pip install csvkit

For developers:

git clone git://github.com/onyxfish/csvkit.git
cd csvkit
mkvirtualenv --no-site-packages csvkit
pip install -r requirements.txt
nosetests

Note

csvkit is routinely tested on OSX, somewhat less frequently on Linux and once in a while on Windows. All platforms are supported. It is tested against Python 2.6, 2.7 and PyPy. Neither Python < 2.6 nor Python >= 3.0 are supported at this time.

Usage

csvkit is comprised of a number of individual command line utilities that be loosely divided into a few major categories: Input, Processing, and Output. Documentation and examples for each utility are described on the following pages.

Input

Processing

Output (and Analysis)

Appendices

Development

csvkit is designed to augment or supercede much of Python’s csv module. Important parts of the API are documented here:

Contributing

Want to hack on csvkit? Here’s how:

Authors

The following individuals have contributed code to csvkit:

  • Christopher Groskopf
  • Joe Germuska
  • Aaron Bycoffe
  • Travis Mehlinger
  • Alejandro Companioni
  • Benjamin Wilson
  • Bryan Silverthorn
  • Evan Wheeler
  • Matt Bone
  • Ryan Pitts
  • Hari Dara
  • Jeff Larson
  • Jim Thaxton
  • Miguel Gonzalez
  • Anton Ian Sipos
  • Gregory Temchenko
  • Kevin Schaul
  • Marc Abramowitz
  • Noah Hoffman
  • Jan Schulz
  • Derek Wilson
  • Chris Rosenthal
  • Davide Setti
  • Gabi Davar
  • Sriram Karra
  • James McKinney
  • Krzysztof Dorosz

License

The MIT License

Copyright (c) 2013 Christopher Groskopf and contributers

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Changelog

0.5.1

  • Added Chris Rosenthal to AUTHORS.
  • Fixed multi-file input to csvsql. (#193)
  • Added csvpys command (python scripting). (#214, #215)
  • Added csvgroup command (equivalent to sql group by). (#215)
  • Added Krzysztof Dorosz to AUTHORS.

0.5.0

  • Implement geojson support in csvjson. (#159)
  • Optimize writing of eight bit codecs. (#175)
  • Created csvpy. (#44)
  • Support –not-columns for excluding columns. (#137)
  • Add Jan Schulz to AUTHORS file.
  • Add Windows scripts. (#111, #176)
  • csvjoin, csvsql and csvstack will no longer hold open all files. (#178)
  • Added Noah Hoffman to AUTHORS.
  • Make csvlook output compatible with emacs table markup. (#174)

0.4.4

  • Add Derek Wilson to AUTHORS.
  • Add Kevin Schaul to AUTHORS.
  • Add DBF support to in2csv. (#11, #160)
  • Support –zero option for zero-based column indexing. (#144)
  • Support mixing nulls and blanks in string columns.
  • Add –blanks option to csvsql. (#149)
  • Add multi-file (glob) support to csvsql. (#146)
  • Add Gregory Temchenko to AUTHORS.
  • Add –no-create option to csvsql. (#148)
  • Add Anton Ian Sipos to AUTHORS.
  • Fix broken pipe errors. (#150)

0.4.3

  • Begin CHANGELOG (a bit late, I’ll admit).

Indices and tables

Project Versions

Table Of Contents

Next topic

1. Getting started

This Page