vd(1)

NAME
VisiData - a terminal utility for exploring and arranging tabular data

SYNOPSIS
vd [options] [input ...]
vd [options] --play cmdlog [-w waitsecs] [--batch] [-o output] [field=value ...]

DESCRIPTION
VisiData is a multipurpose tool built on the vdtui platform that can be used to explore, clean, edit, and restructure data. Rows can be selected, filtered, and grouped; columns can be rearranged, transformed, and derived via regex or Python expressions; and workflows can be saved, documented, and replayed.

  REPLAY MODE
replay a saved cmdlog within the interface wait seconds between commands replay in batch mode (with no interface) save final visible sheet to file as .tsv
--replay-movement=bool
    toggle --play to move cursor cell-by-cell
field=value
    replace "{field}" in cmdlog contents with value

  Commands During Replay
^U  pause/resume replay
Tab execute next row in replaying sheet
^K  cancel current replay

  GLOBAL COMMANDS
In most cases, commands that affect selected rows will affect all rows if no rows are selected.

  Keystrokes for the Cautious
F1   z?
    view this man page
^Q  abort program immediately
^C  cancel user input or abort all async threads on current sheet
q   quit current sheet
gq  quit all sheets (clean exit)

  Cursor Movement
Arrow PgUp Home
    move as expected
h   j   k   l
    move left/down/up/right
gh  gj  gk  gl
    move all the way to the left/bottom/top/right of sheet
G  gg
    move all the way to the bottom/top of sheet
^B  ^F
    scroll one page back/forward

^^ (Ctrl-^)
    jump to previous sheet (swaps with current sheet)

/   ? regex
    search for regex forward/backward in current column
g/  g? regex
    search for regex forward/backward over all visible columns
n   N
    move to next/previous match from last search

<   >
    move up/down in current column to the next value
z<  z>
    move up/down in current column to the next null
{   }
    move up/down in current column to the next selected row

c regex
    move to next column with name matching regex
r regex
    move to next row with key matching regex
zc  zr number
    move to column/row number

H   J   K   L
    slide current row/column left/down/up/right
gH  gJ  gK  gL
    slide current row/column all the way to the left/bottom/top/right of sheet

zh  zj  zk  zl
    scroll one left/down/up/right

  Column Manipulation
_ (underscore)
    adjust width of current column
g_  adjust width of all visible columns
z_ number
    adjust width of current column to number

- (hyphen)
    hide current column
z-  reduce width of current column by half
g-  delete this column (cannot be undeleted)
gv  unhide all columns

! z!
    toggle/unset current column as a key column
~  #  %  $  @
    set type of current column to str/int/float/currency/date
^   edit name of current column
g^  set names of all unnamed visible columns to contents of selected rows (or current row)
z^  set name of current column to contents of current cell
gz^ set name of current column to combined contents of current column for selected rows (or current row)

= expr
    create new column from Python expr, with column names as variables
g= expr
    set current column for selected rows to result of Python expr
gz= expr
    set current column for selected rows to the items in result of Python sequence expr
z= expr
    set current cell to result of evaluated Python expr on current row
' (tick)
    add a frozen copy of current column with all cells evaluated
g'  open a frozen copy of current sheet with all visible columns evaluated
z'  gz'
    reset cache for current/all visible column(s)

: regex
    add new columns from regex split; number of columns determined by example row at cursor
; regex
    add new columns from capture groups of regex (also requires example row)
* regex/subst
    add column derived from current column, replacing regex with subst (may include \1 backrefs)
g* regex/subst
    modify selected rows in current column, replacing regex with subst

  Row Selection
s   t   u
    select/toggle/unselect current row
gs  gt  gu
    select/toggle/unselect all rows
zs  zt  zu
    select/toggle/unselect rows from top to cursor
gzs gzt gzu
    select/toggle/unselect rows from cursor to bottom
|   \ regex
    select/unselect rows matching regex in current column
g|  g\ regex
    select/unselect rows matching regex in any visible column
, (comma)
    select rows matching current cell in current column
g,  select rows matching current row in all visible columns

  Row Sorting/Filtering
[   ]
    sort ascending/descending by current column
g[  g]
    sort ascending/descending by all key columns
"   open duplicate sheet with only selected rows
g"  open duplicate sheet with all rows
gz" open duplicate sheet with deepcopy of selected rows

  Editing Rows and Cells
a   za
    append a blank row/column
ga  gza number
    append number blank rows/columns
d   gd
    delete current/selected row(s) and move to clipboard
y   gy
    yank (copy) current/all selected row(s) to clipboard
zy  yank (copy) current cell to clipboard
p    P
    paste clipboard rows after/before current row
zp  gzp
    set contents of current column for current/selected row(s) to last clipboard value
Y   gY
    yank (copy) current/all selected row(s) to system clipboard (using options.clipboard_copy_cmd)
zY  yank (copy) current cell to system clipboard (using options.clipboard_copy_cmd)
f   fill null cells in current column with contents of non-null cells up the current column
e text
    edit contents of current cell
ge text
    set contents of current column for selected rows to text
zd   Del
    set contents of current cell to None
gzd  gDel
    set contents of cells in current column to None for selected rows

  Commands While Editing Input
Enter  ^C
    accept/abort input
^O  open external $EDITOR to edit contents
^R  reload initial value
^A  ^E
    move to beginning/end of line
^B  ^F
    move back/forward one character
^H  ^D
    delete previous/current character
^T  transpose previous and current characters
^U  ^K
    clear from cursor to beginning/end of line
Backspace  Del
    delete previous/current character
Insert
    toggle insert mode
Up  Down
    set contents to previous/next in history
Tab  Shift+Tab
    autocomplete input (when available)

  Data Toolkit
o input
    open input in VisiData
^S g^S filename
    save current/all sheet(s) to filename in format determined by extension (default .tsv)
    Note: if the format does not support multisave, or the filename ends in a /, a directory will be created.
z^S filename
    save key columns and current column only to filename in format determined by extension (default .tsv)
^D filename.vd
    save CommandLog to filename.vd file
Shift+A number
    open new blank sheet with number columns
R number
    select a random population subset of number rows
Shift+T
    open new sheet with rows and columns transposed

+ aggregator
    add aggregator to current column (see Frequency Table)
z+ aggregator
    display result of aggregator over values in selected rows for current column

  Data Visualization
. (dot)
    plot current numeric column vs key columns. The numeric key column is used for the x-axis; categorical key column values determine color.
g.  plot a graph of all visible numeric columns vs key columns.

If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), . plots the current row, and g. plots all selected rows (or all rows if none selected).

  Canvas-specific Commands
+   -
    increase/decrease zoom level, centered on cursor
_ (underscore)
    zoom to fit full extent
s   t   u
    select/toggle/unselect rows on source sheet contained within canvas cursor
gs  gt  gu
    select/toggle/unselect rows on source sheet visible on screen
d   delete rows on source sheet contained within canvas cursor
gd  delete rows on source sheet visible on screen
Enter
    open sheet of source rows contained within canvas cursor
gEnter
    open sheet of source rows visible on screen
1 - 9
    toggle display of layers
^L  redraw all pixels on canvas
w   toggle show_graph_labels option
mouse scrollwheel
    zoom in/out of canvas
left click-drag
    set canvas cursor
right click-drag
    scroll canvas

  Other Commands
Shift+Q
    quit current sheet and remove it from the CommandLog
Shift+V
    view contents of current cell in a new TextSheet
v   toggle visibility (text wrap on TextSheet, legends/axes on Graph)

Space
    open command selection menu
^A longname
    execute command by its longname

^E  view traceback for most recent error
g^E view traceback for most recent errors
z^E view traceback for error in current cell

^L  refresh screen
^R  reload current sheet
z^R clear cache for current column
^Z  suspend VisiData process
^G  show cursor position and bounds of current sheet on status line
^V  show version and copyright information on status line
^P  open Status History

^Y  z^Y  g^Y
    open current row/cell/sheet as Python object
^X expr
    evaluate Python expr and opens result as Python object
z^X expr
    evaluate Python expr on current row and shows result on status line
g^X stmt
    execute Python stmt in the global scope

  Internal Sheets List
.   Directory Sheet browse and modify properties of files in a directory
Metasheets
.   Columns Sheet (Shift+C) edit column properties (gC for all columns from all sheets)
.   Sheets Sheet (Shift+S) jump between sheets or join them together
.   Options Sheet (Shift+O) edit all configurable options
.   Commandlog (Shift+D) modify and save commands for replay
.   Error Sheet (^E) view last error
.   Status History (^P) view history of status messages
.   Threads Sheet (^T) view, cancel, and profile asynchronous threads

Derived Sheets
.   Frequency Table (Shift+F) group rows by column value, with aggregations of other columns
.   Describe Sheet (Shift+I) view summary statistics for each column
.   Pivot Table (Shift+W) group rows by key and summarize current column
.   Melted Sheet (Shift+M) unpivot non-key columns into variable/value columns

  INTERNAL SHEETS

  Directory Sheet
(sheet-specific commands)
Modifying any cell changes the in-memory value. Changes are only applied to the filesystem with ^S
Enter  gEnter
    open current/selected file(s) as new sheet(s)
^O  g^O
    open current/selected file(s) in external $EDITOR
d   gd
    schedule current/selected file(s) for deletion
z^R   ^R
    reload information for current/all file(s), undoing any pending changes
z^S   ^S
    apply all deferred changes to current/all file(s)

  METASHEETS

  Columns Sheet (Shift+C)
Properties of columns on the source sheet can be changed with standard editing commands (e ge g= Del) on the Columns Sheet. Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column. The 'g' commands affect the selected rows, which are the literal columns on the source sheet.
(sheet-specific commands)
&   add column from concatenating selected source columns
g! gz!
    toggle/unset selected columns as key columns on source sheet
g+ aggregator
    add Ar aggregator No to selected source columns
g- (hyphen)
    hide selected columns on source sheet
g~ g# g% g$ g@
    set type of selected columns on source sheet to str/int/float/currency/date
Enter
    open a Frequency Table sheet grouped by column referenced in current row

  Sheets Sheet (Shift+S)
(sheet-specific commands)
Enter
    jump to sheet referenced in current row
a   add row to reference a new blank sheet
gC  open Columns Sheet with all columns from selected sheets
g^R reload all selected sheets
& jointype
    merge selected sheets with visible columns from all, keeping rows according to jointype:
.   inner keep only rows which match keys on all sheets
.   outer keep all rows from first selected sheet
.   full keep all rows from all sheets (union)
.   diff keep only rows NOT in all sheets
.   append keep all rows from all sheets (concatenation)

  Options Sheet (Shift+O)
(sheet-specific commands)
Enter  e
    edit option

  CommandLog (Shift+D)
(global commands)
gD  open Directory Sheet for options.visidata_dir (default: ~/.visidata/), which contains saved commandlogs and macros
(sheet-specific commands)
x   replay command in current row
gx  replay contents of entire CommandLog
^C  abort replay
z^S keystroke
    save selected rows to macro mapped to keystroke
    Macros are saved to .visidata/macro/command-longname.vd. The list of macros is saved at .visidata/macros.vd (keystroke, filename).

  DERIVED SHEETS

  Frequency Table (Shift+F)
A Frequency Table groups rows by one or more columns, and includes summary columns for those with aggregators.
(global commands)
gF  open Frequency Table, grouped by all key columns on source sheet
zF  open one-line summary for selected rows
(sheet-specific commands)
s   t   u
    select/toggle/unselect these entries in source sheet
Enter
    open sheet of source rows that are grouped in current cell

  Describe Sheet (Shift+I)
(sheet-specific commands)
zs  zu
    select/unselect rows on source sheet that are being described in current cell
!   toggle/unset current column as a key column on source sheet
Enter
    open a Frequency Table sheet grouped on column referenced in current row
zEnter
    open copy of source sheet with rows described in current cell

  Pivot Table (Shift+W)
Set key column(s) and aggregators on column(s) before pressing Shift+W on the column to pivot.
(sheet-specific commands)
Enter
    open sheet of source rows aggregated in current pivot row
zEnter
    open sheet of source rows aggregated in current pivot cell

  Melted Sheet (Shift+M)
Open melted sheet (unpivot), with key columns retained and all non-key columns reduced to Variable-Value rows.
(global commands)
gM regex
    open melted sheet (unpivot), with key columns retained and regex capture groups determining how the non-key columns will be reduced to Variable-Value rows.

  Python Object Sheet (^X ^Y g^Y z^Y)
(sheet-specific commands)
Enter
    dive further into Python object
e   edit contents of current cell
v   toggle show/hide for methods and hidden properties
gv  zv
    show/hide methods and hidden properties
(   expand current column of lists or dicts into multiple columns and hide original column
)   unexpand current column; restore original column and remove other columns at this level

COMMANDLINE OPTIONS
tsv set loader to use for filetype instead of file extension True overwrite existing files without confirmation
--diff=base
    None add colorizer for all sheets against base
--encoding=str               utf-8
    encoding passed to codecs.open
--encoding-errors=str        surrogateescape
    encoding errors passed to codecs.open
--regex-flags=str            I
    flags to pass to re.compile() [AILMSUX]
--default-width=int          20
    default column width
--wrap=bool                  True
    wrap text to fit window width on TextSheet
--cmd-after-edit=str         j
    command keystroke to execute after successful edit
--cmdlog-longname=bool       False
    Use command longname in cmdlog if available
--col-cache-size=int         0
    max number of cache entries in each cached column
--none-is-null=bool          True
    if Python None counts as null
--empty-is-null=bool         False
    if empty string counts as null
--false-is-null=bool         False
    if Python False counts as null
--zero-is-null=bool          False
    if integer 0 counts as null
--error-is-null=bool         False
    if error counts as null
--force-valid-colnames=bool  False
    clean column names to be valid Python identifiers
--debug=bool                 False
    exit on error and display stacktrace
--curses-timeout=int         100
    curses timeout in ms
--force-256-colors=bool      False
    use 256 colors even if curses reports fewer
--use-default-colors=bool    False
    curses use default terminal colors
--note-pending=str           ⌛
    note to display for pending cells
--note-format-exc=str       ?
    cell note for an exception during type conversion or formatting
--note-getter-exc=str       !
    cell note for an exception during computation
--scroll-incr=int            3
    amount to scroll with scrollwheel
--skip=int                   0
    skip first N lines of text input
--profile=str
    filename to save binary profiling data
--min-memory-mb=int          0
    minimum memory to continue loading and async processing
--confirm-overwrite=bool     True
    whether to prompt for overwrite confirmation on save
--header=int                 1
    parse first N rows of .csv/.tsv as column names
--delimiter=str
    delimiter to use for tsv filetype
--filetype=str
    specify file type
--save-filetype=str          tsv
    specify default file type to save as
--tsv-safe-char=str          ·
    replacement for all tabs and newlines when saving to tsv
--clipboard-copy-cmd=str     pbcopy w
    command to copy stdin to system clipboard
--pyobj-show-hidden=bool     False
    show _private properties on pyobjs
--pyobj-show-methods=bool    False
    show methods on pyobjs
--replay-wait=float          0.0
    time to wait between replayed commands, in seconds
--replay-movement=bool       False
    insert movements during replay
--visidata-dir=str           ~/.visidata/
    directory to load and store macros
--rowkey-prefix=str          キ
    string prefix for rowkey in the cmdlog
--cmdlog-histfile=str
    file to autorecord each cmdlog action to
--regex-maxsplit=int         0
    maxsplit to pass to regex.split
--show-graph-labels=bool     True
    show axes and legend on graph
--plot-colors=str            green red yellow cyan magenta white 38 136 168
    list of distinct colors to use for plotting distinct objects
--zoom-incr=float            2.0
    amount to multiply current zoomlevel when zooming
--motd-url=str               http://visidata.org/motd-1.2
    source of randomized startup messages
--csv-dialect=str            excel
    dialect passed to csv.reader
--csv-delimiter=str         ,
    delimiter passed to csv.reader
--csv-quotechar=str          "
    quotechar passed to csv.reader
--csv-skipinitialspace=bool  True
    skipinitialspace passed to csv.reader
--csv-escapechar=NoneType    None
    escapechar passed to csv.reader
--fixed-rows=int             1000
    number of rows to check for fixed width columns

  DISPLAY OPTIONS
Display options can only be set via the Options Sheet or a .visidatarc (see FILES).

disp_note_none      ⌀
    visible contents of a cell whose value is None
disp_date_fmt       %Y-%m-%d
    default fmtstr to strftime for date values
disp_truncator      …
    indicator that the contents are only partially visible
disp_oddspace       ·
    displayable character for odd whitespace
disp_unprintable   .
    substitute character for unprintables
disp_column_sep     |
    separator between columns
disp_keycol_sep     ‖
    separator between key columns and rest of columns
disp_status_fmt     {sheet.name}|
    status line prefix
disp_lstatus_max    0
    maximum length of left status line
disp_status_sep      |
    separator between statuses
disp_edit_fill      _
    edit field fill character
disp_more_left      <
    header note indicating more columns to the left
disp_more_right     >
    header note indicating more columns to the right
disp_error_val
    displayed contents for computation exception
disp_ambig_width    1
    width to use for unicode chars marked ambiguous
color_default       normal
    the default color
color_default_hdr   bold underline
    color of the column headers
color_current_row   reverse
    color of the cursor row
color_current_col   bold
    color of the cursor column
color_current_hdr   reverse underline
    color of the header for the cursor column
color_column_sep    246 blue
    color of column separators
color_key_col       81 cyan
    color of key columns
color_selected_row  215 yellow
    color of selected rows
color_status        bold
    status line color
color_edit_cell     normal
    cell color to use when editing cell
disp_pending
    string to display in pending cells
color_note_pending  bold magenta
    color of note in pending cells
color_note_type     226 yellow
    cell note for numeric types in anytype columns
color_format_exc    48 green
    color of formatting exception note
color_getter_exc    red
    color of computation exception note
color_menu_prefix   green
    color of accepted menu part
color_menu_option   white
    default menu color
color_menu_cursor   bold reverse
    color of menu cursor
color_menu_help     bold
    color of menu help text
disp_menu_helpfmt   {bindings} ⇨ {helpstr}
    string between command keybindings and helpstr in menu
disp_menu_helpsep    |
    string between submenu options
color_change_pending reverse yellow
    color for file attributes pending modification
color_delete_pending red
    color for files pending delete
disp_histogram      *
    histogram element character
disp_histolen       50
    width of histogram column
disp_replay_play    ▶
    status indicator for active replay
disp_replay_pause   ‖
    status indicator for paused replay
disp_pixel_random   False
    randomly choose attr from set of pixels instead of most common
color_graph_hidden  238 blue
    color of legend for hidden attribute
color_graph_axis    bold
    color for graph axis labels
color_diff          red
    color of values different from --diff source
color_diff_add      yellow
    color of rows/columns added to --diff source

EXAMPLES
    vd foo.tsv

    vd -f sqlite bar.db

    vd -b countries.fixed -o countries.tsv

    vd postgres://username:password@hostname:port/database

    vd --play tests/pivot.vd --replay-wait 1 --output tests/pivot.tsv

    ls -l | vd -f fixed --skip 1 --header 0

    vd -p foo.vd -b -o - > foo.tsv

    vd --diff foo.tsv bar.tsv

FILES
At the start of every session, VisiData looks for $HOME/.visidatarc, and calls Python exec() on its contents if it exists. For example:

       options.min_memory_mb=100  # stop processing without 100MB free
       globalCommand('0', 'gh')   # alias '0' to move to first column, like vim
       def median(values):
           L = sorted(values)
           return L[len(L)//2]
       aggregator('median', median)

SUPPORTED SOURCES
These are the supported sources:

tsv (tab-separated value)
Plain and simple. VisiData writes tsv format by default. See the --delimiter option.

csv (comma-separated value)
.csv files are a scourge upon the earth, and still regrettably common.
See the --csv-dialect, --csv-delimiter, --csv-quotechar, and --csv-skipinitialspace options.
Accepted dialects are excel-tab, unix, and excel.

fixed (fixed width text)
Columns are autodetected from the first 1000 rows (adjustable with --fixed-rows).

json (single object) and jsonl (one object per line).
Cells containing lists (e.g. [3]) or dicts ({3}) can be expanded into new columns with (and unexpanded with).

png (requires pypng)
Pixels can be edited and saved in data form. Images can be plotted with. (dot).

The following URL schemes are supported:
http (requires requests); can be used as transport for with another filetype
postgres (requires psycopg2)

The following sources may include multiple tables. The initial sheet is the table directory; Enter loads the entire table into memory.

sqlite
xlsx (requires openpyxl)
xls (requires xlrd)
hdf5 (requires h5py)
ttf/otf (requires fonttools)
mbtiles (requires mapbox-vector-tile)
htm/html (requires lxml)
xpt (SAS; requires xport)
sas7bdat (SAS; requires sas7bdat)
sav (SPSS; requires savReaderWriter)
dta (Stata; requires pandas)
shp (requires pyshp)

In addition, .zip, .gz, .bz2, and .xz files are decompressed on the fly.

SUPPORTED OUTPUT FORMATS
These are the supported savers:

tsv (tab-separated value)
csv (comma-separated value)
json (one object with all rows)
All expanded subcolumns must be closed (with)) to retain the same structure.
.shp files can be saved as geoJSON.
md (org-mode compatible markdown table)
htm/html (requires lxml)
png (requires pypng)

Multisave is supported by html, md, and txt; g^S will save all sheets into a single output file.

AUTHOR
VisiData was made by Saul Pwanson <vd@saul.pw>.