wee_database - WeeWX 5.0
Author: Tom Keffer <tkeffer@gmail.com>

Skip to content

# wee_database

The database utility simplifies typical database maintenance operations. For example, it can rebuild the daily summaries or check a SQLite database for embedded strings where floats are expected.

Specify `--help` to see how it is used:

    wee_database --help

    Usage: wee_database --help
           wee_database --create
           wee_database --reconfigure
           wee_database --transfer --dest-binding=BINDING_NAME [--dry-run]
           wee_database --add-column=NAME [--type=(REAL|INTEGER)]
           wee_database --rename-column=NAME --to-name=NEW_NAME
           wee_database --drop-columns=NAME1,NAME2,...
           wee_database --check
           wee_database --update [--dry-run]
           wee_database --drop-daily
           wee_database --rebuild-daily [--date=YYYY-mm-dd |
                                        [--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
                                        [--dry-run]
           wee_database --reweight [--date=YYYY-mm-dd |
                                   [--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
                                   [--dry-run]
           wee_database --calc-missing [--date=YYYY-mm-dd |
                                       [--from=YYYY-mm-dd[THH:MM]] [--to=YYYY-mm-dd[THH:MM]]]
           wee_database --check-strings
           wee_database --fix-strings [--dry-run]

    Description:

    Manipulate the WeeWX database. Most of these operations are handled
    automatically by WeeWX, but they may be useful in special cases.

    Options:
      -h, --help            show this help message and exit
      --create              Create the WeeWX database and initialize it with the
                            schema.
      --reconfigure         Create a new database using configuration information
                            found in the configuration file. The new database will
                            have the same name as the old database, with a '_new'
                            on the end.
      --transfer            Transfer the WeeWX archive from source database to
                            destination database.
      --add-column=NAME     Add new column NAME to database.
      --type=TYPE           New database column type (INTEGER|REAL) (option --add-
                            column only).  Default is 'REAL'.
      --rename-column=NAME  Rename the column with name NAME.
      --to-name=NEW_NAME    New name of the column (option --rename-column only).
      --drop-columns=NAME1,NAME2,...
                            Drop one or more columns. Names must be separated by
                            commas, with NO SPACES.
      --check               Check the calculations in the daily summary tables.
      --update              Update the daily summary tables if required and
                            recalculate the daily summary maximum windSpeed
                            values.
      --calc-missing        Calculate and store any missing derived observations.
      --check-strings       Check the archive table for null strings that may have
                            been introduced by a SQL editing program.
      --fix-strings         Fix any null strings in a SQLite database.
      --drop-daily          Drop the daily summary tables from a database.
      --rebuild-daily       Rebuild the daily summaries from data in the archive
                            table.
      --reweight            Recalculate the weighted sums in the daily summaries.
      --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
      --date=YYYY-mm-dd     This date only (options --calc-missing and --rebuild-
                            daily only).
      --from=YYYY-mm-dd[THH:MM]
                            Start with this date or date-time (options --calc-
                            missing and --rebuild-daily only).
      --to=YYYY-mm-dd[THH:MM]
                            End with this date or date-time (options --calc-
                            missing and --rebuild-daily only).
      --binding=BINDING_NAME
                            The data binding to use. Default is 'wx_binding'.
      --dest-binding=BINDING_NAME
                            The destination data binding (option --transfer only).
      --dry-run             Print what would happen but do not do it. Default is
                            False.

### `--create`

If the database does not already exist, this action will create it and initialize it with the schema specified in the WeeWX configuration file. Because WeeWX does this automatically, this action is rarely needed.

    wee_database --create

### `--reconfigure`

This action is useful for changing the schema or unit system in your database.

It creates a new database with the same name as the old, except with the suffix `_new` attached at the end (nominally, `weewx.sdb_new` if you are using SQLite, `weewx_new` if you are using MySQL). It then initializes it with the schema specified in `weewx.conf`. Finally, it copies over the data from your old database into the new database.

    wee_database --reconfigure

See the section Changing the database unit system in an existing database in the Customization Guide for step-by-step instructions that use this option.

### `--transfer`

This action is useful for moving your database from one type of database to another, such as from SQLite to MySQL. To use it, you must have two bindings specified in your `weewx.conf` configuration file. One will serve as the source, the other as the destination. Specify the source binding with option `--binding`, the destination binding with option `--dest-binding`. The `--binding` option may be omitted in which case the default `wx-binding` will be used.

    wee_database --transfer --binding=source_binding --dest-binding=dest_binding
    wee_database --transfer --dest-binding=dest_binding

See the Wiki for examples of moving data from SQLite to MySQL, and from MySQL to SQLite, using `wee_database`.

### `--add-column=NAME`

This action adds a new database observation type (column) to the database. If used without the `--type` option, the type will default to `REAL`.

    wee_database --add-column

Optionally, option `--type` can be used with a SQL type `REAL`, `INTEGER`, or any other SQL column definition (such as `INTEGER DEFAULT 0`).

    wee_database --add-column=NAME --type=TYPE

### `--rename-column=NAME`

Use this action to rename a database observation type (column) to a new name. It requires the option `--to-name`.

    wee_database --rename-column=NAME --to-name=NEW_NAME

For example, to rename the column `luminosity` in your schema to `illuminance`:

    wee_database --rename-column=luminosity --to-name=illuminance

### `--drop-columns=NAME`

This action will drop one or more observation types (columns) from the database. If more than one column name is given, they should be separated by commas and no spaces.

It is an error to attempt to drop a non-existing column. In this case, nothing will be done.

    wee_database --drop-columns=NAME1,NAME2

Note

When dropping columns from a SQLite database, the entire database must be copied except for the dropped columns. Because this can be quite slow, if you are dropping more than one column, it is better to do them all in one pass. This is why option `--drop-columns` accepts more than one name.

### `--check`

This action will check the calculations in the daily summary tables as well as checking the archive for null strings (refer to `--check-strings`). If the daily summary tables contain summaries calculated using an old algorithm, the user is advised to update the daily summary tables using the `--update` action. If null strings are found the user is advised to fix them using the `--fix-strings` action.

    wee_database --check

FIXME

### `--update`

This action updates the daily summary tables to use interval weighted calculations as well as recalculating the `windSpeed` maximum daily values and times. Interval weighted calculations are only applied to the daily summaries if not previously applied. The update process is irreversible and users are advised to backup their database before performing this action.

    wee_database --update

For further information on interval weighting and recalculation of daily `windSpeed` maximum values, see the sections Changes to daily summaries and Recalculation of `windSpeed` maximum values in the Upgrade Guide.

### `--drop-daily`

In addition to the regular archive data, every WeeWX database also includes a daily summary table for each observation type. Because there can be dozens of observation types, there can be dozens of these daily summaries. It does not happen very often, but there can be occasions when it's necessary to drop them all and then rebuild them. Dropping them by hand would be very tedious! This action does them all at once.

    wee_database --drop-daily

### `--rebuild-daily`

This action is the inverse of action `--drop-daily` in that it rebuilds the daily summaries from the archive data. In most cases it is not necessary to drop the daily summary tables using the action `--drop-daily` before rebuilding them.

The action `--rebuild-daily` accepts a number of date related options, `--date`, `--from` and `--to` that allow selective rebuilding of the daily summaries for one or more days rather than for the entire archive history. These options may be useful if bogus data has been removed from the archive covering a single day or a period of few days. The daily summaries can then be rebuilt for this period only, resulting in a faster rebuild and detailed low/high values and the associated times being retained for unaffected days.

The `--date` option limits the daily summary rebuild to the specified date.

The `--from` and `--to` options may be used together or individually to limit the daily summary rebuild to a specified period. When used individually the `--to` option limits the rebuild to the inclusive period from the earliest date for which there is data in the database through to and including the specified date. Similarly when used individually the `--from` option limits the rebuild to the inclusive period from the specified date through until the last date for which there is data in the database. When the `--from` and `--to` options are used together the daily summary rebuild is limited to the specified inclusive period.

    wee_database --rebuild-daily
    wee_database --rebuild-daily --date=YYYY-mm-dd
    wee_database --rebuild-daily --from=YYYY-mm-dd
    wee_database --rebuild-daily --to=YYYY-mm-dd
    wee_database --rebuild-daily --from=YYYY-mm-dd --to=YYYY-mm-dd

Note

Whilst the `--from` and `--to` options will accept optional hour and minutes in the format `THH:MM`, any such hour and minute options are ignored by the `--rebuild` action as the daily summaries represent whole days only and it is not possible to partially rebuild a daily summary.

Note

When used with the `--rebuild-daily` action the period defined by `--to` and `--from` is inclusive and the daily summary tables will be rebuild for the day defined by `--from` and the day defined by `--to` and all days in between.

### `--reweight`

As an alternative to dropping and rebuilding the daily summaries, this action simply rebuilds the weighted daily sums (used to calculate averages) from the archive data. It does not touch the highs and lows. It is much faster than `--rebuild-daily`, and has the advantage that the highs and lows remain unchanged.

Other options are as in `--rebuild-daily`.

### `--calc-missing`

This action calculates derived observations for archive records in the database and then stores the calculated observations in the database. This can be useful if erroneous archive data is corrected or some additional observational data is added to the archive that may alter previously calculated or missing derived observations.

The period over which the derived observations are calculated can be limited through use of the `--date`, `--from` and/or `--to` options. When used without any of these options `--calc-missing` will calculate derived observations for all archive records in the database. The `--date` option limits the calculation of derived observations to the specified date only. The `--from` and `--to` options can be used together to specify the start and end date-time respectively of the period over which derived observations will be calculated.

If `--from` is used by itself the period is fom the date-time specified up to and including the last archive record in the database.

If `--to` is used by itself the period is the first archive record in the database through to the specified date-time.

    wee_database --calc-missing
    wee_database --calc-missing --date=YYYY-mm-dd
    wee_database --calc-missing --from=YYYY-mm-dd[THH:MM]
    wee_database --calc-missing --to=YYYY-mm-dd[THH:MM]
    wee_database --calc-missing --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]

Note

When a `YYYY-mm-dd` date is used as the `--from` option the period used by `--calc-missing` includes records after midnight at the start of `YYYY-mm-dd`, if a `YYYY-mm-ddTHH:MM` format date-time is used as the `--from` option the period includes records after `YYYY-mm-dd HH:MM`. When a `YYYY-mm-dd` date is used as the `--to` option the period includes records up to and including midnight at the end of `YYYY-mm-dd`, if a `YYYY-mm-ddTHH:MM` format date-time is used as the `--to` option the period includes records up to and including `YYYY-mm-dd HH:MM`. When using the `--date` option the period is all records after midnight at the start of `YYYY-mm-dd` up to and including midnight at the end of `YYYY-mm-dd`, in effect the same as `--from=YYYY-mm-ddT00:00 --to=YYYY-mm-dd+1T00:00`.

Note

`--calc-missing` uses the `StdWXCalculate` service to calculate missing derived observations. The data binding used by the `StdWXCalculate` service should normally match the data binding of the database being operated on by `--calc-missing`. Those who use custom or additional data bindings should take care to ensure the correct data bindings are used by both `--calc-missing` and the `StdWXCalculate` service. Those who use the default data binding need take no special precautions.

### `--check-strings`

Normally, all entries in the archive are numbers. However, some SQLite database editors use a null string instead of a null value when deleting entries. These null strings can cause problems. This action checks the database to see if it contains any null strings.

    wee_database --check-strings

### `--fix-strings`

This action will check for any null strings in a SQLite database and if found substitute a true null value.

    wee_database --fix-strings