Commands
Every option of every command, as quiver COMMAND --help prints it. Each option also reads the environment variable it names, and a quiverfile.
quiver dump
Dump MySQL or MariaDB into SQL or CSV files
NAME
quiver-dump - Dump MySQL or MariaDB into SQL or CSV files
SYNOPSIS
quiver dump [OPTION]…
OPTIONS
-B DB, --database=DB (absent QUIVER_DATABASE env)
Dump only DB; repeatable.
--batch-bytes=BYTES (absent=262144 or QUIVER_BATCH_BYTES env)
A reader hands a batch of rows on once it holds this many bytes;
larger batches are not faster, smaller ones cost more per row.
--chunk-bytes=BYTES (absent=67108864 or QUIVER_CHUNK_BYTES env)
Target size of one chunk.
--compress=CODEC (absent=none or QUIVER_COMPRESS env)
zstd compresses every data file, which zstd -d reads back; none
does not.
--connect-timeout=SECONDS (absent=10 or QUIVER_CONNECT_TIMEOUT env)
Fail when connecting, TLS or authentication waits for the server
so long.
--csv-dialect=DIALECT (absent=rfc4180 or QUIVER_CSV_DIALECT env)
rfc4180: RFC 4180 with a header line and CRLF; mysql-load-data:
what LOAD DATA reads with backslash escapes and no header.
--data-database=DB (absent QUIVER_DATA_DATABASE env)
Take rows only of DB's tables, repeatable; the other tables of the
dump give only their schema.
--data-table=DB.TABLE (absent QUIVER_DATA_TABLE env)
Take rows only of DB.TABLE, repeatable; the other tables of the
dump give only their schema.
--declare-tables=WHEN (absent=before-rows or QUIVER_DECLARE_TABLES
env)
before-rows tells the sink of a table just before its first chunk;
all-at-once tells it of every table before any rows, so copy
creates or prepares them while rows are read.
--format=FORMAT (absent=sql or QUIVER_FORMAT env)
sql: INSERT statements myloader and mysql read; csv: one CSV file
per chunk.
-h HOST, --host=HOST (absent QUIVER_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--keyless-read=WAY (absent=scan or QUIVER_KEYLESS_READ env)
How a table without a primary key or a unique key over NOT NULL
columns is read: scan in one pass over the whole table, cheap for
the server; ranges by every reader at once, in ranges of another
index's first column, each row looked up through that index.
--mask=DB.TABLE.COLUMN=ACTION (absent QUIVER_MASK env)
Mask the column's values before they leave the reader, repeatable;
a column takes the first rule naming it, and a rule naming no
column of the dump is an error. Names take * and ?, or backquotes
for a name as written. DB.TABLE.COLUMN=ACTION is permute (a keyed
bijection, FF1 of NIST SP 800-38G: an integer to another from 1 to
half its type's largest, or to 2^53 - 1 for BIGINT; text keeping
each character's kind, case and place; binary of the same length),
permute:LOW..HIGH (integers among those); null; text:VALUE; shape
(each letter and digit replaced by a keyed one of its kind, the
rest kept); email (shape before the last @, the domain kept); hash
(a keyed digest of the same length, or an integer of the column's
type); sql:EXPRESSION, evaluated by the source for each row and
cast to the column's type; or expr:EXPRESSION of the masking
language (see quiver mask --help). NULL stays NULL but under null,
sql: and expr:. A column of a primary or unique key takes only
permute, keep or null where it may be NULL, since another action
can give two values one. A permute spreads along the foreign keys
of the source to every column tied to the one it masks that no
rule names, so references keep pointing at their rows; tied
columns masked differently are an error.
--mask-script=FILE (absent QUIVER_MASK_SCRIPT env)
Read definitions and (mask "DB.TABLE.COLUMN" EXPRESSION) rules of
the masking language from FILE, repeatable: a later file calls an
earlier one's definitions, and its rules come after those of
--mask. Its (after-load …) forms run with copy. See quiver mask
--help.
--mask-secret=SECRET (absent QUIVER_MASK_SECRET env)
Key permute, shape, email, hash and the keyed functions of expr:
with SECRET, so a value masks the same way in every column and
every run; a random key of this run when not given. Whoever holds
SECRET can undo permute: keep it apart from the copies, one per
environment.
--mask-strict=DB.TABLE (absent QUIVER_MASK_STRICT env)
Every column of DB.TABLE must be masked or kept (=keep) by a rule,
repeatable; a column nobody decided on stops the run before
anything is read.
--net-timeout=SECONDS (absent=600 or QUIVER_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
--no-data (absent QUIVER_NO_DATA env)
Take every table's schema and no rows.
-o DIR, --outputdir=DIR (required)
Where the dump goes.
--objects=WHICH (absent=all or QUIVER_OBJECTS env)
all takes the dumped databases' views, triggers, routines and
events; none takes none of them.
-p PASSWORD, --password=PASSWORD (absent MYSQL_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
-P PORT, --port=PORT (absent QUIVER_PORT env)
The server's TCP port; 3306 unless given.
--read-order=ORDER (absent=largest-chunks or QUIVER_READ_ORDER env)
largest-chunks reads the largest chunks first, wherever their
tables are; largest-tables reads the largest tables first, their
chunks together, so what a sink does after a table's last part (a
tablespace's check, import and keys) starts early.
--render-threads=N (absent QUIVER_RENDER_THREADS env)
Domains turning batches into output; by default half the readers,
within the cores quiver may use.
-S PATH, --socket=PATH (absent QUIVER_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--snapshot=MODE (absent=ftwrl or QUIVER_SNAPSHOT env)
ftwrl and backup-stage (MariaDB) give one consistent snapshot;
trx-only takes no lock and is only best effort.
--statement-bytes=BYTES (absent=1000000 or QUIVER_STATEMENT_BYTES env)
Approximate size of one INSERT statement.
-t N, --threads=N (absent QUIVER_THREADS env)
Database sessions reading in parallel, one domain each; by default
half the cores quiver may use (at least 4), fewer when the server
suggests fewer. quiver uses this machine's fast physical cores
within its CPU quota, half of them when the server is on this
machine.
-T DB.TABLE, --table=DB.TABLE (absent QUIVER_TABLE env)
Dump only DB.TABLE; repeatable. Its database's views, triggers,
routines and events still come along unless --objects none.
--tls=MODE (absent QUIVER_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--tls-ca=FILE (absent QUIVER_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
-u USER, --user=USER (absent=root or QUIVER_USER env)
The account to log in as.
--zero-dates=MODE (absent=reject or QUIVER_ZERO_DATES env)
reject fails on a zero or invalid date such as 0000-00-00 or
2020-02-30, which has no exact value; keep writes every date as
stored, and restoring it needs a server that accepts it.
--zstd-level=LEVEL (absent=3 or QUIVER_ZSTD_LEVEL env)
The zstd level of --compress zstd, from 1 (fastest) to 19.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver dump exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver dump:
MYSQL_PWD
See option --password.
QUIVER_BATCH_BYTES
See option --batch-bytes.
QUIVER_CHUNK_BYTES
See option --chunk-bytes.
QUIVER_COMPRESS
See option --compress.
QUIVER_CONNECT_TIMEOUT
See option --connect-timeout.
QUIVER_CSV_DIALECT
See option --csv-dialect.
QUIVER_DATABASE
See option --database.
QUIVER_DATA_DATABASE
See option --data-database.
QUIVER_DATA_TABLE
See option --data-table.
QUIVER_DECLARE_TABLES
See option --declare-tables.
QUIVER_FORMAT
See option --format.
QUIVER_HOST
See option --host.
QUIVER_KEYLESS_READ
See option --keyless-read.
QUIVER_MASK
See option --mask.
QUIVER_MASK_SCRIPT
See option --mask-script.
QUIVER_MASK_SECRET
See option --mask-secret.
QUIVER_MASK_STRICT
See option --mask-strict.
QUIVER_NET_TIMEOUT
See option --net-timeout.
QUIVER_NO_DATA
See option --no-data.
QUIVER_OBJECTS
See option --objects.
QUIVER_OUTPUTDIR
See option --outputdir.
QUIVER_PORT
See option --port.
QUIVER_READ_ORDER
See option --read-order.
QUIVER_RENDER_THREADS
See option --render-threads.
QUIVER_SNAPSHOT
See option --snapshot.
QUIVER_SOCKET
See option --socket.
QUIVER_STATEMENT_BYTES
See option --statement-bytes.
QUIVER_TABLE
See option --table.
QUIVER_THREADS
See option --threads.
QUIVER_TLS
See option --tls.
QUIVER_TLS_CA
See option --tls-ca.
QUIVER_USER
See option --user.
QUIVER_ZERO_DATES
See option --zero-dates.
QUIVER_ZSTD_LEVEL
See option --zstd-level.
SEE ALSO
quiver(1)
quiver load
Load a quiver dump into MySQL or MariaDB
NAME
quiver-load - Load a quiver dump into MySQL or MariaDB
SYNOPSIS
quiver load [OPTION]…
OPTIONS
--after-load=DB.TABLE=STATEMENT (absent QUIVER_AFTER_LOAD env)
Run DB.TABLE=STATEMENT on the target once every table's rows and
keys are in, before foreign keys, views and triggers, repeatable:
in the target's database of the table, {table} standing for its
name there; a table's statements in the given order, tables at
once. Names take * and ?; a rule naming no table is an error.
-B DB, --database=DB (absent QUIVER_DATABASE env)
Load the dump's only database under this name.
--binlog=MODE (absent=write or QUIVER_BINLOG env)
skip keeps the load out of the server's binlog (less work for the
server, but replicas never see the data; needs the right to set
sql_log_bin); write logs it as usual.
--connect-timeout=SECONDS (absent=10 or QUIVER_CONNECT_TIMEOUT env)
Fail when connecting, TLS or authentication waits for the server
so long.
-d DIR, --directory=DIR (required)
The dump to load.
-h HOST, --host=HOST (absent QUIVER_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--keyless-read=WAY (absent=scan or QUIVER_KEYLESS_READ env)
How a table without a primary key or a unique key over NOT NULL
columns is read: scan in one pass over the whole table, cheap for
the server; ranges by every reader at once, in ranges of another
index's first column, each row looked up through that index.
--net-timeout=SECONDS (absent=600 or QUIVER_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
-p PASSWORD, --password=PASSWORD (absent MYSQL_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
-P PORT, --port=PORT (absent QUIVER_PORT env)
The server's TCP port; 3306 unless given.
-S PATH, --socket=PATH (absent QUIVER_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--secondary-keys=WHEN (absent=with-table or QUIVER_SECONDARY_KEYS env)
with-table creates tables as dumped; after-data creates them with
their primary key only and adds secondary and foreign keys once a
table's data is in, so InnoDB builds each index by sorting, which
pays off when indexes outgrow memory.
-t N, --threads=N (absent=4 or QUIVER_THREADS env)
Sessions loading in parallel, one domain each.
--tls=MODE (absent QUIVER_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--tls-ca=FILE (absent QUIVER_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
-u USER, --user=USER (absent=root or QUIVER_USER env)
The account to log in as.
--zero-dates=MODE (absent=reject or QUIVER_ZERO_DATES env)
reject fails on a zero or invalid date such as 0000-00-00 or
2020-02-30, which has no exact value; keep writes every date as
stored, and restoring it needs a server that accepts it.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver load exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver load:
MYSQL_PWD
See option --password.
QUIVER_AFTER_LOAD
See option --after-load.
QUIVER_BINLOG
See option --binlog.
QUIVER_CONNECT_TIMEOUT
See option --connect-timeout.
QUIVER_DATABASE
See option --database.
QUIVER_DIRECTORY
See option --directory.
QUIVER_HOST
See option --host.
QUIVER_KEYLESS_READ
See option --keyless-read.
QUIVER_NET_TIMEOUT
See option --net-timeout.
QUIVER_PORT
See option --port.
QUIVER_SECONDARY_KEYS
See option --secondary-keys.
QUIVER_SOCKET
See option --socket.
QUIVER_THREADS
See option --threads.
QUIVER_TLS
See option --tls.
QUIVER_TLS_CA
See option --tls-ca.
QUIVER_USER
See option --user.
QUIVER_ZERO_DATES
See option --zero-dates.
SEE ALSO
quiver(1)
quiver copy
Copy MySQL or MariaDB into another server, with no
NAME
quiver-copy - Copy MySQL or MariaDB into another server, with no
files: each reading session's chunks stream into a loading session of
its own
SYNOPSIS
quiver copy [OPTION]…
OPTIONS
--after-load=DB.TABLE=STATEMENT (absent QUIVER_AFTER_LOAD env)
Run DB.TABLE=STATEMENT on the target once every table's rows and
keys are in, before foreign keys, views and triggers, repeatable:
in the target's database of the table, {table} standing for its
name there; a table's statements in the given order, tables at
once. Names take * and ?; a rule naming no table is an error.
-B DB, --database=DB (absent QUIVER_DATABASE env)
Dump only DB; repeatable.
--batch-bytes=BYTES (absent=262144 or QUIVER_BATCH_BYTES env)
A reader hands a batch of rows on once it holds this many bytes;
larger batches are not faster, smaller ones cost more per row.
--binlog=MODE (absent=write or QUIVER_BINLOG env)
skip keeps the load out of the server's binlog (less work for the
server, but replicas never see the data; needs the right to set
sql_log_bin); write logs it as usual.
--build-threads=N (absent QUIVER_BUILD_THREADS env)
Threads finishing and checking built files, between all tables at
once: the larger a table, the more it borrows; half the fast cores
when not given.
--chunk-bytes=BYTES (absent=67108864 or QUIVER_CHUNK_BYTES env)
Target size of one chunk.
--connect-timeout=SECONDS (absent=10 or QUIVER_CONNECT_TIMEOUT env)
Fail when connecting, TLS or authentication waits for the server
so long.
--data-database=DB (absent QUIVER_DATA_DATABASE env)
Take rows only of DB's tables, repeatable; the other tables of the
dump give only their schema.
--data-table=DB.TABLE (absent QUIVER_DATA_TABLE env)
Take rows only of DB.TABLE, repeatable; the other tables of the
dump give only their schema.
--datadir=DIR (absent QUIVER_DATADIR env)
The target's data directory as this machine sees it, for --ingest
tablespace; the server's own when not given.
--declare-tables=WHEN (absent=before-rows or QUIVER_DECLARE_TABLES
env)
before-rows tells the sink of a table just before its first chunk;
all-at-once tells it of every table before any rows, so copy
creates or prepares them while rows are read.
-h HOST, --host=HOST (absent QUIVER_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--ingest=WAY (absent=load-data or QUIVER_INGEST env)
tablespace builds each table it can as an InnoDB tablespace beside
the target and imports it (MySQL 8.4 only, with --binlog skip,
quiver on the target's machine); other tables, and every table
with load-data, go through LOAD DATA.
--keyless-read=WAY (absent=scan or QUIVER_KEYLESS_READ env)
How a table without a primary key or a unique key over NOT NULL
columns is read: scan in one pass over the whole table, cheap for
the server; ranges by every reader at once, in ranges of another
index's first column, each row looked up through that index.
--mask=DB.TABLE.COLUMN=ACTION (absent QUIVER_MASK env)
Mask the column's values before they leave the reader, repeatable;
a column takes the first rule naming it, and a rule naming no
column of the dump is an error. Names take * and ?, or backquotes
for a name as written. DB.TABLE.COLUMN=ACTION is permute (a keyed
bijection, FF1 of NIST SP 800-38G: an integer to another from 1 to
half its type's largest, or to 2^53 - 1 for BIGINT; text keeping
each character's kind, case and place; binary of the same length),
permute:LOW..HIGH (integers among those); null; text:VALUE; shape
(each letter and digit replaced by a keyed one of its kind, the
rest kept); email (shape before the last @, the domain kept); hash
(a keyed digest of the same length, or an integer of the column's
type); sql:EXPRESSION, evaluated by the source for each row and
cast to the column's type; or expr:EXPRESSION of the masking
language (see quiver mask --help). NULL stays NULL but under null,
sql: and expr:. A column of a primary or unique key takes only
permute, keep or null where it may be NULL, since another action
can give two values one. A permute spreads along the foreign keys
of the source to every column tied to the one it masks that no
rule names, so references keep pointing at their rows; tied
columns masked differently are an error.
--mask-script=FILE (absent QUIVER_MASK_SCRIPT env)
Read definitions and (mask "DB.TABLE.COLUMN" EXPRESSION) rules of
the masking language from FILE, repeatable: a later file calls an
earlier one's definitions, and its rules come after those of
--mask. Its (after-load …) forms run with copy. See quiver mask
--help.
--mask-secret=SECRET (absent QUIVER_MASK_SECRET env)
Key permute, shape, email, hash and the keyed functions of expr:
with SECRET, so a value masks the same way in every column and
every run; a random key of this run when not given. Whoever holds
SECRET can undo permute: keep it apart from the copies, one per
environment.
--mask-strict=DB.TABLE (absent QUIVER_MASK_STRICT env)
Every column of DB.TABLE must be masked or kept (=keep) by a rule,
repeatable; a column nobody decided on stops the run before
anything is read.
--net-timeout=SECONDS (absent=600 or QUIVER_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
--no-data (absent QUIVER_NO_DATA env)
Take every table's schema and no rows.
--objects=WHICH (absent=all or QUIVER_OBJECTS env)
all takes the dumped databases' views, triggers, routines and
events; none takes none of them.
-p PASSWORD, --password=PASSWORD (absent MYSQL_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
-P PORT, --port=PORT (absent QUIVER_PORT env)
The server's TCP port; 3306 unless given.
--read-order=ORDER (absent=largest-chunks or QUIVER_READ_ORDER env)
largest-chunks reads the largest chunks first, wherever their
tables are; largest-tables reads the largest tables first, their
chunks together, so what a sink does after a table's last part (a
tablespace's check, import and keys) starts early.
--render-threads=N (absent QUIVER_RENDER_THREADS env)
Domains turning batches into output; by default half the readers,
within the cores quiver may use.
-S PATH, --socket=PATH (absent QUIVER_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--secondary-keys=WHEN (absent=with-table or QUIVER_SECONDARY_KEYS env)
with-table creates tables as dumped; after-data creates them with
their primary key only and adds secondary and foreign keys once a
table's data is in, so InnoDB builds each index by sorting, which
pays off when indexes outgrow memory.
--snapshot=MODE (absent=ftwrl or QUIVER_SNAPSHOT env)
ftwrl and backup-stage (MariaDB) give one consistent snapshot;
trx-only takes no lock and is only best effort.
--sort-dir=DIR (absent QUIVER_SORT_DIR env)
Where a table's secondary records wait, sorted, until its file is
finished; beside the file in the target's data directory when not
given.
-t N, --threads=N (absent QUIVER_THREADS env)
Database sessions reading in parallel, one domain each; by default
half the cores quiver may use (at least 4), fewer when the server
suggests fewer. quiver uses this machine's fast physical cores
within its CPU quota, half of them when the server is on this
machine.
-T DB.TABLE, --table=DB.TABLE (absent QUIVER_TABLE env)
Dump only DB.TABLE; repeatable. Its database's views, triggers,
routines and events still come along unless --objects none.
--tablespace-keys=WHO (absent=quiver or QUIVER_TABLESPACE_KEYS env)
Who builds the secondary keys of a table built as a tablespace:
quiver builds those ordered by bytes (integers, dates and times,
decimals, binary strings) into the file, the server adds the
others after the import (the default); server adds them all after
the import.
--target-threads=N (absent QUIVER_TARGET_THREADS env)
Sessions loading into the target, one chunk each; defaults to
--threads. Readers wait for a free one, so loading at its own pace
never stalls rendering.
--tls=MODE (absent QUIVER_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--tls-ca=FILE (absent QUIVER_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
--to-connect-timeout=SECONDS (absent=10 or QUIVER_TO_CONNECT_TIMEOUT
env)
Fail when connecting, TLS or authentication waits for the server
so long.
--to-database=DB (absent QUIVER_TO_DATABASE env)
Copy the only database read under this name.
--to-host=HOST (absent QUIVER_TO_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--to-net-timeout=SECONDS (absent=600 or QUIVER_TO_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
--to-password=PASSWORD (absent QUIVER_TO_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
--to-port=PORT (absent QUIVER_TO_PORT env)
The server's TCP port; 3306 unless given.
--to-socket=PATH (absent QUIVER_TO_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--to-tls=MODE (absent QUIVER_TO_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--to-tls-ca=FILE (absent QUIVER_TO_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--to-tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
--to-user=USER (absent=root or QUIVER_TO_USER env)
The account to log in as.
-u USER, --user=USER (absent=root or QUIVER_USER env)
The account to log in as.
--zero-dates=MODE (absent=reject or QUIVER_ZERO_DATES env)
reject fails on a zero or invalid date such as 0000-00-00 or
2020-02-30, which has no exact value; keep writes every date as
stored, and restoring it needs a server that accepts it.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver copy exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver copy:
MYSQL_PWD
See option --password.
QUIVER_AFTER_LOAD
See option --after-load.
QUIVER_BATCH_BYTES
See option --batch-bytes.
QUIVER_BINLOG
See option --binlog.
QUIVER_BUILD_THREADS
See option --build-threads.
QUIVER_CHUNK_BYTES
See option --chunk-bytes.
QUIVER_CONNECT_TIMEOUT
See option --connect-timeout.
QUIVER_DATABASE
See option --database.
QUIVER_DATADIR
See option --datadir.
QUIVER_DATA_DATABASE
See option --data-database.
QUIVER_DATA_TABLE
See option --data-table.
QUIVER_DECLARE_TABLES
See option --declare-tables.
QUIVER_HOST
See option --host.
QUIVER_INGEST
See option --ingest.
QUIVER_KEYLESS_READ
See option --keyless-read.
QUIVER_MASK
See option --mask.
QUIVER_MASK_SCRIPT
See option --mask-script.
QUIVER_MASK_SECRET
See option --mask-secret.
QUIVER_MASK_STRICT
See option --mask-strict.
QUIVER_NET_TIMEOUT
See option --net-timeout.
QUIVER_NO_DATA
See option --no-data.
QUIVER_OBJECTS
See option --objects.
QUIVER_PORT
See option --port.
QUIVER_READ_ORDER
See option --read-order.
QUIVER_RENDER_THREADS
See option --render-threads.
QUIVER_SECONDARY_KEYS
See option --secondary-keys.
QUIVER_SNAPSHOT
See option --snapshot.
QUIVER_SOCKET
See option --socket.
QUIVER_SORT_DIR
See option --sort-dir.
QUIVER_TABLE
See option --table.
QUIVER_TABLESPACE_KEYS
See option --tablespace-keys.
QUIVER_TARGET_THREADS
See option --target-threads.
QUIVER_THREADS
See option --threads.
QUIVER_TLS
See option --tls.
QUIVER_TLS_CA
See option --tls-ca.
QUIVER_TO_CONNECT_TIMEOUT
See option --to-connect-timeout.
QUIVER_TO_DATABASE
See option --to-database.
QUIVER_TO_HOST
See option --to-host.
QUIVER_TO_NET_TIMEOUT
See option --to-net-timeout.
QUIVER_TO_PORT
See option --to-port.
QUIVER_TO_PWD
See option --to-password.
QUIVER_TO_SOCKET
See option --to-socket.
QUIVER_TO_TLS
See option --to-tls.
QUIVER_TO_TLS_CA
See option --to-tls-ca.
QUIVER_TO_USER
See option --to-user.
QUIVER_USER
See option --user.
QUIVER_ZERO_DATES
See option --zero-dates.
SEE ALSO
quiver(1)
quiver agent
Dump MySQL or MariaDB to a collector over quiver's own
NAME
quiver-agent - Dump MySQL or MariaDB to a collector over quiver's own
protocol: batches travel as they lie in memory, compressed, spread
over --streams connections
SYNOPSIS
quiver agent [OPTION]…
OPTIONS
-B DB, --database=DB (absent QUIVER_DATABASE env)
Dump only DB; repeatable.
--batch-bytes=BYTES (absent=262144 or QUIVER_BATCH_BYTES env)
A reader hands a batch of rows on once it holds this many bytes;
larger batches are not faster, smaller ones cost more per row.
--chunk-bytes=BYTES (absent=67108864 or QUIVER_CHUNK_BYTES env)
Target size of one chunk.
--chunks-in-flight=N (absent QUIVER_CHUNKS_IN_FLIGHT env)
Chunks begun and not yet committed by the collector; defaults to
twice --threads, so readers go on while commits travel back. The
collector loads a server with as many sessions.
--collector=HOST:PORT (required)
The collector that receives the dump (quiver collect).
--connect-timeout=SECONDS (absent=10 or QUIVER_CONNECT_TIMEOUT env)
Fail when connecting, TLS or authentication waits for the server
so long.
--data-database=DB (absent QUIVER_DATA_DATABASE env)
Take rows only of DB's tables, repeatable; the other tables of the
dump give only their schema.
--data-table=DB.TABLE (absent QUIVER_DATA_TABLE env)
Take rows only of DB.TABLE, repeatable; the other tables of the
dump give only their schema.
--declare-tables=WHEN (absent=before-rows or QUIVER_DECLARE_TABLES
env)
before-rows tells the sink of a table just before its first chunk;
all-at-once tells it of every table before any rows, so copy
creates or prepares them while rows are read.
--dscp=N (absent QUIVER_DSCP env)
The DSCP every packet of the dump carries; 8 (CS1) asks routers to
let other traffic go first.
-h HOST, --host=HOST (absent QUIVER_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--keyless-read=WAY (absent=scan or QUIVER_KEYLESS_READ env)
How a table without a primary key or a unique key over NOT NULL
columns is read: scan in one pass over the whole table, cheap for
the server; ranges by every reader at once, in ranges of another
index's first column, each row looked up through that index.
--mask=DB.TABLE.COLUMN=ACTION (absent QUIVER_MASK env)
Mask the column's values before they leave the reader, repeatable;
a column takes the first rule naming it, and a rule naming no
column of the dump is an error. Names take * and ?, or backquotes
for a name as written. DB.TABLE.COLUMN=ACTION is permute (a keyed
bijection, FF1 of NIST SP 800-38G: an integer to another from 1 to
half its type's largest, or to 2^53 - 1 for BIGINT; text keeping
each character's kind, case and place; binary of the same length),
permute:LOW..HIGH (integers among those); null; text:VALUE; shape
(each letter and digit replaced by a keyed one of its kind, the
rest kept); email (shape before the last @, the domain kept); hash
(a keyed digest of the same length, or an integer of the column's
type); sql:EXPRESSION, evaluated by the source for each row and
cast to the column's type; or expr:EXPRESSION of the masking
language (see quiver mask --help). NULL stays NULL but under null,
sql: and expr:. A column of a primary or unique key takes only
permute, keep or null where it may be NULL, since another action
can give two values one. A permute spreads along the foreign keys
of the source to every column tied to the one it masks that no
rule names, so references keep pointing at their rows; tied
columns masked differently are an error.
--mask-script=FILE (absent QUIVER_MASK_SCRIPT env)
Read definitions and (mask "DB.TABLE.COLUMN" EXPRESSION) rules of
the masking language from FILE, repeatable: a later file calls an
earlier one's definitions, and its rules come after those of
--mask. Its (after-load …) forms run with copy. See quiver mask
--help.
--mask-secret=SECRET (absent QUIVER_MASK_SECRET env)
Key permute, shape, email, hash and the keyed functions of expr:
with SECRET, so a value masks the same way in every column and
every run; a random key of this run when not given. Whoever holds
SECRET can undo permute: keep it apart from the copies, one per
environment.
--mask-strict=DB.TABLE (absent QUIVER_MASK_STRICT env)
Every column of DB.TABLE must be masked or kept (=keep) by a rule,
repeatable; a column nobody decided on stops the run before
anything is read.
--max-rate=MB/S (absent QUIVER_MAX_RATE env)
Megabytes a second that all data connections together send at
most, so the dump leaves room on a shared path.
--most-streams=N (absent=64 or QUIVER_MOST_STREAMS env)
While the network holds the dump back, open more data connections,
up to so many, keeping those that carry more; as many as --streams
keeps their number.
--net-timeout=SECONDS (absent=600 or QUIVER_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
--no-data (absent QUIVER_NO_DATA env)
Take every table's schema and no rows.
--objects=WHICH (absent=all or QUIVER_OBJECTS env)
all takes the dumped databases' views, triggers, routines and
events; none takes none of them.
-p PASSWORD, --password=PASSWORD (absent MYSQL_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
-P PORT, --port=PORT (absent QUIVER_PORT env)
The server's TCP port; 3306 unless given.
--read-order=ORDER (absent=largest-chunks or QUIVER_READ_ORDER env)
largest-chunks reads the largest chunks first, wherever their
tables are; largest-tables reads the largest tables first, their
chunks together, so what a sink does after a table's last part (a
tablespace's check, import and keys) starts early.
--reconnect=SECONDS (absent=60. or QUIVER_RECONNECT env)
How long a broken control connection may take to be replaced
before the session fails; the agent keeps its snapshot and sends
again what did not arrive.
--render-threads=N (absent QUIVER_RENDER_THREADS env)
Domains turning batches into output; by default half the readers,
within the cores quiver may use.
-S PATH, --socket=PATH (absent QUIVER_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--snapshot=MODE (absent=ftwrl or QUIVER_SNAPSHOT env)
ftwrl and backup-stage (MariaDB) give one consistent snapshot;
trx-only takes no lock and is only best effort.
--stall=SECONDS (absent=2. or QUIVER_STALL env)
A data connection whose oldest frame waits this long for its
acknowledgement, while another connection waits for none or eight
times less, is taken for a dead path: its frames go to the other
connections and a new one replaces it.
--streams=N (absent QUIVER_STREAMS env)
Data connections; defaults to --threads. Every connection carries
batches of any chunk, so more of them than readers fill a long or
lossy path.
-t N, --threads=N (absent QUIVER_THREADS env)
Database sessions reading in parallel, one domain each; by default
half the cores quiver may use (at least 4), fewer when the server
suggests fewer. quiver uses this machine's fast physical cores
within its CPU quota, half of them when the server is on this
machine.
-T DB.TABLE, --table=DB.TABLE (absent QUIVER_TABLE env)
Dump only DB.TABLE; repeatable. Its database's views, triggers,
routines and events still come along unless --objects none.
--tls=MODE (absent QUIVER_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--tls-ca=FILE (absent QUIVER_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
-u USER, --user=USER (absent=root or QUIVER_USER env)
The account to log in as.
--wire-cert=FILE (absent QUIVER_WIRE_CERT env)
This node's certificate chain, PEM; it must name the host the
connecting side dials, and it serves both ways on a hop.
--wire-key=FILE (absent QUIVER_WIRE_KEY env)
The private key of --wire-cert, PEM.
--wire-peers=FILE (absent QUIVER_WIRE_PEERS env)
PEM certificates a peer's must chain to: the CA of your nodes, or
the peers' own self-signed certificates.
--wire-tls=MODE (absent=verified or QUIVER_WIRE_TLS env)
verified: TLS 1.3 on every connection of quiver's protocol, and
each side must present a certificate the other trusts; disabled:
plaintext, where anyone who reaches the port can send or read the
dump, for private networks only.
--zero-dates=MODE (absent=reject or QUIVER_ZERO_DATES env)
reject fails on a zero or invalid date such as 0000-00-00 or
2020-02-30, which has no exact value; keep writes every date as
stored, and restoring it needs a server that accepts it.
--zstd-level=LEVEL (absent=auto or QUIVER_ZSTD_LEVEL env)
Compression of batches on the wire: a zstd level, or auto to move
between none and zstd -5, 1 and 3 by what holds the dump back.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver agent exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver agent:
MYSQL_PWD
See option --password.
QUIVER_BATCH_BYTES
See option --batch-bytes.
QUIVER_CHUNKS_IN_FLIGHT
See option --chunks-in-flight.
QUIVER_CHUNK_BYTES
See option --chunk-bytes.
QUIVER_COLLECTOR
See option --collector.
QUIVER_CONNECT_TIMEOUT
See option --connect-timeout.
QUIVER_DATABASE
See option --database.
QUIVER_DATA_DATABASE
See option --data-database.
QUIVER_DATA_TABLE
See option --data-table.
QUIVER_DECLARE_TABLES
See option --declare-tables.
QUIVER_DSCP
See option --dscp.
QUIVER_HOST
See option --host.
QUIVER_KEYLESS_READ
See option --keyless-read.
QUIVER_MASK
See option --mask.
QUIVER_MASK_SCRIPT
See option --mask-script.
QUIVER_MASK_SECRET
See option --mask-secret.
QUIVER_MASK_STRICT
See option --mask-strict.
QUIVER_MAX_RATE
See option --max-rate.
QUIVER_MOST_STREAMS
See option --most-streams.
QUIVER_NET_TIMEOUT
See option --net-timeout.
QUIVER_NO_DATA
See option --no-data.
QUIVER_OBJECTS
See option --objects.
QUIVER_PORT
See option --port.
QUIVER_READ_ORDER
See option --read-order.
QUIVER_RECONNECT
See option --reconnect.
QUIVER_RENDER_THREADS
See option --render-threads.
QUIVER_SNAPSHOT
See option --snapshot.
QUIVER_SOCKET
See option --socket.
QUIVER_STALL
See option --stall.
QUIVER_STREAMS
See option --streams.
QUIVER_TABLE
See option --table.
QUIVER_THREADS
See option --threads.
QUIVER_TLS
See option --tls.
QUIVER_TLS_CA
See option --tls-ca.
QUIVER_USER
See option --user.
QUIVER_WIRE_CERT
See option --wire-cert.
QUIVER_WIRE_KEY
See option --wire-key.
QUIVER_WIRE_PEERS
See option --wire-peers.
QUIVER_WIRE_TLS
See option --wire-tls.
QUIVER_ZERO_DATES
See option --zero-dates.
QUIVER_ZSTD_LEVEL
See option --zstd-level.
SEE ALSO
quiver(1)
quiver collect
Receive one dump from quiver agent and write it into
NAME
quiver-collect - Receive one dump from quiver agent and write it into
files (-o) or load it into a server (--to-host and the other --to-
options)
SYNOPSIS
quiver collect [OPTION]…
OPTIONS
--after-load=DB.TABLE=STATEMENT (absent QUIVER_AFTER_LOAD env)
Run DB.TABLE=STATEMENT on the target once every table's rows and
keys are in, before foreign keys, views and triggers, repeatable:
in the target's database of the table, {table} standing for its
name there; a table's statements in the given order, tables at
once. Names take * and ?; a rule naming no table is an error.
--binlog=MODE (absent=write or QUIVER_BINLOG env)
skip keeps the load out of the server's binlog (less work for the
server, but replicas never see the data; needs the right to set
sql_log_bin); write logs it as usual.
--build-threads=N (absent QUIVER_BUILD_THREADS env)
Threads finishing and checking built files, between all tables at
once: the larger a table, the more it borrows; half the fast cores
when not given.
--compress=CODEC (absent=none or QUIVER_COMPRESS env)
zstd compresses every data file, which zstd -d reads back; none
does not.
--csv-dialect=DIALECT (absent=rfc4180 or QUIVER_CSV_DIALECT env)
rfc4180: RFC 4180 with a header line and CRLF; mysql-load-data:
what LOAD DATA reads with backslash escapes and no header.
--datadir=DIR (absent QUIVER_DATADIR env)
The target's data directory as this machine sees it, for --ingest
tablespace; the server's own when not given.
--format=FORMAT (absent=sql or QUIVER_FORMAT env)
sql: INSERT statements myloader and mysql read; csv: one CSV file
per chunk.
--forward=HOST:PORT (absent QUIVER_FORWARD env)
Also pass everything on to this collector as it arrives, so one
read of the source reaches a chain of targets; a chunk counts once
every target committed it.
--ingest=WAY (absent=load-data or QUIVER_INGEST env)
tablespace builds each table it can as an InnoDB tablespace beside
the target and imports it (MySQL 8.4 only, with --binlog skip,
quiver on the target's machine); other tables, and every table
with load-data, go through LOAD DATA.
--listen=HOST:PORT (required)
Where to wait for the agent or the hop before.
-o DIR, --outputdir=DIR (absent QUIVER_OUTPUTDIR env)
Write files here instead of loading into a server.
--reconnect=SECONDS (absent=60. or QUIVER_RECONNECT env)
How long a broken control connection may take to be replaced
before the session fails; the agent keeps its snapshot and sends
again what did not arrive.
--secondary-keys=WHEN (absent=with-table or QUIVER_SECONDARY_KEYS env)
with-table creates tables as dumped; after-data creates them with
their primary key only and adds secondary and foreign keys once a
table's data is in, so InnoDB builds each index by sorting, which
pays off when indexes outgrow memory.
--sort-dir=DIR (absent QUIVER_SORT_DIR env)
Where a table's secondary records wait, sorted, until its file is
finished; beside the file in the target's data directory when not
given.
--statement-bytes=BYTES (absent=1000000 or QUIVER_STATEMENT_BYTES env)
Approximate size of one INSERT statement.
--tablespace-keys=WHO (absent=quiver or QUIVER_TABLESPACE_KEYS env)
Who builds the secondary keys of a table built as a tablespace:
quiver builds those ordered by bytes (integers, dates and times,
decimals, binary strings) into the file, the server adds the
others after the import (the default); server adds them all after
the import.
--to-connect-timeout=SECONDS (absent=10 or QUIVER_TO_CONNECT_TIMEOUT
env)
Fail when connecting, TLS or authentication waits for the server
so long.
--to-database=DB (absent QUIVER_TO_DATABASE env)
Load the only database received under this name.
--to-host=HOST (absent QUIVER_TO_HOST env)
The server's host. As for the mysql client, without it or with
localhost quiver connects to this machine's Unix socket: socket in
the [client] groups of my.cnf, MYSQL_UNIX_PORT, or where packages
put it. With --port or --tls but no host it is 127.0.0.1 over TCP.
--to-net-timeout=SECONDS (absent=600 or QUIVER_TO_NET_TIMEOUT env)
Fail when a connection sends and receives nothing for so long. A
global read lock can wait for long queries, so keep it above the
longest one.
--to-password=PASSWORD (absent QUIVER_TO_PWD env)
The account's password; none when not given. Prefer the
environment variable: a command line is visible to every user of
the machine.
--to-port=PORT (absent QUIVER_TO_PORT env)
The server's TCP port; 3306 unless given.
--to-socket=PATH (absent QUIVER_TO_SOCKET env)
Connect over this Unix socket instead of TCP, as to a server
started with skip_networking; there is no TLS on it, and it takes
no --host, --port or --tls options.
--to-tls=MODE (absent QUIVER_TO_TLS env)
verified encrypts the connection and verifies the server: by the
system's CAs (on 127.0.0.1 only the certificate's validity
period), by --tls-ca or by --tls-fingerprint (the default);
disabled is for local test servers.
--to-tls-ca=FILE (absent QUIVER_TO_TLS_CA env)
PEM file of the CAs the server's chain must lead to; the host name
is checked too, on every host.
--to-tls-fingerprint=SHA256
Accept only the server certificate with this SHA-256 digest, as
openssl x509 -fingerprint -sha256 prints it; chain and host name
are not checked.
--to-user=USER (absent=root or QUIVER_TO_USER env)
The account to log in as.
--wire-cert=FILE (absent QUIVER_WIRE_CERT env)
This node's certificate chain, PEM; it must name the host the
connecting side dials, and it serves both ways on a hop.
--wire-key=FILE (absent QUIVER_WIRE_KEY env)
The private key of --wire-cert, PEM.
--wire-peers=FILE (absent QUIVER_WIRE_PEERS env)
PEM certificates a peer's must chain to: the CA of your nodes, or
the peers' own self-signed certificates.
--wire-tls=MODE (absent=verified or QUIVER_WIRE_TLS env)
verified: TLS 1.3 on every connection of quiver's protocol, and
each side must present a certificate the other trusts; disabled:
plaintext, where anyone who reaches the port can send or read the
dump, for private networks only.
--zstd-level=LEVEL (absent=3 or QUIVER_ZSTD_LEVEL env)
The zstd level of --compress zstd, from 1 (fastest) to 19.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver collect exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver collect:
QUIVER_AFTER_LOAD
See option --after-load.
QUIVER_BINLOG
See option --binlog.
QUIVER_BUILD_THREADS
See option --build-threads.
QUIVER_COMPRESS
See option --compress.
QUIVER_CSV_DIALECT
See option --csv-dialect.
QUIVER_DATADIR
See option --datadir.
QUIVER_FORMAT
See option --format.
QUIVER_FORWARD
See option --forward.
QUIVER_INGEST
See option --ingest.
QUIVER_LISTEN
See option --listen.
QUIVER_OUTPUTDIR
See option --outputdir.
QUIVER_RECONNECT
See option --reconnect.
QUIVER_SECONDARY_KEYS
See option --secondary-keys.
QUIVER_SORT_DIR
See option --sort-dir.
QUIVER_STATEMENT_BYTES
See option --statement-bytes.
QUIVER_TABLESPACE_KEYS
See option --tablespace-keys.
QUIVER_TO_CONNECT_TIMEOUT
See option --to-connect-timeout.
QUIVER_TO_DATABASE
See option --to-database.
QUIVER_TO_HOST
See option --to-host.
QUIVER_TO_NET_TIMEOUT
See option --to-net-timeout.
QUIVER_TO_PORT
See option --to-port.
QUIVER_TO_PWD
See option --to-password.
QUIVER_TO_SOCKET
See option --to-socket.
QUIVER_TO_TLS
See option --to-tls.
QUIVER_TO_TLS_CA
See option --to-tls-ca.
QUIVER_TO_USER
See option --to-user.
QUIVER_WIRE_CERT
See option --wire-cert.
QUIVER_WIRE_KEY
See option --wire-key.
QUIVER_WIRE_PEERS
See option --wire-peers.
QUIVER_WIRE_TLS
See option --wire-tls.
QUIVER_ZSTD_LEVEL
See option --zstd-level.
SEE ALSO
quiver(1)
quiver relay
Pass one dump from quiver agent on to the next hop
NAME
quiver-relay - Pass one dump from quiver agent on to the next hop
(quiver collect or another relay) byte for byte: nothing is
decompressed or written, and a chunk still counts only once the last
hop committed it
SYNOPSIS
quiver relay [OPTION]…
OPTIONS
--forward=HOST:PORT (required)
The next hop.
--listen=HOST:PORT (required)
Where to wait for the agent or the hop before.
--reconnect=SECONDS (absent=60. or QUIVER_RECONNECT env)
How long a broken control connection may take to be replaced
before the session fails; the agent keeps its snapshot and sends
again what did not arrive.
--wire-cert=FILE (absent QUIVER_WIRE_CERT env)
This node's certificate chain, PEM; it must name the host the
connecting side dials, and it serves both ways on a hop.
--wire-key=FILE (absent QUIVER_WIRE_KEY env)
The private key of --wire-cert, PEM.
--wire-peers=FILE (absent QUIVER_WIRE_PEERS env)
PEM certificates a peer's must chain to: the CA of your nodes, or
the peers' own self-signed certificates.
--wire-tls=MODE (absent=verified or QUIVER_WIRE_TLS env)
verified: TLS 1.3 on every connection of quiver's protocol, and
each side must present a certificate the other trusts; disabled:
plaintext, where anyone who reaches the port can send or read the
dump, for private networks only.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver relay exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver relay:
QUIVER_FORWARD
See option --forward.
QUIVER_LISTEN
See option --listen.
QUIVER_RECONNECT
See option --reconnect.
QUIVER_WIRE_CERT
See option --wire-cert.
QUIVER_WIRE_KEY
See option --wire-key.
QUIVER_WIRE_PEERS
See option --wire-peers.
QUIVER_WIRE_TLS
See option --wire-tls.
SEE ALSO
quiver(1)
quiver mask
Try an expression of the masking language on values,
NAME
quiver-mask - Try an expression of the masking language on values,
with no server
SYNOPSIS
quiver mask [OPTION]… EXPRESSION [VALUE]…
THE MASKING LANGUAGE
--mask 'DB.TABLE.COLUMN=expr:EXPRESSION' and (mask "DB.TABLE.COLUMN"
EXPRESSION) in a --mask-script file give each row of the column the
expression's value. It is typed against the table before anything is
read and always ends: there are no loops, and a definition calls only
those above it.
Values are text, integers (64 bits), truth values, dates, dates with
times, and NULL. A function given NULL gives NULL, as in SQL, except
is-null, coalesce, and, or and if, which take NULL as false. A column
of text, binary, JSON, ENUM, SET or DECIMAL is text; one of integers
an integer; DATE a date; DATETIME and TIMESTAMP a date with a time;
other types cannot take an expression (use sql:).
value
The column's value in this row.
(row NAME)
Another column of the row, as read, before any masking.
"text", 42, true, false, null
Constants.
if, cond, let
(if C A B); (cond (C A) … (else B)); (let ((NAME E) …) BODY).
TEXT
Characters are code points: Cyrillic, accents and emoji count one
each.
(concat TEXT …) → text
Joins its texts.
(length TEXT) → integer
How many characters.
(substring TEXT START COUNT) → text
COUNT characters from START, counted from 0; what lies outside is
left out.
(left TEXT COUNT) → text
The first COUNT characters.
(right TEXT COUNT) → text
The last COUNT characters.
(upper TEXT) → text
Latin and Cyrillic letters in upper case.
(lower TEXT) → text
Latin and Cyrillic letters in lower case.
(title TEXT) → text
Each word's first letter in upper case, the others in lower.
(trim TEXT) → text
Without the spaces at either end.
(reverse TEXT) → text
The characters back to front.
(replace TEXT FROM TO) → text
Every FROM, left to right, replaced by TO; an empty FROM changes
nothing.
(before TEXT SEPARATOR) → text
What comes before the first SEPARATOR; the whole text when there
is none.
(after TEXT SEPARATOR) → text
What comes after the first SEPARATOR; empty when there is none.
(before-last TEXT SEPARATOR) → text
As before, at the last SEPARATOR.
(after-last TEXT SEPARATOR) → text
As after, at the last SEPARATOR.
(split-part TEXT SEPARATOR N) → text
The N-th field between SEPARATORs, from 1; empty past the last.
(index-of TEXT PART) → integer
Where PART first starts, in characters from 0; -1 when it is
absent, 0 when it is empty.
(pad-left TEXT WIDTH FILL) → text
FILL's first character put in front until the text is WIDTH
characters long (at most 65535).
(pad-right TEXT WIDTH FILL) → text
As pad-left, at the end.
(repeat TEXT N) → text
N copies, at most 65535; empty past one mebibyte.
(format TEMPLATE [TEXT …]) → text
Each {} of TEMPLATE replaced by the next text, in order; the {}
left over stay.
(translit TEXT) → text
Cyrillic written in Latin letters, the rest kept.
(only-digits TEXT) → text
Only the digits 0 to 9.
(redact TEXT LEFT RIGHT) → text
Each letter and digit but the first LEFT and the last RIGHT
characters becomes *; spaces and punctuation stay.
(str NUMBER) → text
An integer written in decimal.
(int TEXT) → integer
Decimal digits with an optional leading -, spaces around allowed,
as an integer; NULL otherwise.
(starts-with TEXT PREFIX) → truth
Whether the text starts with PREFIX.
(ends-with TEXT SUFFIX) → truth
Whether the text ends with SUFFIX.
(contains TEXT PART) → truth
Whether PART is in the text.
(like TEXT PATTERN) → truth
SQL LIKE, with case: % any characters, _ one, a backslash makes
the next one plain.
(luhn DIGITS) → text
The Luhn check digit of the text's digits, which card numbers end
with.
NUMBERS
Integers of 64 bits. A function given NULL gives NULL.
(+ A B …) → integer
The sum.
(- A [B]) → integer
A minus B, or minus A alone.
(* A B …) → integer
The product.
(/ A B) → integer
The quotient, toward zero; NULL when B is 0.
(mod A B) → integer
The remainder, with A's sign; NULL when B is 0.
(abs A) → integer
Without its sign.
(min VALUE …) → any
The least of values of one kind.
(max VALUE …) → any
The greatest of values of one kind.
(round-to N STEP) → integer
To the nearest multiple of STEP, halves away from zero; NULL when
STEP is not positive.
(floor-to N STEP) → integer
Down to a multiple of STEP.
(clamp N LOW HIGH) → integer
Kept between LOW and HIGH; NULL when LOW is above HIGH.
DATES AND TIMES
A DATE column is a date, DATETIME and TIMESTAMP a date with a time.
Years run from 0000 to 9999; what would leave them is NULL. A time
finer than its column stops the copy.
(year DATE) → integer
The year.
(month DATE) → integer
The month, from 1.
(day DATE) → integer
The day of the month.
(make-date YEAR MONTH DAY) → date
The date; NULL for a day that does not exist.
(add-days DATE DAYS) → date
So many days later, or earlier when negative.
(days-between FROM TO) → integer
Days from FROM to TO.
(date->text DATE) → text
YYYY-MM-DD.
(text->date TEXT) → date
A date from YYYY-MM-DD; NULL otherwise.
(date-of TIME) → date
The day of a time.
(seconds-of-day TIME) → integer
Seconds since that day's midnight.
(make-datetime DATE SECONDS) → time
That day, SECONDS after midnight, from 0 to 86399.
(add-seconds TIME SECONDS) → time
So many seconds later.
(datetime->text TIME) → text
YYYY-MM-DD HH:MM:SS, then the fraction of a second if any.
(text->datetime TEXT) → time
A time from YYYY-MM-DD, HH:MM:SS and up to six fractional digits;
NULL otherwise.
LOGIC
NULL compares to nothing; and, or and if take it as false.
(= A B) → truth
Whether A equals B.
(<> A B) → truth
Whether A differs from B.
(< A B) → truth
Whether A comes before B: text by bytes, numbers and dates by
value.
(<= A B) → truth
Whether A comes before B or equals it.
(> A B) → truth
Whether A comes after B.
(>= A B) → truth
Whether A comes after B or equals it.
(and TRUTH …) → truth
Whether all are true.
(or TRUTH …) → truth
Whether one is true.
(not TRUTH) → truth
The opposite.
(is-null VALUE) → truth
Whether the value is NULL.
(coalesce VALUE …) → any
The first value that is not NULL.
KEYED
Keyed by --mask-secret with HMAC-SHA256: under one key a seed always
gives the same result, in every column and every run, so masked keys
still join; without the key the result tells nothing of the seed. The
examples are under the key of the secret "example".
(shape TEXT) → text
Each letter and digit replaced by one of its kind (Latin,
Cyrillic, digit), any other letter by a Latin one, the rest kept:
as many characters, never more bytes.
(email TEXT) → text
shape before the last @, the domain kept.
(hash TEXT) → text
As many hex digits as the text has characters.
(hash-bytes BYTES) → text
As many bytes as the value: for binary columns.
(hash-int NUMBER) → integer
Any 64-bit integer.
(keyed-int SEED LOW HIGH) → integer
An integer from LOW to HIGH.
(keyed-date SEED FROM TO) → date
A date from FROM to TO.
(keyed-noise N PERCENT) → integer
N moved by up to PERCENT of itself either way, PERCENT from 0 to
100.
(keyed-uuid SEED) → text
A version 4 UUID.
(digits SEED COUNT) → text
COUNT decimal digits, at most 65535.
(pick SEED CHOICE …) → any
One of the choices.
(pick-word SEED WORDS) → text
One line of a (words NAME "FILE") list, named here without quotes.
SCRIPT FORMS
(define (NAME PARAMETER …) BODY)
A function, calling only those above it.
(mask SELECTOR BODY)
SELECTOR is "DB.TABLE.COLUMN" with * and ?, or (where (table
"DB.TABLE") (column "NAME") (type TYPE …)) with any clause left
out, TYPE one of text, binary, json, integer, decimal, float,
date, datetime, timestamp, time, year, enum, set, geometry. A
column takes the first rule naming it.
(keep SELECTOR)
Its columns are copied as they are, and count as decided.
(permute SELECTOR [LOW HIGH])
As --mask DB.TABLE.COLUMN=permute: a keyed bijection, spreading
along foreign keys.
(references "DB.TABLE.COLUMN" "DB.TABLE.COLUMN")
The first column refers to the second though no foreign key says
so.
(strict "DB.TABLE" …)
Every column of these tables must be masked or kept by some rule,
or nothing is read.
(words NAME "FILE")
Lines of FILE, beside the script, for pick-word; # starts a
comment.
(after-load "DB.TABLE" "STATEMENT")
With copy, as --after-load.
ARGUMENTS
EXPRESSION (required)
What a column takes.
VALUE
The column's values, each line of standard input when none is
given; \N is NULL.
OPTIONS
--kind=KIND (absent=text or QUIVER_KIND env)
The column's kind: text (text, binary, JSON, ENUM, SET, DECIMAL),
int, date (DATE) or datetime (DATETIME, TIMESTAMP).
--mask-script=FILE (absent QUIVER_MASK_SCRIPT env)
Definitions the expression may call, repeatable.
--mask-secret=SECRET (absent QUIVER_MASK_SECRET env)
Key permute, shape, email, hash and the keyed functions of expr:
with SECRET, so a value masks the same way in every column and
every run; a random key of this run when not given. Whoever holds
SECRET can undo permute: keep it apart from the copies, one per
environment.
--row=NAME=VALUE (absent QUIVER_ROW env)
Another column of the row, as (row NAME) reads it, repeatable;
NAME:int=, NAME:date= or NAME:datetime= for another kind than
text.
COMMON OPTIONS
--help[=FMT] (default=auto)
Show this help in format FMT. The value FMT must be one of auto,
pager, groff or plain. With auto, the format is pager or plain
whenever the TERM env var is dumb or undefined.
--version
Show version information.
EXIT STATUS
quiver mask exits with:
0 on success.
123 on indiscriminate errors reported on standard error.
124 on command line parsing errors.
125 on unexpected internal errors (bugs).
ENVIRONMENT
These environment variables affect the execution of quiver mask:
QUIVER_KIND
See option --kind.
QUIVER_MASK_SCRIPT
See option --mask-script.
QUIVER_MASK_SECRET
See option --mask-secret.
QUIVER_ROW
See option --row.
EXAMPLES
(define (phone p) (concat "+7 (9" (digits p 2) ") " (digits (concat p "/") 3) "-" (right p 4)))
(words names "names.txt")
(strict "shop.users")
(keep "shop.users.id")
(mask "shop.users.phone" (phone value))
(mask "shop.users.name" (concat (pick-word value names) " " (shape (after value " "))))
(mask "shop.users.born" (keyed-date (str (row id)) (make-date (year value) 1 1) (make-date (year value) 12 31)))
(mask (where (table "shop.*") (column "*email*") (type text)) (email value))
(mask (where (table "shop.users") (type binary)) (hash-bytes value))
SEE ALSO
quiver(1)