The masking language

A policy for masking columns as rows are read, in S-expressions. quiver types it against every column before anything is read, compiles it and runs it on its render threads.

quiver copy … --mask-script policy.scm --mask-secret "$MASK_SECRET"
quiver copy … --mask 'shop.users.inn=expr:(hash-int value)'
quiver mask --mask-script policy.scm '(phone value)' '+7 (912) 345-67-89'

Values

A column of text, binary, JSON, ENUM, SET or DECIMAL is text; one of integers an integer; DATE a date; DATETIME and TIMESTAMP a time; other types take sql: instead. Every kind has NULL, which goes through functions as in SQL. A value given to a column of another kind, NULL in a NOT NULL column, or a time finer than its column stops the copy.

Forms

A SELECTOR is "DB.TABLE.COLUMN" with * and ?, names in backquotes taken as written, or (where (table "DB.TABLE") (column "NAME") (type TYPE …)) with any clause left out; TYPE is one of text, binary, json, integer, decimal, float, date, datetime, timestamp, time, year, enum, set, geometry.

Functions

• concat
• length
• substring
• left
• right
• upper
• lower
• title
• trim
• reverse
• replace
• before
• after
• before-last
• after-last
• split-part
• index-of
• pad-left
• pad-right
• repeat
• format
• translit
• only-digits
• redact
• str
• int
• starts-with
• ends-with
• contains
• like
• luhn
• +
• -
• *
• /
• mod
• abs
• min
• max
• round-to
• floor-to
• clamp
• year
• month
• day
• make-date
• add-days
• days-between
• date->text
• text->date
• date-of
• seconds-of-day
• make-datetime
• add-seconds
• datetime->text
• text->datetime
• =
• <>
• <
• <=
• >
• >=
• and
• or
• not
• is-null
• coalesce
• shape
• email
• hash
• hash-bytes
• hash-int
• keyed-int
• keyed-date
• keyed-noise
• keyed-uuid
• digits
• pick
• pick-word

Text

Characters are code points: Cyrillic, accents and emoji count one each.

(concat TEXT …) → text

Joins its texts.

(length TEXT) → integer

How many characters.

(substring TEXT START COUNT) → text

COUNT characters from START, counted from 0; what lies outside is left out.

(left TEXT COUNT) → text

The first COUNT characters.

(right TEXT COUNT) → text

The last COUNT characters.

(upper TEXT) → text

Latin and Cyrillic letters in upper case.

(lower TEXT) → text

Latin and Cyrillic letters in lower case.

(title TEXT) → text

Each word's first letter in upper case, the others in lower.

(trim TEXT) → text

Without the spaces at either end.

(reverse TEXT) → text

The characters back to front.

(replace TEXT FROM TO) → text

Every FROM, left to right, replaced by TO; an empty FROM changes nothing.

(before TEXT SEPARATOR) → text

What comes before the first SEPARATOR; the whole text when there is none.

(after TEXT SEPARATOR) → text

What comes after the first SEPARATOR; empty when there is none.

(before-last TEXT SEPARATOR) → text

As before, at the last SEPARATOR.

(after-last TEXT SEPARATOR) → text

As after, at the last SEPARATOR.

(split-part TEXT SEPARATOR N) → text

The N-th field between SEPARATORs, from 1; empty past the last.

(index-of TEXT PART) → integer

Where PART first starts, in characters from 0; -1 when it is absent, 0 when it is empty.

(pad-left TEXT WIDTH FILL) → text

FILL's first character put in front until the text is WIDTH characters long (at most 65535).

(pad-right TEXT WIDTH FILL) → text

As pad-left, at the end.

(repeat TEXT N) → text

N copies, at most 65535; empty past one mebibyte.

(format TEMPLATE [TEXT …]) → text

Each {} of TEMPLATE replaced by the next text, in order; the {} left over stay.

(translit TEXT) → text

Cyrillic written in Latin letters, the rest kept.

(only-digits TEXT) → text

Only the digits 0 to 9.

(redact TEXT LEFT RIGHT) → text

Each letter and digit but the first LEFT and the last RIGHT characters becomes *; spaces and punctuation stay.

(str NUMBER) → text

An integer written in decimal.

(int TEXT) → integer

Decimal digits with an optional leading -, spaces around allowed, as an integer; NULL otherwise.

(starts-with TEXT PREFIX) → truth

Whether the text starts with PREFIX.

(ends-with TEXT SUFFIX) → truth

Whether the text ends with SUFFIX.

(contains TEXT PART) → truth

Whether PART is in the text.

(like TEXT PATTERN) → truth

SQL LIKE, with case: % any characters, _ one, a backslash makes the next one plain.

(luhn DIGITS) → text

The Luhn check digit of the text's digits, which card numbers end with.

Numbers

Integers of 64 bits. A function given NULL gives NULL.

(+ A B …) → integer

The sum.

(- A [B]) → integer

A minus B, or minus A alone.

(* A B …) → integer

The product.

(/ A B) → integer

The quotient, toward zero; NULL when B is 0.

(mod A B) → integer

The remainder, with A's sign; NULL when B is 0.

(abs A) → integer

Without its sign.

(min VALUE …) → any

The least of values of one kind.

(max VALUE …) → any

The greatest of values of one kind.

(round-to N STEP) → integer

To the nearest multiple of STEP, halves away from zero; NULL when STEP is not positive.

(floor-to N STEP) → integer

Down to a multiple of STEP.

(clamp N LOW HIGH) → integer

Kept between LOW and HIGH; NULL when LOW is above HIGH.

Dates and times

A DATE column is a date, DATETIME and TIMESTAMP a date with a time. Years run from 0000 to 9999; what would leave them is NULL. A time finer than its column stops the copy.

(year DATE) → integer

The year.

(month DATE) → integer

The month, from 1.

(day DATE) → integer

The day of the month.

(make-date YEAR MONTH DAY) → date

The date; NULL for a day that does not exist.

(add-days DATE DAYS) → date

So many days later, or earlier when negative.

(days-between FROM TO) → integer

Days from FROM to TO.

(date->text DATE) → text

YYYY-MM-DD.

(text->date TEXT) → date

A date from YYYY-MM-DD; NULL otherwise.

(date-of TIME) → date

The day of a time.

(seconds-of-day TIME) → integer

Seconds since that day's midnight.

(make-datetime DATE SECONDS) → time

That day, SECONDS after midnight, from 0 to 86399.

(add-seconds TIME SECONDS) → time

So many seconds later.

(datetime->text TIME) → text

YYYY-MM-DD HH:MM:SS, then the fraction of a second if any.

(text->datetime TEXT) → time

A time from YYYY-MM-DD, HH:MM:SS and up to six fractional digits; NULL otherwise.

Logic

NULL compares to nothing; and, or and if take it as false.

(= A B) → truth

Whether A equals B.

(<> A B) → truth

Whether A differs from B.

(< A B) → truth

Whether A comes before B: text by bytes, numbers and dates by value.

(<= A B) → truth

Whether A comes before B or equals it.

(> A B) → truth

Whether A comes after B.

(>= A B) → truth

Whether A comes after B or equals it.

(and TRUTH …) → truth

Whether all are true.

(or TRUTH …) → truth

Whether one is true.

(not TRUTH) → truth

The opposite.

(is-null VALUE) → truth

Whether the value is NULL.

(coalesce VALUE …) → any

The first value that is not NULL.

Keyed

Keyed by --mask-secret with HMAC-SHA256: under one key a seed always gives the same result, in every column and every run, so masked keys still join; without the key the result tells nothing of the seed. The examples are under the key of the secret "example".

(shape TEXT) → text

Each letter and digit replaced by one of its kind (Latin, Cyrillic, digit), any other letter by a Latin one, the rest kept: as many characters, never more bytes.

(email TEXT) → text

shape before the last @, the domain kept.

(hash TEXT) → text

As many hex digits as the text has characters.

(hash-bytes BYTES) → text

As many bytes as the value: for binary columns.

(hash-int NUMBER) → integer

Any 64-bit integer.

(keyed-int SEED LOW HIGH) → integer

An integer from LOW to HIGH.

(keyed-date SEED FROM TO) → date

A date from FROM to TO.

(keyed-noise N PERCENT) → integer

N moved by up to PERCENT of itself either way, PERCENT from 0 to 100.

(keyed-uuid SEED) → text

A version 4 UUID.

(digits SEED COUNT) → text

COUNT decimal digits, at most 65535.

(pick SEED CHOICE …) → any

One of the choices.

(pick-word SEED WORDS) → text

One line of a (words NAME "FILE") list, named here without quotes.