The port's canonical values in the carriers the rest of pg-datahike already uses.
The parser returns PostgreSQL's own internal form -- a day count, or
microseconds from an epoch. Everything downstream expects
java.time.LocalDate, LocalTime, OffsetTime, LocalDateTime or
java.util.Date, with two established conventions for the values
those classes cannot hold. This namespace is the one place the two
meet, and it is deliberately thin: the conversion is where precision
is lost, so it should be visible rather than spread across call
sites.
THREE THINGS THE CARRIERS CANNOT HOLD, each with a convention that predates this port:
24:00:00 is a legal PostgreSQL time and java.time.LocalTime has
no end-of-day value. It is carried as the TEXT "24:00:00", which
stores, renders and orders correctly because time values are kept
as their canonical text.
The two infinities have no java.util.Date room, so they are the
sentinels types/pos-infinity and types/neg-infinity.
MICROSECONDS survive LocalDateTime, LocalTime and
OffsetDateTime, and do NOT survive java.util.Date, which is
millisecond-only. The parser is not the lossy step and never was --
this conversion is, and only on the paths that must hand back a
Date.
The port's canonical values in the carriers the rest of pg-datahike already uses. The parser returns PostgreSQL's own internal form -- a day count, or microseconds from an epoch. Everything downstream expects `java.time.LocalDate`, `LocalTime`, `OffsetTime`, `LocalDateTime` or `java.util.Date`, with two established conventions for the values those classes cannot hold. This namespace is the one place the two meet, and it is deliberately thin: the conversion is where precision is lost, so it should be visible rather than spread across call sites. THREE THINGS THE CARRIERS CANNOT HOLD, each with a convention that predates this port: `24:00:00` is a legal PostgreSQL time and `java.time.LocalTime` has no end-of-day value. It is carried as the TEXT `"24:00:00"`, which stores, renders and orders correctly because time values are kept as their canonical text. The two infinities have no `java.util.Date` room, so they are the sentinels `types/pos-infinity` and `types/neg-infinity`. MICROSECONDS survive `LocalDateTime`, `LocalTime` and `OffsetDateTime`, and do NOT survive `java.util.Date`, which is millisecond-only. The parser is not the lossy step and never was -- this conversion is, and only on the paths that must hand back a `Date`.
The leaf decoders of PostgreSQL's datetime parser: the small functions
DecodeDateTime and DecodeTimeOnly call to turn ONE lexed field
into numbers. They are separated from the state machine because each
is independently testable and several have rules no one would guess.
SIGN CONVENTION. PostgreSQL carries a zone offset internally as
SECONDS WEST of Greenwich -- DecodeTimezone computes seconds east
and then stores *tzp = -tz (datetime.c:3060). The abbreviation
table, and tznames/Default that it comes from, are SECONDS EAST.
Everything here is named for which one it is, because a sign error
between them is a silent double offset rather than a failure.
ERRORS are ex-info carrying ::dterr, mirroring the C's DTERR_*
returns. The entry points map those to SQLSTATEs; nothing here knows
about SQL.
The leaf decoders of PostgreSQL's datetime parser: the small functions `DecodeDateTime` and `DecodeTimeOnly` call to turn ONE lexed field into numbers. They are separated from the state machine because each is independently testable and several have rules no one would guess. SIGN CONVENTION. PostgreSQL carries a zone offset internally as SECONDS WEST of Greenwich -- `DecodeTimezone` computes seconds east and then stores `*tzp = -tz` (datetime.c:3060). The abbreviation table, and `tznames/Default` that it comes from, are SECONDS EAST. Everything here is named for which one it is, because a sign error between them is a silent double offset rather than a failure. ERRORS are `ex-info` carrying `::dterr`, mirroring the C's DTERR_* returns. The entry points map those to SQLSTATEs; nothing here knows about SQL.
The five input functions: date_in, time_in, timetz_in,
timestamp_in, timestamptz_in.
Each is the same three steps -- lex, decode, convert -- differing in the work-buffer size, which decoder it calls, whether it accepts a zone, and what it does with the parts it does not want. Those differences are small and every one of them is observable, so the five are written out rather than generated from a table.
date_in is the one that surprises: it runs the WHOLE timestamp
decoder and then discards the time. So the time is still validated
and '2001-02-03 25:00:00'::date is an error, not 2001-02-03
(date.c:117-170).
ALL FIVE pass a non-NULL tzp, so all five ACCEPT a zone --
date_in at date.c:134 and time_in at date.c:1399. The ones that
have nowhere to put it parse it and discard it, which is why
'2000-01-01 12:00:00 PST'::date is 2000-01-01 and ::time is
12:00:00 rather than errors. I had both refusing, on the assumption
that a type with no zone would not take one; tzp == NULL is for
other internal callers, not for the input functions.
ERRORS. Every dterr becomes the SQLSTATE DateTimeParseError
gives it (datetime.c:4063-4120):
bad-format 22007 invalid input syntax for type … field-overflow 22008 date/time field value out of range md-field-overflow 22008 …plus a "datestyle" HINT tzdisp-overflow 22009 time zone displacement out of range bad-timezone 22023 time zone "…" not recognized
The three overflow codes are distinct and are routinely got wrong: a zone past ±15 is 22009, not 22008.
The five input functions: `date_in`, `time_in`, `timetz_in`, `timestamp_in`, `timestamptz_in`. Each is the same three steps -- lex, decode, convert -- differing in the work-buffer size, which decoder it calls, whether it accepts a zone, and what it does with the parts it does not want. Those differences are small and every one of them is observable, so the five are written out rather than generated from a table. `date_in` is the one that surprises: it runs the WHOLE timestamp decoder and then discards the time. So the time is still validated and `'2001-02-03 25:00:00'::date` is an error, not 2001-02-03 (date.c:117-170). ALL FIVE pass a non-NULL `tzp`, so all five ACCEPT a zone -- `date_in` at date.c:134 and `time_in` at date.c:1399. The ones that have nowhere to put it parse it and discard it, which is why `'2000-01-01 12:00:00 PST'::date` is 2000-01-01 and `::time` is 12:00:00 rather than errors. I had both refusing, on the assumption that a type with no zone would not take one; `tzp == NULL` is for other internal callers, not for the input functions. ERRORS. Every `dterr` becomes the SQLSTATE `DateTimeParseError` gives it (datetime.c:4063-4120): bad-format 22007 invalid input syntax for type … field-overflow 22008 date/time field value out of range md-field-overflow 22008 …plus a "datestyle" HINT tzdisp-overflow 22009 time zone displacement out of range bad-timezone 22023 time zone "…" not recognized The three overflow codes are distinct and are routinely got wrong: a zone past ±15 is 22009, not 22008.
ParseDateTime (datetime.c:753-977): the datetime LEXER.
It splits a literal into fields and tags each with a TYPE, without
deciding what any of them means. That split -- lex, then interpret --
is the whole reason this exists rather than a regex: the meaning of a
field depends on the other fields present and on DateStyle, and a
pattern cannot carry that. A bare three-digit number is a day-of-year
if a year is already set and a month otherwise; 01/02/03 is three
different dates; Mon Feb 10 17:32:01 1997 PST has a word that is a
month, a word that is a weekday to discard, and a word that is a zone.
Six field types, and the doc comment at datetime.c:740-752 is worth repeating because it is the only place that says the quiet part -- several of them hold things their names do not suggest:
:number digits and possibly a decimal point. ALSO holds a date:
yy.ddd.
:string text with no digits or punctuation. ALSO holds months
(january) and zone abbreviations (pst).
:date digits with two delimiters, or digits and text. ALSO
holds zone NAMES: america/new_york, gmt-8.
:time digits with colon delimiters, possibly a decimal point.
:tz a leading + or - then digits (and it eats :, .
and - as well -- see the note on that below).
:special a leading + or - then text.
Alpha runs are lowercased; digits and punctuation are not, and the
fold goes through tokens/ascii-lower rather than
clojure.string/lower-case.
That assumes a UTF-8 database, and the assumption is worth stating.
pg_tolower also folds bytes with the high bit set when isupper
says to, and the C's isalpha likewise respects LC_CTYPE -- but in
a UTF-8 locale every single byte over 0x7F is false for both, so the
ASCII-only fold agrees exactly. In a single-byte database (LATIN1
with a matching LC_CTYPE) PostgreSQL would lex 1-Été-2000 and we
reject it. pg-datahike serves UTF-8 only, so that case is
unreachable here; it is recorded because the reason it is safe is
not visible in the code.
Punctuation that is not part of a field is DISCARDED as a delimiter
(datetime.c:929-934). So a double-quoted zone in a literal is not a
quoted anything: '2000-01-01 12:00 "PST"' lexes exactly like
… PST, because " is punctuation. There is no quoted-zone concept
at the literal level at all.
Returns a vector of {:text :type}, or throws bad-format.
`ParseDateTime` (datetime.c:753-977): the datetime LEXER.
It splits a literal into fields and tags each with a TYPE, without
deciding what any of them means. That split -- lex, then interpret --
is the whole reason this exists rather than a regex: the meaning of a
field depends on the other fields present and on DateStyle, and a
pattern cannot carry that. A bare three-digit number is a day-of-year
if a year is already set and a month otherwise; `01/02/03` is three
different dates; `Mon Feb 10 17:32:01 1997 PST` has a word that is a
month, a word that is a weekday to discard, and a word that is a zone.
Six field types, and the doc comment at datetime.c:740-752 is worth
repeating because it is the only place that says the quiet part --
several of them hold things their names do not suggest:
:number digits and possibly a decimal point. ALSO holds a date:
`yy.ddd`.
:string text with no digits or punctuation. ALSO holds months
(`january`) and zone abbreviations (`pst`).
:date digits with two delimiters, or digits and text. ALSO
holds zone NAMES: `america/new_york`, `gmt-8`.
:time digits with colon delimiters, possibly a decimal point.
:tz a leading `+` or `-` then digits (and it eats `:`, `.`
and `-` as well -- see the note on that below).
:special a leading `+` or `-` then text.
Alpha runs are lowercased; digits and punctuation are not, and the
fold goes through `tokens/ascii-lower` rather than
`clojure.string/lower-case`.
That assumes a UTF-8 database, and the assumption is worth stating.
`pg_tolower` also folds bytes with the high bit set when `isupper`
says to, and the C's `isalpha` likewise respects LC_CTYPE -- but in
a UTF-8 locale every single byte over 0x7F is false for both, so the
ASCII-only fold agrees exactly. In a single-byte database (LATIN1
with a matching LC_CTYPE) PostgreSQL would lex `1-Été-2000` and we
reject it. pg-datahike serves UTF-8 only, so that case is
unreachable here; it is recorded because the reason it is safe is
not visible in the code.
Punctuation that is not part of a field is DISCARDED as a delimiter
(datetime.c:929-934). So a double-quoted zone in a literal is not a
quoted anything: `'2000-01-01 12:00 "PST"'` lexes exactly like
`… PST`, because `"` is punctuation. There is no quoted-zone concept
at the literal level at all.
Returns a vector of `{:text :type}`, or throws `bad-format`.DecodeDateTime (datetime.c:979-1534): the state machine that turns
lexed fields into a date and time.
It walks the fields once, left to right, and each field CLAIMS the
parts of the result it set. A field that claims something already
claimed is an error -- that single rule is what makes
'2001-02-03 +05 +06' and 'Feb Feb 10 1997' errors rather than
last-one-wins, and it is why fmask and tmask are both needed:
one is everything set so far, the other is what this field set.
Two pieces of state carry ACROSS fields and are the reason this cannot be a fold over independent fields:
ptype is a prefix set by a UNITS or ISOTIME token that changes how
the NEXT field is read. J makes the next number a Julian day, T
makes it a time. It must be consumed by the following field --
a literal ending with a dangling J is an error.
text-month? records that a month arrived as a word, which changes
DecodeNumber's placement rules and enables a retroactive swap.
The zone is deliberately NOT resolved during the walk. An
abbreviation, a zone name and the session default all need the
DATE to resolve against -- an offset is a function of the instant --
so the walk only records WHICH zone, and the resolution happens at
the end, after ValidateDate. Getting this backwards resolves the
zone against the wrong day across a DST boundary.
`DecodeDateTime` (datetime.c:979-1534): the state machine that turns lexed fields into a date and time. It walks the fields once, left to right, and each field CLAIMS the parts of the result it set. A field that claims something already claimed is an error -- that single rule is what makes `'2001-02-03 +05 +06'` and `'Feb Feb 10 1997'` errors rather than last-one-wins, and it is why `fmask` and `tmask` are both needed: one is everything set so far, the other is what this field set. Two pieces of state carry ACROSS fields and are the reason this cannot be a fold over independent fields: `ptype` is a prefix set by a UNITS or ISOTIME token that changes how the NEXT field is read. `J` makes the next number a Julian day, `T` makes it a time. It must be consumed by the following field -- a literal ending with a dangling `J` is an error. `text-month?` records that a month arrived as a word, which changes `DecodeNumber`'s placement rules and enables a retroactive swap. The zone is deliberately NOT resolved during the walk. An abbreviation, a zone name and the session default all need the DATE to resolve against -- an offset is a function of the instant -- so the walk only records WHICH zone, and the resolution happens at the end, after `ValidateDate`. Getting this backwards resolves the zone against the wrong day across a DST boundary.
PostgreSQL's datetime token tables, transcribed from the pinned 17.7.
TWO tables, and they must stay two. datetktbl (datetime.c:105-179)
is compiled into the backend and holds months, weekdays, am/pm, ad/bc,
the RESERV words (now, today, epoch, infinity, allballs) and
the ISO unit letters. The ZONE ABBREVIATIONS are not in it --
datetime.c:100-103 says so outright: "The static table contains no TZ,
DTZ, or DYNTZ entries; rather those are loaded from configuration
files" -- they come from src/timezone/tznames/Default, and
DecodeTimezoneAbbrev is consulted BEFORE DecodeSpecial
(datetime.c:1304-1309).
That order is not negotiable, and the comment at datetime.c:3203-3208
says why: tzdb deliberately contains zone NAMES identical to offset
ABBREVIATIONS. With the Default set the two token sets happen not to
collide at all (checked below), so the precedence is currently a
no-op -- it stops being one the moment timezone_abbreviations is
settable, which is why it is written down rather than relied upon.
An abbreviation is a FIXED OFFSET; a zone NAME is a RULE. pst here
is -28800 seconds, always, with no DST -- while PST8PDT is a tzdb
LINK to America/Los_Angeles and does observe DST. Resolving an
abbreviation through ZoneId/of with SHORT_IDS (which maps PST to
America/Los_Angeles) gets every July timestamp wrong by an hour, with
no error, which is what this table exists to stop.
Keys are LOWERCASE. ParseDateTime lowercases every alpha run with
pg_tolower, which is ASCII-only (pgstrcasecmp.c:122-129), and
tzparser.c:79-83 lowercases every abbreviation with the comment
"must match datetime.c's conversion". So a lookup must fold ASCII
only -- never clojure.string/lower-case, which is locale-sensitive
and breaks under a Turkish default locale.
GENERATED -- the extraction commands are in the comment below.
PostgreSQL's datetime token tables, transcribed from the pinned 17.7. TWO tables, and they must stay two. `datetktbl` (datetime.c:105-179) is compiled into the backend and holds months, weekdays, am/pm, ad/bc, the RESERV words (`now`, `today`, `epoch`, `infinity`, `allballs`) and the ISO unit letters. The ZONE ABBREVIATIONS are not in it -- datetime.c:100-103 says so outright: "The static table contains no TZ, DTZ, or DYNTZ entries; rather those are loaded from configuration files" -- they come from `src/timezone/tznames/Default`, and `DecodeTimezoneAbbrev` is consulted BEFORE `DecodeSpecial` (datetime.c:1304-1309). That order is not negotiable, and the comment at datetime.c:3203-3208 says why: tzdb deliberately contains zone NAMES identical to offset ABBREVIATIONS. With the `Default` set the two token sets happen not to collide at all (checked below), so the precedence is currently a no-op -- it stops being one the moment `timezone_abbreviations` is settable, which is why it is written down rather than relied upon. An abbreviation is a FIXED OFFSET; a zone NAME is a RULE. `pst` here is -28800 seconds, always, with no DST -- while `PST8PDT` is a tzdb LINK to America/Los_Angeles and does observe DST. Resolving an abbreviation through `ZoneId/of` with `SHORT_IDS` (which maps PST to America/Los_Angeles) gets every July timestamp wrong by an hour, with no error, which is what this table exists to stop. Keys are LOWERCASE. `ParseDateTime` lowercases every alpha run with `pg_tolower`, which is ASCII-only (pgstrcasecmp.c:122-129), and `tzparser.c:79-83` lowercases every abbreviation with the comment "must match datetime.c's conversion". So a lookup must fold ASCII only -- never `clojure.string/lower-case`, which is locale-sensitive and breaks under a Turkish default locale. GENERATED -- the extraction commands are in the comment below.
DetermineTimeZoneOffset (datetime.c:1575-1733): a local date and
time plus a zone to an OFFSET.
This is a separate step from parsing because an offset is a function of the INSTANT, and the instant is only known once the date has been decoded and validated.
THE AMBIGUITY RULE IS THE OPPOSITE OF java.time's, in both
directions, and this is the whole reason the namespace exists rather
than a call to ZonedDateTime/of:
spring forward (a GAP, the local time never happened) PostgreSQL takes the BEFORE offset and keeps the local fields java.time shifts the time forward and takes the AFTER offset
fall back (an OVERLAP, the local time happened twice) PostgreSQL takes the AFTER offset java.time takes the earlier, BEFORE offset
datetime.c:1697-1706 states the rule and why it is not phrased as "prefer standard time": that older rule could not resolve zones where both readings report as standard time -- Europe/Moscow in October 2014 -- and in zones like Europe/Dublin there is widespread disagreement about which offset is "standard" at all.
An hour wrong, twice a year, with no error, is exactly the failure this port is meant to remove, so it is implemented from the C's branches rather than from the convenience method.
SIGN: everything here returns and accepts SECONDS WEST, as PostgreSQL stores internally.
`DetermineTimeZoneOffset` (datetime.c:1575-1733): a local date and
time plus a zone to an OFFSET.
This is a separate step from parsing because an offset is a function
of the INSTANT, and the instant is only known once the date has been
decoded and validated.
THE AMBIGUITY RULE IS THE OPPOSITE OF `java.time`'s, in both
directions, and this is the whole reason the namespace exists rather
than a call to `ZonedDateTime/of`:
spring forward (a GAP, the local time never happened)
PostgreSQL takes the BEFORE offset and keeps the local fields
java.time shifts the time forward and takes the AFTER offset
fall back (an OVERLAP, the local time happened twice)
PostgreSQL takes the AFTER offset
java.time takes the earlier, BEFORE offset
datetime.c:1697-1706 states the rule and why it is not phrased as
"prefer standard time": that older rule could not resolve zones
where both readings report as standard time -- Europe/Moscow in
October 2014 -- and in zones like Europe/Dublin there is widespread
disagreement about which offset is "standard" at all.
An hour wrong, twice a year, with no error, is exactly the failure
this port is meant to remove, so it is implemented from the C's
branches rather than from the convenience method.
SIGN: everything here returns and accepts SECONDS WEST, as
PostgreSQL stores internally.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |