Liking cljdoc? Tell your friends :D

Consolidation plan: types, functions and translation

Status: 2026-09-19. A living plan; update it as phases land.

Inputs:

  • the function-coverage sweep (bb fncov);
  • three per-domain gap reports and two implementation audits (.internal/fncov-reports/, .internal/audit/; local, not committed; file:line references there drift);
  • the leak patterns found while fixing #177–#180.

Every finding marked verified below was reproduced against the PostgreSQL 17.7 oracle.

Why things slipped through

  1. Class-dispatched behaviour. Several PostgreSQL types share one JVM carrier:

    • money and numeric are both BigDecimal;
    • time and interval are kept as text;
    • date, timestamp and timestamptz are all instants;
    • bytea is hex text.

    Code that decides by the value's class instead of the static type is wrong for at least one of them.

  2. Passthrough on failure. (catch … v), (str e) and (or (parse s) s) keep an unparsed value flowing as text wearing a type. A parse that correctly yields false is also taken for a failure.

  3. Identity stubs accepted for syntax: AT TIME ZONE was one; has_*_privilege always answers true.

  4. Duplicated paths that drift:

    • 8 text renderers;
    • 9 timestamp/date parsers, 7 boolean parsers, ~15 type-name tables;
    • 4 LIKE/regex implementations, 3 ANY/ALL paths;
    • a second expression evaluator for UPDATE SET, RETURNING and CHECK;
    • 3 settings stores and 10 function-lookup routes.
  5. Shortcuts that bypass the translator: classify.clj single-function SELECTs, shape.clj catalog probes, lexical literal templating.

  6. Tests asserting our output instead of PostgreSQL's. Admitted money.sql "strict slices" expected 79.83 where upstream says $79.83.

The remedy is structural rather than a list of patches:

  • one registry per type, keyed by OID (input, output, binary, compare);
  • one function-signature registry generated from pg_proc.dat;
  • one expression evaluator;
  • no fallbacks that return their input;
  • expectations taken only from the oracle or upstream .out files.

Principles for every change

  • Closed resolution. Functions, operators, casts and types resolve against tables generated from the pinned catalog (pg_proc.dat, pg_cast.dat, pg_operator.dat, pg_type.dat). Unknown means PostgreSQL's error (42883 / 42725 / 42704), never "try something". The catalog is src/datahike/pg/pg_catalog.edn, generated by bb gen-catalog; types/explicit-casts (#177) and types/aggregate-resolution (#180) read it.
  • Type-directed values. Dispatch on the static OID first; class only when the type is unknown. Never .toString a Java value as PostgreSQL output.
  • No passthrough. A value that fails input raises the type's SQLSTATE (22P02 / 22007 / 22008 / 22003).
  • One implementation per semantics. When a second path is found, it calls the first or is deleted.
  • Expectations from PostgreSQL. New tests take expected values from the oracle or from upstream expected/*.out, never from our output.

Phase 0 — stop verified silent wrong answers

These corrupt data or return wrong rows today, and they come before feature work. The items are ordered by dependency.

0.0 Catalog tables are generated, with a drift test. (Done: #182.) explicit-casts and aggregate-resolution are pasted snapshots today. Commit the generator (it reads pg_proc.dat, pg_cast.dat, pg_type.dat and pg_operator.dat from a pinned PostgreSQL checkout) and a test that fails when a table and its generator disagree. Keep an explicit extension manifest for names that PostgreSQL doesn't have: pgvector distances, period predicates, uuidv4/uuidv7, date_add/date_diff. Closed resolution needs both.

#ItemVerified behaviourFixGate
0.1Scalar input functions (done: #183) (bool, int2/int4/int8, oid, float4/float8, numeric, uuid). This is Phase 1.3 pulled forward for the non-datetime types.verified:
• WHERE b = 'false' / 'no' → 0 rows;
• WHERE i = 'abc' → 0 rows instead of 22P02;
• '1.5'::int → 1 instead of 22P02, and the error names numeric;
• '0x1F', '1_000' rejected (PostgreSQL 16+ accepts them);
• the Java text-parameter decoder turns every bool other than t/true/1 into false;
• int2/int4 have no 22003
One input function per OID, ported from int.c/int8.c/bool.c/float.c/numeric.c/uuid.c, raising 22P02 or 22003. Route every path through it: unknown literals, casts from text/unknown, INSERT/UPDATE, text parameters (PgParamCodec calls in or mirrors it, with a shared test vector) and array elements. coerce-unknown stops returning its input. Datetime types wait for the decoder in 1.3.oracle matrix {bool,int2,int4,int8,oid,float8,numeric,uuid} × valid/invalid/edge inputs × {cast, WHERE literal, INSERT, text parameter, array element}; fuzzer literal class
0.2CHECK and DEFAULT evaluation (done: #184; constraint identity #186)verified: CHECK (name > 'm'), CHECK (d < '2030-01-01') and CHECK (s LIKE 'a%') accept violating rows. The CHECK evaluator treats anything it doesn't recognise as true, compares only numbers, and swallows exceptions. eval-default exists twice and has drifted (now differs between the copies, and DEFAULT '123' on a text column is stored as a Long). The DDL regex evaluator still folds now() AT TIME ZONE … to now.Compile a CHECK or DEFAULT once through the expression translator, then bind each row's values. The result is three-valued (only false rejects) and errors propagate. One eval-default. Domain CHECKs use the same path.CHECK over every operator × type must reject what PostgreSQL rejects; DEFAULT types match the oracle's pg_typeof
0.3One evaluator for UPDATE SET / RETURNING (steps 1–2 done: #187, #188)verified:
• SET s = s \|\| '[1]' stores ["b", 1] (jsonb chosen because parse-jsonb happened to succeed);
• RETURNING i + 1 → NULL;
• interpret-form uses Clojure arithmetic (/ returns a Ratio, division by zero gives nil) and treats NULL = NULL as true
Two primitives on the SELECT translator, as PostgreSQL has one expression compiler used two ways. Step 1 (#187): a row scope — the written row as a relation whose columns are typed placeholders — carries RETURNING, CHECK, domain CHECK and ON CONFLICT … WHERE (ExecCheck / ExecQual / ExecProcessReturning). Step 2 (#188): plain UPDATE is SELECT db_id, <set exprs> FROM t WHERE … run by the SELECT executor (preprocess_targetlist / ExecGetUpdateNewTuple). Step 3 (#191): UPDATE … FROM is one joined query. Steps 4–6: ON CONFLICT SET is a row-scope projection over the conflicting row and excluded, an UPDATE's WITH clause rides into its SET query, and SET (a,b) = (subquery) assigns per column — eval-update-expr / eval-update-cond are deleted.fuzzer: UPDATE … SET c = E RETURNING c ≡ SELECT E; pgbench tps must not drop
0.4Closed function resolution (done: #199)verified: an unknown SQL name calls any resolvable Clojure var — slurp('/etc/passwd') read the file, spit('/tmp/x','y') wrote it. The decorated forms resolved nothing at all: f(x) OVER (…) built a window spec for any name, and f(x) FILTER (WHERE …) fell through to a default aggregate, so nosuchfn(a) FILTER (WHERE true) answered a COUNT. A FROM-clause function took the last dot-separated segment as its name, so nosuchschema.unnest(…) returned rows.Every call resolves against what we implement: 42883 otherwise (argument types and hint, as ParseFuncOrColumn words it), 42809 for a function of the wrong kind with OVER / FILTER, one qualifier rule for every path. The symbols the translator emits resolve through pg-datahike's own table (datahike.pg.resolve), bound around every handler call; the process's resolver and Datahike's registry are left alone.fuzzer "unknown identifier as function"; manifest test; client suites
0.5ANY/ALL: one implementation (DONE)verified: there were FOUR runtimes, not five — the WHERE predicate (two-valued, over clojure.core operators), two value-position copies for = and <> only, and the equality WHERE branch's own pg-arr/member? path. They disagreed with PostgreSQL and each other: SELECT v > ANY(arr) was 42883, WHERE v > ANY(arr) threw on a NULL element, WHERE v > ALL(ARRAY[1,NULL]) threw, v <> ANY(i.indkey) did not read an int2vector, NULL > ANY(ARRAY[]::int[]) answered NULL where PostgreSQL says f, and 12 = ANY('12') answered NULL where PostgreSQL raises 22P02one Kleene runtime (quantified-result-var) for every operator in both positions, one array reader, one literal expansion — used only where it is sound and over the SQL comparisonsfuzzer surface widened to every operator in both positions incl. NULL/empty/NULL-array operands; pg-any-all-test
0.6classify.clj / server shortcuts (DONE)the value functions are translated (#195), nextval is per row, pg_backend_pid() and txid_current() read the session-state atom (#211). The last three needed the CONNECTION rather than a value read from it -- pg_sleep, pg_notify and the six advisory-lock functions -- so SELECT pg_sleep(0), 2 and CASE WHEN pg_try_advisory_lock(1) …, which is how a migration tool guards a step, were 42883all eight are translated functions now and the shortcuts are deleted: the advisory ones read the session id from the session-state atom (plans marked session-dependent), and the registry moved to datahike.pg.locks so the SQL layer can require itexclusion across sessions, re-entrancy, the two-key namespace, xact release at COMMIT, release on close, 25P01 outside a transaction -- all checked through the translated path
0.7shape.clj probes (DONE)FK names were made up (fk_<hash of SQL>) — deleted, pg_constraint answers. The column-metadata probe hid a LEFT JOIN that multiplied one row per column into twenty; with that fixed the real query answers exactly as PostgreSQL does, so it was deleted too. The primary-key probe regexed the table name out of pgjdbc's SQL and invented an IS_NOT_NULL column; it needed four things, all now done: composite field selection (x).n (#212), a record surviving a derived table (#217), information_schema._pg_expandarray + indnkeyatts + a pg_class row for the implicit index, and pg_index.indkey as an array (#214)shape.clj is deleted — no statement is answered by its shape any more. pgjdbc's getPrimaryKeys goes through the catalog and answers as PostgreSQL doespgjdbc metadata test over the real driver; oracle diff of both probe SQL variants
0.8LIMIT/OFFSET templating (audited; gate added)audited: the simple-query path rewrites numeric literals into $N before parsing (template/parameterize-numbers) so one plan serves a statement family; the extended path never re-templates, and the templater already refuses to parameterise after LIMIT/OFFSET/FETCH/TOP and inside the constructs that read literals at translate time. Running the whole 854-sample SELECT corpus BOTH ways found no divergence — the one disagreement is the known to_char refusal. So the lexical templater is not producing wrong answers todaythe gate is permanent: the fuzzer's select surface now runs every sample over both protocols and compares the pair, so a templating divergence fails the run. Templating as an AST transform stays worth doing for robustness — a lexical rewrite is one exotic literal away from a wrong parse — but it is a refactor, not a bug fix, and is no longer blockingfuzzer templated-vs-untemplated identity
0.11The fuzzer's join surface joins the gate (done)it reported 47 disagreements over 132 samples; the nullable-side and FULL JOIN fixes took that to 10, all of them the aggregate-over-FULL-JOIN refusal, which is registered in expected-divergences.edn:join is in the all surfaces, so the gate now covers INNER/LEFT/RIGHT/FULL over both wire protocolsthe surface itself
0.9DDL slowdown (done: #185)verified: the schema map grows with every DROP/CREATEdatahike #1089 (green), then bump the dependencychurn benchmark stays flat
0.10Cache and catalog-basis stability (DONE for the reported failures; one conservatism left)fixed: a transaction holding an INSERT aborted with 40001 whenever any other session committed (52 of 100 concurrent INSERT-then-UPDATE transactions on disjoint tables; now 0, no lost updates, real overlaps still detected). fixed: an unrelated CREATE TABLE aborted an open transaction at COMMIT — the guard compared the WHOLE catalog for equality; it now examines the difference and admits a catalog that merely GAINED relations (basis/only-new-relations?), while a change to anything the capture held still aborts. verified: the parse cache does not leak a session's state atom — session-value-expr! marks those plans session-dependent!, and a two-session pg_backend_pid() check agreesleft: an ALTER of an UNRELATED table still aborts, where PostgreSQL would not. That needs per-statement dependency scoping; catalog/admission.clj already models it for :literal-insert-v1 and :target-delete-v1 and is the shaped paththe six validation surfaces; pg-catalog-drift-test (both the new-relation admission and the added-column refusal)

Found while doing Phase 0

Each is verified against the oracle and has its own item; none is a regression from the work above.

ItemVerified behaviourFix
Name resolution, level by levelan unqualified outer column inside a subquery that has its own FROM is 42703 (SELECT (SELECT count(*) FROM y WHERE b IS NULL) FROM z); an intermediate subquery level does not shadow an outer onecorrelated-subquery-refs collects only qualified names. One resolver following colNameToVar (parse_relation.c), the innermost level first. Pairs with 0.6.
Common type of all-unknownCASE WHEN … THEN NULL END is not typed text, so an assignment that PostgreSQL rejects with 42804 is acceptedselect_common_type: all-unknown resolves to TEXT for resolution, not for projection
Assignment into enum and domain columnsSET enumcol = 'x'::text is accepted; PostgreSQL raises 42804assignment-cast-exists? answers true for every type it has no category for; enums and domains need their own rule (a domain checks its base type)
Whole-row JSON key orderrow_to_json(t) emits keys alphabetically; PostgreSQL uses CREATE TABLE orderthe field order is lost between the record and the serialiser
Sequences inside expressionsnextval() in a subquery or in RETURNING raises 0A000both are deferred markers that only the SELECT executor resolves; run row projections through it (step 4's work)
A parameter inside a materialised relation (fixed)SELECT * FROM (SELECT id FROM t WHERE id = ?) x and the CTE form answered 0 rows for every binding: the body is run at PARSE, before Bind, so it saw placeholders. Literals were fine, so it needed a parameter and a materialised relation together — Metabase's column introspection has bothfixed: the statement is marked while parsing and re-parsed at Execute with params/*bound-params* in scope, as runtime subqueries already do; the parse-time plan is not cached
A golden records what we did, not what PostgreSQL does (guarded)three goldens held :rows [] — the empty answers a probe or an unreadable type had been giving — and each defended that answer until the underlying bug was fixed by other meansguarded: the goldens test now fails on a probe that answers nothing, with an explicit may-be-empty opt-out. The stronger form, regenerating goldens against the oracle rather than against ourselves, is still worth doing: CI already runs a real PostgreSQL for the fuzzer
ANY has three runtime implementationsthe int2vector fix had to be applied in three places — the value-position branch, the WHERE ?pg-<kind>-pred branch, and the shared any-all-op-fn — and #198 found the same duplication in the element reader. A change that lands in two of the three is a silent divergence between SELECT x = ANY(c) and WHERE x = ANY(c)one runtime for ANY/ALL, taking the element reader and the comparator as arguments
An untyped literal array is NULL, not an errorSELECT 12 = ANY('12') answers NULL; PostgreSQL raises malformed array literal: "12" (22P02). Ours neither errors nor matchesthe array input function should raise rather than yield nothing
The first run after a hot code reload is wrongreproduced on demand: tools/rl.sh (reload into the live dev server), then bb fuzz join 150 20260918 reports 131 disagreements of 132 samples, INNER joins included; the identical command run again reports 27, stably, and the difference is the reload, not the corpus. A dev-loop artifact rather than a server defect -- no client reloads code into a running server -- but it invalidates any measurement taken on the first run after a reloadfind what survives require :reload with state built by the old code (translation cache, catalog basis, the protocol handler's per-database memoisations); until then, run a measurement TWICE after a reload and trust the second
FULL JOIN, three defects (fixed)the FULL→LEFT rewrite mutated the cached AST, so the FIRST execution answered a FULL JOIN and every later one a plain LEFT JOIN; the halves were combined by removing equal projections, losing right-only rows; describeResult did not know the shape, so over the extended protocol the client met DataRows with no RowDescription and every later statement on that connection failedfixed: a private AST for a FULL JOIN statement, an unmatched-only second half, a Describe branch. An aggregate, DISTINCT, LIMIT or OFFSET over a FULL JOIN is now refused with 0A000 rather than answered per half -- the remaining gap, and the only join divergence left in the fuzzer
A ref-based LEFT JOIN answers nothing (fixed)over a :db.type/ref column whose OWNER is the table being joined FROM -- person p LEFT JOIN company c ON p.company = c.db_id -- the join answered no rows at all (INNER over the same columns was right). The lowering was written for the other direction, the ref's owner on the JOINED side, and took the joined alias for the owner either way: it emitted [?c_eid :person/company ?c_eid], entity and value one variablefixed: the ON clause records which side owns the ref, and the owner-on-left direction gets its own lowering -- one pattern [?p_eid :person/company ?c_eid] says the join, the right side is read through the joined entity var, and the left row's own ref column stays bound outside so a row that fails a further condition is null-extended rather than dropped
FULL JOIN ignores its ORDER BY… FULL JOIN … ORDER BY 1,2,3 answers the right rows in the wrong order (1\|2 before 1\|1); the union of the two rewritten LEFT joins is concatenated, not re-sortedsort the union, not its branches
An outer join with a multi-condition ON answers wrong rowsover la(x,y) = (1,1),(1,2),(2,1) and lb(x,y,v) = (1,1,p),(1,9,q),(2,2,r): la LEFT JOIN lb ON (b.x=a.x AND b.y=a.y) answers (1,1,p) (1,2,r) (2,1,p) where PostgreSQL answers (1,1,p) (1,2,NULL) (2,1,NULL) — rows that satisfy neither condition pair. ON (b.x=a.x AND b.y=1) DROPS the unmatched left row, and ON (b.x=a.x AND b.y>a.y) answers nothing at all. RIGHT and FULL are wrong the same way; one condition, and INNER with the same ON, are correctevery ON condition belongs inside the matched branch of the outer-join lowering, with the unmatched branch null-extending and nothing constraining the left relation
An outer join whose ON has no equality was refused (fixed)LEFT JOIN … ON true, ON (b.y > a.y), ON (b.x <> a.x), ON (b.v = 'p'), ON false, ON (b.v IS NULL) were all 0A000 ("non-equality outer join conditions are not supported"); PostgreSQL answers each by considering every right row for every left row and filtering, and most of the ON-clause space has no equality in itlower it as a NESTED LOOP: the equi-join's shape with the row-existence marker in place of the key pattern, so the branch enumerates the right relation instead of seeking into it. The unmatched branch is then a negation whose body mentions the left row only through the join variable — a cross product by construction — which Datahike's planner answered as "exclude every row" (replikativ/datahike#1092, fixed there): a negation whose body has no solution excludes nothing. Needs the Datahike release carrying that fix.
A record does not survive a derived table (fixed)SELECT r FROM (SELECT row(7,8) AS r) s rendered datahike.pg.records.PgRecord@… and (s.r).f1 raised 42703: a materialised relation stores Datahike scalars, and the record was Java-str'd into onestore it as canonical PG text plus :pg/record-fields (the record twin of :pg/array-elem, carrying the field names and OIDs the text drops), and rebuild it at (expr).field. One of the three things 0.7's last probe needs
A derived table's column-alias list is ignored (fixed)SELECT a FROM (SELECT 1 AS x) AS s(a) is 42703 column "a" does not exist; PostgreSQL answers 1. FROM (VALUES …) AS v(a,b) works — only that path passes the alias columns to the materialiserpass alias-cols through the inner-select branch of materialize-derived-select! as the VALUES branch does, and rename the materialised columns
The type-name cast functions are missingtext(123), int4('5'), bool('t'), float8('1.5') and name('x') are 42883; PostgreSQL has a pg_proc entry for each (they are pg_cast's cast functions) and answers x::T. name('x') used to answer by accident, through clojure.core/nametranslate T(x) as x::T for the pairs pg_cast names, from the generated cast table
bool_and(p) FILTER (WHERE true) is XX000class java.lang.Long cannot be cast to class clojure.lang.Symbol; FILTER (WHERE a > 1) and sum/count/max … FILTER (WHERE true) are all fine, and the oracle answers tthe constant-true filter argument reaches the aggregate where a variable is expected
A function name is not folded like an identifier (fixed)SELECT "upper"('a') was 42883 — the quotes stayed part of the name — and psycopg2 quotes every identifier it composes, so SELECT "pg_notify"(…) only worked while a whole-statement shortcut caught itresolution-name folds an unquoted name and strips the quotes from a quoted one, keeping its case: "UPPER"('a') is 42883, as in PostgreSQL
CHECK is resolved on first use, not at CREATE TABLECREATE TABLE t (a int CHECK (nosuchfn(a) > 0)) is accepted; PostgreSQL raises 42883 for the DDL. The first INSERT raises it here.translate the CHECK expression once when the constraint is created (0.2's evaluator, run at DDL with an empty row scope)
A FROM-less SELECT drops its ORDER BY / GROUP BY / HAVINGSELECT 1 ORDER BY nosuchfn(1), … GROUP BY nosuchfn(1) and … HAVING nosuchfn(1) = 2 all answer one row; with a FROM they are 42883. The clause is never translated.translate the clauses of a FROM-less SELECT too
An unknown schema qualifier is 42883, not 3F000nosuch.f(1) and SELECT * FROM nosuch.unnest(…) report function nosuch.f(...) does not exist; PostgreSQL reports schema "nosuch" does not existcheck the schema against the catalog before resolving the function
42883 argument types are missing on two pathsa FROM-clause function reports generate_series() and a WITHIN GROUP call f(...) WITHIN GROUP, both without argument typesboth know their argument expressions; use call-arg-oids as the scalar path does
ORDER BY over a json value is XX000SELECT jsonb_build_object('a', a) FROM t ORDER BY 1 fails with class clojure.lang.PersistentArrayMap cannot be cast to class java.lang.Comparable; PostgreSQL orders jsonbPhase 1.4's per-type comparison
Constraint layerCHECK is evaluated 3× per inserted row; pg_dump omits CHECKs; RENAME / DROP COLUMN leave constraints stale; an INHERITS child writes its own namespace for an inherited column; SET on a GENERATED ALWAYS identity is not 428C9; four copies of the FK readerits own track after Phase 0

Why these were not caught

Each item above was found by hand, late. The pattern is worth naming, because the fix is cheaper than the next bug:

  1. A probe is a coverage hole. shape.clj intercepted pgjdbc's metadata SQL and answered it from a hand-written handler, so the LEFT JOIN inside that SQL never ran. The bug it hid — a two-condition ON answering wrong rows — was found the moment the probe was switched off. Every remaining probe, shortcut and classify special case should be read as "this query shape has never been executed here".
  2. The differential fuzzer's grammar decides what is verified. It compares against a real PostgreSQL for 58 SELECT classes and found none of these, because its join classes emit a single equi-join condition and INNER/LEFT only. An ON clause of 1-3 conjuncts mixing equality, inequality and constants, crossed with {INNER, LEFT, RIGHT, FULL} and NULL-bearing keys, would have caught all three outer-join items on the first run.
  3. One protocol is tested, two are served. psql -c uses the simple protocol; the FULL JOIN failure appears only over the extended one. The fuzzer has a prepared mode, but it covers scalar predicates, not joins — running the same corpus through both protocols is nearly free.
  4. Hand-written expectations agree with the implementation. Our tests assert what we believed; the oracle-differential ones assert what PostgreSQL does. Where a test is hand-written, a wrong answer stays wrong and green.

Actions, in order of value: extend the fuzzer's join grammar and run the corpus through both protocols; add a differential case for each query a probe or shortcut still intercepts, before deleting it; prefer oracle comparison to hand-written expectations for anything with join, NULL or type semantics.

Moved out of Phase 0:

  • Recursive CTE UNION ALL deduplicates. Execution is a Datalog rule, and set semantics are built in. Keeping duplicates needs a working-table loop, a rewrite of its own, so it becomes a separate item after Phase 1.
  • Binary results carrying text bytes. Binary encoding re-parses the stringified value. The real fix is the per-type binary send in Phase 1.2; until then, stop advertising binary for types without a correct send.

Phase 1 — one value registry (finish #179)

Target (value-layer audit §5): per-OID {:in :out :text :recv :send :compare :storage :typmod} and one resolve-type-name. Shippable steps:

  1. Output complete. Every result path renders by OID. RETURNING and the shortcuts currently call value->string without one, and LocalTime/LocalDateTime go through str. Make ->pg-text total (bool via OID, bytea, bit, vector, NaN, json-null) and reduce value->string to it. Fix record_out quote doubling. to_jsonb dispatches by OID.

  2. Registry for output and binary send, generated into the existing maps. The dh-type->oid and dh-type->pg-name tables already disagree (bytes maps to text in one and bytea in the other) so old callers keep working; the Java codec gets typlen/array maps through a register call.

  3. Input, one family per PR:

    • bool → integers → float/numeric/money → uuid;
    • date/timestamp/timestamptz: one datetime decoder (datetime.c ParseDateTime/DecodeDateTime) replacing parse-timestamp-string and its 8 siblings, including COPY. This is the agreed first item: 'garbage'::timestamptz passes through, and a timestamp offset is converted instead of ignored;
    • time/timetz → bytea → arrays (typed elements).

    Each PR routes cast, INSERT/UPDATE, unknown literal, COPY, text parameter and pg_input_is_valid through the family's :in.

  4. Comparison, equality and hashing from the registry. Merge order-cmp, sql-order-cmp, null-safe-order-cmp and insert-select-order-cmp. GROUP BY, DISTINCT, window partitions and IN-sets use the type's equality and hash, not Clojure =/group-by/contains?. For example, NaN = NaN in PostgreSQL, and '1 mon' = '30 days' for intervals. Fixes signed uuid ordering and array elements compared as strings.

  5. Session TimeZone. Needs the single settings store (moved here from Phase 4) and a parse-cache key that includes TimeZone and DateStyle; today translation-cache-key includes only search_path. Storage convention:

    • timestamptz is stored as an absolute instant;
    • timestamp and date are stored as UTC wall-clock;
    • TimeZone affects only timestamptz input and output, and casts between timestamp and timestamptz.

    Also fix current_setting: it is case-sensitive and ignores SET. set_config doesn't store its value. is_superuser contradicts pg_roles. The role name is hard-coded three times.

Phase 2 — interval as a value

A PgInterval carrier (months, days, micros) with input, output, compare and arithmetic. Comparison, equality and hashing follow interval_cmp_value (timestamp.c): a month is 30 days and a day is 24 hours. A record's structural =/hash would split GROUP BY and joins. Unlocks about 90 sweep entries: timestamp ± interval, age, justify_*, make_interval, date_bin, OVERLAPS. Fixes interval comparison, which is wrong today ('10 days' < '9 days' is true).

Phase 3 — bytea end to end

Four fixes: the literal-cast fold, the column OID (17, not text), a strict bytea-in (hex and escape formats), and byte[] arms in the text functions and ->pg-text. Then encode/decode, sha2, get/set_byte, convert_*. Every driver reading bytea is affected, so this needs the client suites plus a dump round-trip.

Phase 4 — signature registry and resolvers

  • pg_proc.dat registry for all functions: argument types, return type, strictness, volatility, kind. It drives resolution (42883/42725), return-type inference and strict NULL handling, and replaces sql-fn-arities, the sql-function-specs flags, oid_infer's function tables and classify's shortcuts. Translation audit §4.3 gives the extraction order.
  • One name/OID resolver (relation, type, namespace, role, function) with PostgreSQL's errors. It fixes regclass/regtype/regnamespace, the ~78 has_*_privilege/pg_has_role entries (a single superuser role: true for existing objects, errors for unknown ones), to_reg* and pg_*_is_visible.
  • Parameter types from the translator instead of the per-statement *-param-oids walkers.

Phase 5 — text semantics

  • One pattern-matching layer: PostgreSQL ARE → java.util.regex translation, flags, LIKE ESCAPE (a trailing escape raises 22025), DOTALL, UNICODE_CASE, SIMILAR TO, 2201B. Replaces 4 copies, including two LIKE compilers.
    • A syntax translation can't reproduce ARE's longest-match alternation: 'a|ab' matches ab in PostgreSQL and a in Java. Either port the engine or document the divergence with a test.
  • Set-returning functions implemented once: the correlated producer duplicates materialize-table-function. Its generate_series truncates to long, and a zero step returns [] where PostgreSQL raises 22023. regexp_split_to_table drops trailing empty strings.
  • Code-point string functions (emoji and non-BMP characters).
  • format() as a port of text_format; bpchar padding semantics.

Phase 6 — feature gaps

  • the formatting.c template engine (to_char/to_date/to_timestamp over dates and timestamps);
  • the now() family and transaction-stable time;
  • float math precision (round half-even, stable asinh/atanh/acosh, glibc cbrt);
  • missing aggregates (bool_or, bit_*, …) — the signature table already names them;
  • operator-implementation functions as a generated alias table;
  • remaining small functions per report.

Process per phase

  • Oracle-verified tests, taken from the report's "tests to add" sections.
  • A differential-fuzzer class for the area; the CI seed must stay clean, and extra seeds are swept before merging.
  • bb fncov ratchet: the wrong-answer count must not rise; add it to CI as a report with a threshold.
  • A review agent after each phase, checking consistency and deduplication against these principles. It gets the code, not our conclusions.
  • Full surfaces before a PR:
    • unit tests, sqllogictest, the full curated pgjdbc suite (test/integration/pgjdbc/run.sh, not bb pgjdbc);
    • SQLAlchemy, asyncpg (manifest gate), node-postgres, pg_dump;
    • bb fuzz all.
  • Strict regression slices take expectations from upstream expected/*.out, and the campaign inventory should check that.

Remaining passthroughs to remove (with their phase)

LocationBehaviourPhase
jsonb/parse-jsonbreturns the input string0.3
parse-timestamp-stringreturns the input string1.3
json_to_record cells (stmt ~1336)keeps the raw value on failure1.3
copy.clj ref valueskeep the raw value on failure1.3
bytea (stmt ~6983)falls back to UTF-8 bytes3
arrays.clj elementsuuid, numeric and date elements stay strings0.1 / 1.3
interval castreturns its input2
cast-scalarreturns the value unchanged for targets it doesn't know4 (resolver: 42704)

Tracked separately

  • node-postgres flake, on main as well: "catalog changed while statement was being executed" under concurrent clients (2 of 6 runs), a race in catalog admission.
  • Baseline DDL cost of 100–250 ms per statement even on a fresh database.
  • Performance: drop-table-tx-data scans the whole schema and runs one query per attribute.

Can you improve this documentation?Edit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close