Prepared-statement parameter substitution and PG OID inference.
Three concerns live together here because they all deal with the
? / $N placeholder lifecycle:
ParamRef record + param-ref?. Emitted by translators at Parse
time inside tx-data / query structures, replaced by real values
at Execute time via substitute-params.
*bound-params* dynamic var: when bound to a 1-indexed vector
of resolved values, translator branches (e.g. the JdbcParameter
expression) resolve placeholders in-line instead of emitting
ParamRef. This lets the same translator body serve both
prepared-Parse (returns structured AST with ParamRefs) and
re-translation-during-Execute (returns fully-bound form).
OID inference (infer-param-oid-for-column,
insert-param-oids, update-param-oids, where-param-oids) —
walks a JSqlParser AST, maps each placeholder index to the PG
type OID of the column it's bound against. Consumed by
describeParams to populate the ParameterDescription message so
pgjdbc / psycopg2 / etc. size their binary binds correctly.
*parse-db* is bound by parse-sql (higher-level) to the current
db snapshot so OID inference can consult :pg/type metadata that
(:schema db) doesn't surface.
Prepared-statement parameter substitution and PG OID inference. Three concerns live together here because they all deal with the `?` / `$N` placeholder lifecycle: 1. ParamRef record + `param-ref?`. Emitted by translators at Parse time inside tx-data / query structures, replaced by real values at Execute time via `substitute-params`. 2. `*bound-params*` dynamic var: when bound to a 1-indexed vector of resolved values, translator branches (e.g. the JdbcParameter expression) resolve placeholders in-line instead of emitting ParamRef. This lets the same translator body serve both prepared-Parse (returns structured AST with ParamRefs) and re-translation-during-Execute (returns fully-bound form). 3. OID inference (`infer-param-oid-for-column`, `insert-param-oids`, `update-param-oids`, `where-param-oids`) — walks a JSqlParser AST, maps each placeholder index to the PG type OID of the column it's bound against. Consumed by describeParams to populate the ParameterDescription message so pgjdbc / psycopg2 / etc. size their binary binds correctly. `*parse-db*` is bound by parse-sql (higher-level) to the current db snapshot so OID inference can consult :pg/type metadata that (:schema db) doesn't surface.
Dynamically bound at Execute time to a 1-indexed vector (or nil if
no params). When set, translate-expr's JdbcParameter branch resolves
placeholders to concrete values in-line; otherwise (Parse time) it
emits ?pN in-param vars and records the index in ctx.
This split lets the same translator body serve both prepared-Parse and re-translation-during-Execute (UPDATE/DELETE keep where-expr as a JSqlParser AST and re-translate on each Execute).
Dynamically bound at Execute time to a 1-indexed vector (or nil if no params). When set, translate-expr's JdbcParameter branch resolves placeholders to concrete values in-line; otherwise (Parse time) it emits `?pN` in-param vars and records the index in ctx. This split lets the same translator body serve both prepared-Parse and re-translation-during-Execute (UPDATE/DELETE keep where-expr as a JSqlParser AST and re-translate on each Execute).
When bound (by build-update-tx handling UPDATE ... FROM (VALUES ...)),
a map {alias-name → {col-name → literal}} used by the Column branches
of translate-expr and eval-update-expr to substitute row-level values
for references like __tmp.col to the VALUES alias.
When bound (by build-update-tx handling UPDATE ... FROM (VALUES ...)),
a map {alias-name → {col-name → literal}} used by the Column branches
of translate-expr and eval-update-expr to substitute row-level values
for references like `__tmp.col` to the VALUES alias.Bound by parse-sql to the live db snapshot so downstream helpers (e.g. pg-type-of-attr) can consult Datahike for attribute metadata that :schema doesn't surface (:pg/type and friends). Not meant to flow beyond the parse phase — clear it before dispatching to the execute path.
Bound by parse-sql to the live db snapshot so downstream helpers (e.g. pg-type-of-attr) can consult Datahike for attribute metadata that :schema doesn't surface (:pg/type and friends). Not meant to flow beyond the parse phase — clear it before dispatching to the execute path.
Bound by parse-sql to itself so top-level translate-* entries in
datahike.pg.sql.stmt can seed :parse-sql into make-ctx without a
cyclic require on sql.clj. Downstream expression translators call
it to re-parse inner SQL strings for IN / EXISTS subqueries.
Bound by parse-sql to itself so top-level translate-* entries in datahike.pg.sql.stmt can seed `:parse-sql` into make-ctx without a cyclic require on sql.clj. Downstream expression translators call it to re-parse inner SQL strings for IN / EXISTS subqueries.
(ast-param-indices node)Recursively walk a JSqlParser AST, returning a sorted set of
1-based parameter indices (? / $N placeholders).
Uses reflection to traverse any node's zero-arg getters. This is called once per Parse to determine how many parameters a prepared statement expects — the cost is bounded by AST size, which is tiny compared to query execution.
Recursively walk a JSqlParser AST, returning a sorted set of 1-based parameter indices (`?` / `$N` placeholders). Uses reflection to traverse any node's zero-arg getters. This is called once per Parse to determine how many parameters a prepared statement expects — the cost is bounded by AST size, which is tiny compared to query execution.
(call-marker? v)True if v is a deferred function-call marker emitted by translate-*
(currently :nextval and :now). These must survive the result-
cache intact and be resolved per execute.
True if v is a deferred function-call marker emitted by translate-* (currently `:nextval` and `:now`). These must survive the result- cache intact and be resolved per execute.
(collect-table-aliases from-item joins)Given a FROM item and a sequence of JOINs, build a map
{alias-name → real-table-name}. Tables without an alias still get
an entry mapping the table name to itself (so lookup is uniform).
Given a FROM item and a sequence of JOINs, build a map
`{alias-name → real-table-name}`. Tables without an alias still get
an entry mapping the table name to itself (so lookup is uniform).(fold-identifier s)Apply PostgreSQL's case rule to an already-unquoted identifier: fold ASCII A-Z to lower case, and truncate to 63 bytes.
ASCII-only, and deliberately not .toLowerCase of any flavour.
PostgreSQL's downcase_truncate_identifier (scansup.c) only maps
A-Z under standard encodings — verified against a UTF8 server, where
CREATE TABLE ÄTest yields the relation Ätest: the T folded, the
Ä did not. A Unicode-aware fold would produce ätest and miss it.
Being ASCII-only also sidesteps the locale trap that would otherwise
need guarding against: clojure.string/lower-case uses the default
locale, under which Turkish folds "ID" to "ıd" and every
identifier on the server silently mis-resolves.
Truncation is by BYTES, without splitting a character — PG uses pg_mbcliplen for the same reason.
Apply PostgreSQL's case rule to an already-unquoted identifier: fold ASCII A-Z to lower case, and truncate to 63 bytes. ASCII-only, and deliberately not `.toLowerCase` of any flavour. PostgreSQL's `downcase_truncate_identifier` (scansup.c) only maps A-Z under standard encodings — verified against a UTF8 server, where `CREATE TABLE ÄTest` yields the relation `Ätest`: the `T` folded, the `Ä` did not. A Unicode-aware fold would produce `ätest` and miss it. Being ASCII-only also sidesteps the locale trap that would otherwise need guarding against: `clojure.string/lower-case` uses the default locale, under which Turkish folds "ID" to "ıd" and every identifier on the server silently mis-resolves. Truncation is by BYTES, without splitting a character — PG uses pg_mbcliplen for the same reason.
(has-param-marker? sql)Fast scan: does SQL contain a ? or $N placeholder OUTSIDE a
quoted string, dollar-quoted body, or comment? When false, the
parser doesn't need to walk the AST for parameter indices — a
real win for pg_dump-style INSERTs (literal-only) where the
reflection-based AST walk dominated parse time.
pgjdbc rewrites ? to numbered $N before sending Parse, so the
on-wire SQL never has ? from a JDBC client — must detect both
forms.
Fast scan: does SQL contain a `?` or `$N` placeholder OUTSIDE a quoted string, dollar-quoted body, or comment? When false, the parser doesn't need to walk the AST for parameter indices — a real win for pg_dump-style INSERTs (literal-only) where the reflection-based AST walk dominated parse time. pgjdbc rewrites `?` to numbered `$N` before sending Parse, so the on-wire SQL never has `?` from a JDBC client — must detect both forms.
(infer-param-oid-for-column schema table-ns col-name)(infer-param-oid-for-column schema table-ns col-name db)Given a schema and a (table-namespace, column-name), return the PG OID that matches the attribute's :db/valueType, or nil if we don't know. Used by describeParams to populate ParameterDescription so drivers can size buffers correctly for typed INSERT/UPDATE/WHERE placeholders.
:pg/type on the attr — recorded by translate-create-table for types
that don't have a 1:1 Datahike mapping (date/time/timestamp all
collapse to :db.type/instant; jsonb vs json both use :db.type/string)
— takes precedence so we round-trip the original SQL type. Without
this, pgjdbc Describes a DATE column's param as timestamp (1114)
and rejects subsequent binds as "Can't change resolved type for
param: 1 from 1082 to 1114".
db is optional — pass nil and the :pg/type refinement is skipped.
Accepts schema either as the map {ident → props} returned by
pgs/schema-of, or as a seq of {:db/ident ... :db/valueType ...}
entries (the :db/add transaction form).
Given a schema and a (table-namespace, column-name), return the PG
OID that matches the attribute's :db/valueType, or nil if we don't
know. Used by describeParams to populate ParameterDescription so
drivers can size buffers correctly for typed INSERT/UPDATE/WHERE
placeholders.
:pg/type on the attr — recorded by translate-create-table for types
that don't have a 1:1 Datahike mapping (date/time/timestamp all
collapse to :db.type/instant; jsonb vs json both use :db.type/string)
— takes precedence so we round-trip the original SQL type. Without
this, pgjdbc Describes a DATE column's param as `timestamp` (1114)
and rejects subsequent binds as "Can't change resolved type for
param: 1 from 1082 to 1114".
`db` is optional — pass nil and the :pg/type refinement is skipped.
Accepts schema either as the map {ident → props} returned by
`pgs/schema-of`, or as a seq of `{:db/ident ... :db/valueType ...}`
entries (the `:db/add` transaction form).(insert-param-oids insert schema)(insert-param-oids insert schema db)Walk an INSERT AST: for each VALUES (..., ?, ...) row, positional
column i → attribute type → PG OID. Returns a map {param-index → oid}.
Only covers the flat single-row / multi-row VALUES case — which is what JDBC setObject/setString produces for the common ORM path.
When the INSERT omits the explicit column list (INSERT INTO t VALUES (?, ?, ?)), falls back to the table's declared column order
from pgs/column-info (which honours both schema entity-ID order and
the :datahike.pg/column-order hint). This is what pgjdbc's
executeBatch needs: setLong(1, …) wants param 1's OID at Describe
time, and without inferred OIDs pgjdbc's resolved-type tracker
raises Can't change resolved type for param: N from <oid> to 0.
Walk an INSERT AST: for each `VALUES (..., ?, ...)` row, positional
column i → attribute type → PG OID. Returns a map {param-index → oid}.
Only covers the flat single-row / multi-row VALUES case — which is
what JDBC setObject/setString produces for the common ORM path.
When the INSERT omits the explicit column list (`INSERT INTO t
VALUES (?, ?, ?)`), falls back to the table's declared column order
from `pgs/column-info` (which honours both schema entity-ID order and
the `:datahike.pg/column-order` hint). This is what pgjdbc's
`executeBatch` needs: setLong(1, …) wants param 1's OID at Describe
time, and without inferred OIDs pgjdbc's resolved-type tracker
raises `Can't change resolved type for param: N from <oid> to 0`.PostgreSQL's NAMEDATALEN - 1 (pg_config_manual.h). Identifiers longer than this are truncated; PG also emits a NOTICE, which we don't.
PostgreSQL's NAMEDATALEN - 1 (pg_config_manual.h). Identifiers longer than this are truncated; PG also emits a NOTICE, which we don't.
(nextval-marker? v)Back-compat alias: true only for the nextval flavour of call-marker.
Back-compat alias: true only for the nextval flavour of call-marker.
(param-ref? x)True when x is a parameter placeholder emitted by translate-* during
prepared-statement parsing. Appears in tx-data (INSERT/UPDATE VALUES)
and nested inside parsed structures that the handler walks at Bind/
Execute time to substitute real values.
True when `x` is a parameter placeholder emitted by translate-* during prepared-statement parsing. Appears in tx-data (INSERT/UPDATE VALUES) and nested inside parsed structures that the handler walks at Bind/ Execute time to substitute real values.
(resolve-nextvals! x nextval-fn)Walk x replacing every {:fn :nextval :seq-name S} marker with the
long produced by an actual nextval('S') against the live conn.
Each call commits independently via CAS-retry — same path
handle-nextval uses for SELECT nextval(...). PG semantics:
non-transactional advances; concurrent callers get distinct values.
nextval-fn is (fn [seq-name] long-or-throw). Decoupling the
resolver from the conn lets server.clj wire handle-nextval in
without params.clj taking a server.clj dependency.
Sibling shape to substitute-params: leaves functions, records,
and other opaque values alone, recurses into map values / vectors /
seqs.
Walk `x` replacing every `{:fn :nextval :seq-name S}` marker with the
long produced by an actual `nextval('S')` against the live conn.
Each call commits independently via CAS-retry — same path
`handle-nextval` uses for `SELECT nextval(...)`. PG semantics:
non-transactional advances; concurrent callers get distinct values.
`nextval-fn` is `(fn [seq-name] long-or-throw)`. Decoupling the
resolver from the conn lets server.clj wire `handle-nextval` in
without `params.clj` taking a server.clj dependency.
Sibling shape to `substitute-params`: leaves functions, records,
and other opaque values alone, recurses into map values / vectors /
seqs.(substitute-params x bound)Walk x replacing every ParamRef with the corresponding bound value
from bound (1-indexed: (->ParamRef 1) → (bound 1) ... so bound
is either a vector (we nth by idx-1) or a function idx→value).
Leaves everything else untouched.
The wire layer calls this at Execute time to resolve placeholders inside INSERT tx-data.
Maps with nil values after substitution have those keys dissoc'd.
An INSERT like INSERT INTO t (a, b) VALUES (?, ?) with
setString(1, null) ends up as {:t/a nil :t/b "x"} here —
d/transact rejects [:db/add eid :t/a nil] as :transact/syntax,
but the correct PG behaviour for a nullable column is to simply
not assert the attribute. The translate-time row-builder already
drops nil literals (NullValue), but those land as ParamRef sentinels
at parse time and only resolve to nil here.
Identity preservation: deferred call-markers ({:fn :nextval ...},
{:fn :now}) pass through unchanged — same Clojure object in,
same object out. Otherwise reduce-kv would mint new marker maps
and resolve-nextvals! would call the underlying function multiple
times when the same logical use appears in multiple parts of
tx-data (e.g. a :db.fn/call arg AND an outer entity-map via
assoc).
Walk `x` replacing every ParamRef with the corresponding bound value
from `bound` (1-indexed: `(->ParamRef 1)` → `(bound 1)` ... so `bound`
is either a vector (we nth by idx-1) or a function idx→value).
Leaves everything else untouched.
The wire layer calls this at Execute time to resolve placeholders
inside INSERT tx-data.
Maps with nil values after substitution have those keys dissoc'd.
An INSERT like `INSERT INTO t (a, b) VALUES (?, ?)` with
`setString(1, null)` ends up as `{:t/a nil :t/b "x"}` here —
`d/transact` rejects `[:db/add eid :t/a nil]` as `:transact/syntax`,
but the correct PG behaviour for a nullable column is to simply
not assert the attribute. The translate-time row-builder already
drops nil literals (NullValue), but those land as ParamRef sentinels
at parse time and only resolve to nil here.
Identity preservation: deferred call-markers (`{:fn :nextval ...}`,
`{:fn :now}`) pass through unchanged — same Clojure object in,
same object out. Otherwise reduce-kv would mint new marker maps
and resolve-nextvals! would call the underlying function multiple
times when the same logical use appears in multiple parts of
tx-data (e.g. a `:db.fn/call` arg AND an outer entity-map via
`assoc`).(unquote-ident s)Normalise a SQL identifier to the form used as a Datahike name.
PostgreSQL folds UNQUOTED identifiers to lower case at parse time and
leaves quoted ones alone, so CREATE TABLE MixedCase names the table
mixedcase while "MixedCase" names it MixedCase. We used to
store whatever was typed, which meant SELECT ... FROM mixedcase
raised 42P01 for a table created as MixedCase, and — worse —
pg_class.relname reported MixedCase, so a client that folds the
name (as PostgreSQL does) and reflects it found nothing.
The fold has to happen HERE rather than in the callers: the quoted/unquoted distinction lives only in the raw text JSqlParser hands us, and is gone the moment the quotes are stripped.
A quoted identifier also un-escapes doubled quotes: "a""b" is
the single name a"b.
Normalise a SQL identifier to the form used as a Datahike name. PostgreSQL folds UNQUOTED identifiers to lower case at parse time and leaves quoted ones alone, so `CREATE TABLE MixedCase` names the table `mixedcase` while `"MixedCase"` names it `MixedCase`. We used to store whatever was typed, which meant `SELECT ... FROM mixedcase` raised 42P01 for a table created as `MixedCase`, and — worse — `pg_class.relname` reported `MixedCase`, so a client that folds the name (as PostgreSQL does) and reflects it found nothing. The fold has to happen HERE rather than in the callers: the quoted/unquoted distinction lives only in the raw text JSqlParser hands us, and is gone the moment the quotes are stripped. A quoted identifier also un-escapes doubled quotes: `"a""b"` is the single name `a"b`.
(update-param-oids update schema)Walk an UPDATE AST: for each SET col = ?, map param index to the column attribute's PG OID.
Walk an UPDATE AST: for each SET col = ?, map param index to the column attribute's PG OID.
(where-param-oids expr schema default-table-ns)(where-param-oids expr schema default-table-ns aliases)Walk an expression tree (a WHERE clause) and for each comparison
col = ? / ? = col / col IN (?,?) / col BETWEEN ? AND ?,
map the ? param index to the column's PG OID. Best-effort.
default-table-ns is used when a column has no explicit table
qualifier. aliases is an optional {alias-name → real-table-name}
map (derived by the caller from FROM/JOIN clauses) so JOIN t a ON a.col = ? resolves a to t's real schema.
Walk an expression tree (a WHERE clause) and for each comparison
`col = ?` / `? = col` / `col IN (?,?)` / `col BETWEEN ? AND ?`,
map the `?` param index to the column's PG OID. Best-effort.
`default-table-ns` is used when a column has no explicit table
qualifier. `aliases` is an optional `{alias-name → real-table-name}`
map (derived by the caller from FROM/JOIN clauses) so `JOIN t a ON
a.col = ?` resolves `a` to `t`'s real schema.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |