datahike.pg.sql.stmt — org.replikativ/pg-datahike 0.1.45

Statement-level translation: SELECT / INSERT / UPDATE / DELETE / CTE.

The top half of this namespace (translate-select + its join / HAVING / materialization machinery) maps a PlainSelect AST to a Datalog query-map plus row-formatting metadata the handler uses at execute time.

The middle half handles INSERT / UPDATE / DELETE: extract-value lifts JSqlParser literal/expression nodes into Clojure values, coerce-insert-value adapts them to the target column's :db/valueType, and translate-insert / translate-update / translate-delete produce tx-data + (for UPDATE/DELETE) an eids-walk query. INSERT RETURNING and UPDATE RETURNING land in extract-returning; CHECK / UPDATE expressions evaluated per-row at handler time go through eval-check-predicate / eval-update-expr.

The bottom half implements CTEs: translate-cte-branch materializes one WITH-clause body against an enriched db so the outer SELECT / DML can reference it as a virtual table; translate-recursive-cte handles the WITH RECURSIVE form by iterating until the CTE converges.

The three blocks live in one namespace because they are mutually recursive:

translate-select → translate-recursive-cte (WITH in SELECT) translate-recursive-cte → translate-select (CTE body) translate-insert → translate-select (INSERT ... SELECT) translate-insert → translate-recursive-cte (INSERT ... WITH) translate-select → extract-value (scalar subqueries)

Dependencies on already-extracted namespaces are all one-way:

stmt → expr (translate-expr, translate-predicate, …) stmt → ctx (make-ctx, col-var!, resolve-column, …) stmt → fns (aggregate lookups) stmt → params (ParamRef, from-bindings, parse-db) stmt → jsonb, schema, types (type coercion + jsonb ops)

*cte-relations* clj

Lowercased names of CTEs (WITH items) in scope for the statement being translated. Bound by parse-sql* so the undefined-table 42P01 check exempts CTE references that aren't materialised into the schema — most notably data-modifying CTE bodies (WITH x AS (INSERT … RETURNING …)), which are skipped during WITH-fold.

*eval-update-db* clj

Bound by build-update-tx-for-bindings to the live db when evaluating per-row UPDATE SET expressions. Read by the Function (concat, etc.) and ParenthesedSelect (scalar subquery) branches of eval-update-expr — which need a db handle to dispatch to translate-select for inner subqueries. nil when an UPDATE has no subquery / function-call assignments, which is the common case.

apply-sql-cast clj

(apply-sql-cast inner ce)

Apply a SQL CAST to a value. Returns the value cast to the target type. Handles nil safely (returns nil). Idempotent: if inner is already the target type, returns it unchanged (avoids lossy round-trips through str, e.g. Date → (str d) → unparseable string).

coerce-insert-value clj

(coerce-insert-value val attr schema)

Coerce a value to match the schema type for an attribute.

:db.type/ref columns: SQL FK semantics says INSERT/UPDATE … SET col = N writes the target's PK value (matching what the read-side ref-deref returns). Datahike's transact requires either an entity-id or a lookup-ref. We convert the user-supplied PK value to a lookup-ref [target-pk-attr val] using the same convention derive-ref-targets uses on the read side, keeping read and write FK semantics symmetric. Falls through to the raw value when no target is resolvable (hint-only refs without a threaded db, or genuinely-unmapped refs).

correlated-select-item-spec clj

(correlated-select-item-spec e0 outer-aliases)

Detect a correlated scalar subquery in a SELECT-list item expression e, returning a deferral spec (without :out-pos/:alias/:oid, which the caller adds) or nil. Two shapes:

• :scalar — the item IS a scalar subquery (SELECT … <outer ref> …).
• :case — the item is a CASE whose THEN/ELSE contains a correlated subquery (asyncpg's CASE WHEN typtype='c' THEN (SELECT array_agg(…) WHERE c.reltype = t.oid) END). The single-rule CASE compiler would pre-evaluate the subquery once at parse (→ NULL); instead we defer and let exec-select evaluate the whole CASE per outer row.

:corr-refs is the set of [outer-alias col] references threaded into :find as hidden columns so exec-select can bind from-bindings per row.

correlated-splice clj

(correlated-splice visible out-pos->val n-output)

Assemble n-output columns from visible (non-correlation columns in order) and out-pos->val (subquery output-position → value).

correlated-subquery-refs clj

(correlated-subquery-refs inner outer-aliases)

Given a scalar-subquery inner (a JSqlParser Select) and the set of outer-aliases (lowercased outer FROM aliases/table names), return the set of [outer-alias col] correlation references the inner makes, or nil when uncorrelated.

Detection is lexical — it finds alias.col occurrences of an OUTER alias in the inner SQL (negative-lookbehind so xt. doesn't match alias t). Robust enough for catalog / introspection shapes; AST-precise detection (which would also respect inner shadowing) is a later refinement. This is the first slice of the correlated-subquery / LATERAL executor — see doc/design-alignment.md.

enrich-schema-with-pg-array-meta clj

(enrich-schema-with-pg-array-meta schema db)

Datahike's :schema map only carries :db/* keys; pgwire-side metadata like :pg/array-elem lives as ident-entity facts. For array column INSERTs we need that metadata available via (get-in schema [attr :pg/array-elem]), so this helper queries db for every ident's array-elem/ndim and merges the results into the schema map. Memoised per schema-identity — was ~0.7 ms/row of pure recomputation on the Pagila pg_dump replay before caching.

eval-check-predicate clj

(eval-check-predicate expr entity-map ns-str schema)

Evaluate a CHECK-style JSqlParser Expression against an entity map and return a tri-state: true (satisfied), false (violation), or nil (unknown — PG treats as satisfied). This is distinct from eval-update-expr which returns the arithmetic value of the expression; predicates need comparison / logical ops that only make sense at enforcement time.

Column refs resolve via eval-update-expr so any entity-map-aware coercion (namespace-qualified lookups) stays consistent between SET-value evaluation and CHECK evaluation.

eval-update-expr clj

(eval-update-expr value-expr entity-map ns-str schema)

Evaluate an UPDATE SET expression for a specific entity. For simple literals, returns the literal value. For expressions (col + 1, col * 2), evaluates against the entity's current values.

JdbcParameter placeholders return a ParamRef — the tx-build step runs substitute-params once bound values are available from Bind.

eval-values-literal clj

(eval-values-literal expr)

Evaluate a literal JSqlParser expression from a VALUES row. Handles simple literals, casts, and parenthesis; returns the raw value. Anything else returns :unhandled — the caller decides whether to fall back.

A JdbcParameter returns a ParamRef that the wire layer resolves at Bind time, allowing prepared INSERTs / UPDATE FROM VALUES with ?/$N.

extract-from-values clj

(extract-from-values update)

If an UPDATE's FROM clause is (VALUES (...), (...)) AS alias(col1, col2, ...), extract it. Returns {:alias str :cols [str] :rows [[literal ...] ...]} or nil.

extract-returning clj

(extract-returning returning-clause)

Extract RETURNING clause column names from a ReturningClause. Returns nil if no RETURNING, :* for RETURNING *, or [col-name ...] for specific columns.

extract-value clj

(extract-value e)

(extract-value e schema db)

Extract a Clojure value from a JSqlParser expression for INSERT VALUES. Optional schema+db params enable scalar subquery evaluation.

When the expression is a JdbcParameter (prepared-statement placeholder):

• if params/*bound-params* is bound (lexical-template fast path or execute-time re-translation), resolve the parameter inline to the bound value — no ParamRef leaves this function;
• otherwise emit a ParamRef sentinel that the wire layer resolves at Bind time against the decoded client value.

join-type clj

(join-type join)

Determine the type of a JOIN: :inner, :left, :right, :full, or :cross. LEFT JOIN and LEFT OUTER JOIN are both :left (JSqlParser may or may not set isOuter).

match-aggregate-index clj

(match-aggregate-index f find-elems find-aliases)

Try to find the index of an aggregate function in the find-elements. For COUNT(*) → look for (count ?x), for SUM(col) → (sum ?x) or (datahike.pg.sql/filter-sum ?x). Returns the 0-based index or nil.

Matches both the raw Datalog aggregate symbol and our ns-qualified null-filtering variant (filter-sum/avg/min/max/count[-distinct]) so HAVING clauses resolve regardless of which variant the SELECT projection emitted.

materialize-derived-select! clj

(materialize-derived-select! ps db schema)

Given a ParenthesedSelect in FROM/JOIN position, return an enriched db that has a virtual table populated with the subquery's results.

Handles two inner shapes:

1. (SELECT … FROM <table-or-derived> …) — runs the inner select against db and materializes its rows (existing behaviour).
2. (SELECT * FROM table_function(...) [WITH ORDINALITY]) — expands the table function directly without running a query.

Returns {:db spec-db :name sub-name :alias sub-alias :aliases cols} or nil if the shape isn't recognised.

materialize-recursive-cte! clj

(materialize-recursive-cte! wi target-name db schema)

Run a WITH RECURSIVE CTE rule to a fixed point and materialize the resulting rows into a speculative db under :<target-name>/<col> virtual attrs. Mirrors the result shape of materialize-set-op! so callers (parse-sql) can swap implementations based on (.isRecursive wi).

When the CTE is parameterised (a $n appears in its body, so in-args carries ParamRef sentinels that aren't bound until Bind), DATA can't be produced at parse time. We then enrich only the SCHEMA (column attrs, with value-types inferred from the anchor branch) and return a :deferred spec; the server re-runs the rule at Execute via materialize-recursive-rows! once the params are bound.

Reuses translate-recursive-cte for the rule construction; rule eval here is the SELECT counterpart of build-update-with-recursive-tx in the server.

materialize-recursive-iterative! clj

(materialize-recursive-iterative! wi target-name db schema)

FALLBACK recursive-CTE evaluator (B1): semi-naive iteration instead of a single Datalog rule. Used when materialize-recursive-cte!'s rule encoding can't represent the body (LEFT JOIN → not-join, correlated subqueries, nested recursion — e.g. asyncpg's typeinfo introspection).

Runs the anchor as an ordinary SELECT, materialises its rows under :<target>/<col>, then repeatedly runs the recursive branch — translated once against the seeded virtual table — folding NOVEL rows back in until a fixed point. Each iteration is a plain query the engine already handles.

Parameterised CTEs are DEFERRED to Execute (B2-style): when the anchor has a $n and a real FROM clause (asyncpg's FROM {typeinfo} ti WHERE ti.oid = any($1)), only the schema is enriched at parse and a :deferred {:kind :iterative …} spec is returned; the server runs the anchor with the bound params and iterates at Execute (materialize-recursive-iterative-rows!).

Returns the standard {:db :schema :name :alias :aliases [:deferred]} map, or nil when it can't apply: a non-UNION shape, a parameterised TABLE-FREE anchor (SELECT $1::int — the param constant-folds in translation, so the rule path / B2 must own it), or any translation/eval failure.

materialize-recursive-iterative-rows! clj

(materialize-recursive-iterative-rows!
  {:keys [anchor recursive col-names col-types target-name row-marker]}
  db)

Execute-time materialisation for a DEFERRED iterative recursive CTE (see materialize-recursive-iterative!). spec is the :deferred map with its anchor/recursive :in-args already param-substituted by resolve-param-refs. Runs the anchor against its parse-time enriched-db, folds the rows into the recursive branch's enriched-db (which carries the CTE schema + any derived tables it referenced), and iterates to a fixed point. The recursive step is TOLERANT — if an iteration throws (e.g. an array-membership join the engine can't resolve), we stop and keep what we have (asyncpg needs only the anchor rows for a composite of core-typed fields). Returns the data-enriched db (the caller's query-db, whose schema already has the CTE attrs); on anchor failure returns db unchanged.

materialize-recursive-rows! clj

(materialize-recursive-rows! {:keys [rule rule-name rule-vars col-names
                                     col-types in-params in-args target-name
                                     row-marker]}
                             db)

Execute-time counterpart for a DEFERRED recursive CTE (see materialize-recursive-cte!): ground the now-bound params into the rule (in-args already substituted by resolve-param-refs), run it to a fixed point, coerce the rows to the parse-time col-types, and db-with the data into db (whose schema already carries the CTE attrs from parse). Returns the data-enriched db; on rule-eval failure returns db unchanged so the outer query degrades to an empty CTE rather than crashing.

materialize-set-op! clj

(materialize-set-op! inner target-name db schema)

Run a SELECT (PlainSelect or SetOperationList) and persist its rows under target-name/<col> in a speculative db. Returns the same {:db :schema :name :alias :aliases} map shape as materialize-derived-select! so callers can swap them.

Used by both the derived-table path (FROM (...) AS t) and the CTE path (WITH t AS (...)), since SQL set operations over heterogeneous tables can't be expressed natively in Datalog — we have to flatten them into a single virtual table.

materialize-table-function clj

(materialize-table-function tf)

(materialize-table-function tf eval-fn)

Produce rows for a TableFunction FROM item. Supports the common constant-arg set-returning functions:

• unnest(ARRAY[…]) (+ WITH ORDINALITY)
• generate_series(start,stop[,step]) over integers (+ WITH ORDINALITY)
• now() / current_timestamp / {statement,transaction,clock}_timestamp (one row, current time)

Returns {:aliases [col-names] :rows [[v1 v2 …] …] :vtypes [kw kw …]} or nil if the function isn't one we expand.

eval-fn resolves an argument expression to a value; it defaults to srf-const-eval (literals only). The LATERAL nested-loop will pass an eval-fn that resolves correlated arguments per outer row — see srf-const-eval's note. That is why arguments flow through eval-fn here rather than being pattern-matched as literals inline.

parse-bytea-hex clj

(parse-bytea-hex s)

Decode a PostgreSQL bytea hex-format literal (\xDEADBEEF) to a byte array. Accepts both \x... and \\x... prefixes (JDBC/psycopg2 escape variants). Returns nil for values that don't look like hex bytea literals.

resolve-correlated-rows clj

(resolve-correlated-rows parse-fn parsed rows query-db inner-schema)

Resolve a parsed SELECT's deferred correlated subqueries against raw result rows: per outer row, run each subquery with the correlation columns bound into from-bindings, splice the value at its out-pos, and drop the hidden _corr columns. Returns [resolved-rows resolved-aliases resolved-oids], where resolved-oids carries each subquery's declared OID at its spliced position (nil for visible columns) so a caller materialising the result can type array columns from the subquery's OID rather than the runtime value class (e.g. array_agg(atttypid) → oid[] not int8[]). No-op (returns [rows find-aliases nil]) when there are no correlated subqueries.

run-recursive-rule clj

(run-recursive-rule db rule rule-name rule-vars in-params in-args)

Evaluate a recursive-CTE Datalog rule to a fixed point and return the raw result rows. in-args must already be free of ParamRef sentinels (substituted at Bind for the parameterised path).

Forces the query planner on regardless of caller context — Datahike's legacy engine can't evaluate the recursive bodies translate-recursive-cte emits (head var bound through a function op then filtered by a predicate, datahike PR #825).

select-item-alias clj

(select-item-alias item)

translate-cte-branch clj

(translate-cte-branch ps
                      cte-name
                      col-names
                      rule-vars
                      rule-name
                      schema
                      db
                      virtual-cte-schema)

Translate one branch of a recursive CTE (anchor or recursive PlainSelect) into Datalog rule body clauses.

• cte-name: the CTE's name (e.g. "__parent_store_compute")
• col-names: CTE column names in order (e.g. ["id" "parent_path"])
• rule-vars: corresponding rule output vars (e.g. [?id ?parent_path])
• rule-name: the rule's name as a symbol (for self-references)
• schema, db: from the outer context
• virtual-cte-schema: if non-nil, the CTE is referenceable as a virtual table in this branch (for the recursive branch only).

Returns a vector of clauses for use as a rule body, with the SELECT items bound to the rule output vars.

translate-delete clj

(translate-delete delete _schema)

Translate a DELETE statement to Datahike retraction query + tx-data.

translate-having-expr clj

(translate-having-expr expr find-elems find-aliases)

Translate a HAVING expression into a Clojure predicate fn form. The result is a map {:op symbol :col-idx int :value val} for simple cases, or a nested structure for AND/OR. The server applies this as a post-filter on result tuples.

translate-insert clj

(translate-insert insert schema db)

Translate an INSERT statement to Datahike transaction data. Supports single-row and multi-row VALUES, with or without column list. Handles ON CONFLICT (UPSERT) via :db.fn/call for atomic execution.

translate-join clj

(translate-join ctx join _default-table)

Add join clauses for a SQL JOIN to the context. For INNER joins with ref-based ON (a.ref_col = b.db_id), unifies the ref column variable with the target entity variable. For LEFT joins, records the ref-attr and right-table alias for later or-join wrapping in translate-select. Returns {:name str :alias str :join-type keyword :ref-attr kw :left-entity-var sym}.

translate-recursive-cte clj

(translate-recursive-cte wi schema db)

Translate a WITH RECURSIVE CTE definition into a Datalog rule. Returns: {:rule [...] :rule-name sym :col-names [...] :rule-vars [...] :in-params [...] :in-args [...]}

translate-select clj

(translate-select select schema & [db])

Translate a PlainSelect into a Datalog query map + metadata. Returns {:query map :find-aliases [...] :has-aggregates? bool}

translate-update clj

(translate-update update schema db)

Translate an UPDATE statement to Datahike retract+assert pairs. Handles UPDATE with WITH RECURSIVE CTE — for these, the result is {:type :update-with-recursive ...} containing the rule, columns, and target table info for the server to execute.