toucan2.pipeline — io.github.camsaul/toucan2 1.0.555

This is a low-level namespace implementing our query execution pipeline. Most of the stuff you'd use on a regular basis are implemented on top of stuff here.

Pipeline order is

1. toucan2.query/parse-args (entrypoint fn: [[transduce-unparsed]])
2. toucan2.model/resolve-model (entrypoint fn: [[transduce-parsed]])
3. resolve
4. transduce-query
5. build
6. compile
7. results-transform
8. transduce-execute-with-connection

The main pipeline entrypoint is [[transduce-unparsed]].

*build* clj

(*build* query-type model parsed-args resolved-query)

The function to use when building a query. Normally build, but you can bind this to intercept build behavior to do something different.

*compile* clj

(*compile* query-type model built-query)

The function to use when compiling a query. Normally compile, but you can bind this to intercept normal compilation behavior to do something different.

*parsed-args* clj

The parsed args seen at the beginning of the pipeline. This is bound in case methods in later stages of the pipeline, such as results-transform, need it for one reason or another. (See for example toucan2.tools.default-fields, which applies different behavior if a query was initiated with [model & columns] syntax vs. if it was not.)

*resolved-query* clj

The query after it has been resolved. This is bound in case methods in the later stages of the pipeline need it for one reason or another.

build clj

(build query-type₁ model₂ parsed-args resolved-query₃)

Build a query by applying parsed-args to resolved-query into something that can be compiled by compile, e.g. build a Honey SQL query by applying parsed-args to an initial resolved-query map.

You can implement this method to write a custom query compilation backend, for example to compile some certain record type in a special way. See toucan2.honeysql2 for an example implementation.

In addition to dispatching on query-type and model, this dispatches on the type of resolved-query, in a special way: for plain maps this will dispatch on the current [[map/backend]].

build is defined in toucan2.pipeline (toucan2/pipeline.clj:145).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• [:default :default clojure.lang.Sequential], defined in toucan2.pipeline (toucan2/pipeline.clj:186)

  It has the following documentation:

  Default implementation of vector [query & args] queries.

• [:default :default nil], defined in toucan2.pipeline (toucan2/pipeline.clj:164)

  It has the following documentation:

  Something like (select my-model nil) should basically mean SELECT * FROM my_model WHERE id IS NULL

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:160)

• [:toucan.query-type/delete.* :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:237)

  It has the following documentation:

  Build a Honey SQL 2 DELETE query.

  If the table for model should not be aliased (i.e., toucan2.model/namespace returns nil), builds a query that compiles to something like:

  DELETE FROM my_table
  WHERE ...
  

  If the table is aliased, this looks like

  DELETE FROM my_table AS t1
  WHERE ...
  

  for Postgres/H2/etc., or like

  DELETE t1
  FROM my_table AS t1
  WHERE ...
  

  for MySQL/MariaDB. MySQL/MariaDB does not seem to support aliases in DELETE FROM, so we need to use this alternative syntax; H2 doesn't support it however. So it has to be compiled differently based on the DB.

• [:toucan.query-type/select.count :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:140)

  It has the following documentation:

  Build an efficient count(*) query to power toucan2.select/count.

• [:toucan.query-type/update.* :default :default], defined in toucan2.update (toucan2/update.clj:31)

  It has the following documentation:

  Default method for building UPDATE queries. Code for building Honey SQL for UPDATE lives in toucan2.honeysql2.

  This doesn't really do much, but if the query has no :changes, returns the special flag :toucan2.pipeline/no-op.

• [:default :default java.lang.String], defined in toucan2.pipeline (toucan2/pipeline.clj:181)

  It has the following documentation:

  Default implementation for plain strings. Wrap the string in a vector and recurse.

• [:toucan.query-type/insert.* :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:163)

  It has the following documentation:

  Build a Honey SQL 2 INSERT query.

  if rows is just a single empty row then insert it with

  INSERT INTO table DEFAULT VALUES
  

  (Postgres/H2/etc.)

  or

  INSERT INTO table () VALUES ()
  

  (MySQL/MariaDB)

• [:toucan.query-type/insert.* :default :default], defined in toucan2.insert (toucan2/insert.clj:60)

  It has the following documentation:

  Default INSERT query method. No-ops if there are no :rows to insert in either parsed-args or resolved-query.

• [:toucan.query-type/update.* :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:196)

  It has the following documentation:

  Build a Honey SQL 2 UPDATE query.

• [:toucan.query-type/select.exists :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:148)

  It has the following documentation:

  Build an efficient query like SELECT exists(SELECT 1 FROM ...) query to power toucan2.select/exists?.

• [:toucan.query-type/insert.* :toucan2.tools.before-insert/before-insert clojure.lang.IPersistentMap], defined in toucan2.tools.before-insert (toucan2/tools/before_insert.clj:39)

  It has the following documentation:

  Apply [[before-insert]] to :rows in the resolved-query or parsed-args for Honey SQL queries.

• [:toucan.query-type/select.* :toucan2.tools.before-select/model :default], defined in toucan2.tools.before-select (toucan2/tools/before_select.clj:29)

• [:default :default java.lang.Long], defined in toucan2.pipeline (toucan2/pipeline.clj:176)

  It has the following documentation:

  Treat lone integers as queries to select an integer primary key.

• [:default :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:111)

  It has the following documentation:

  Base map backend implementation. Applies the :kv-args in parsed-args using query/apply-kv-args, and ignores other parsed args.

• [:toucan.query-type/select.instances.from-update :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:217)

  It has the following documentation:

  Treat the resolved query as a conditions map but otherwise behave the same as the :toucan.query-type/select.instances impl.

• [:toucan.query-type/select.* :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:120)

  It has the following documentation:

  Build a Honey SQL 2 SELECT query.

• [:default :default java.lang.Integer], defined in toucan2.pipeline (toucan2/pipeline.clj:171)

  It has the following documentation:

  Treat lone integers as queries to select an integer primary key.

compile clj

(compile query-type₁ model₂ built-query₃)

Compile a built-query to something that can be executed natively by the query execution backend, e.g. compile a Honey SQL map to a [sql & args] vector.

You can implement this method to write a custom query compilation backend, for example to compile some certain record type in a special way. See toucan2.honeysql2 for an example implementation.

In addition to dispatching on query-type and model, this dispatches on the type of built-query, in a special way: for plain maps this will dispatch on the current [[map/backend]].

compile is defined in toucan2.pipeline (toucan2/pipeline.clj:109).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:123)

  It has the following documentation:

  Default implementation: return query as-is (i.e., consider it to already be compiled). Check that the query is non-nil and, if it is a collection, non-empty. Everything else is fair game.

• [:default :default java.lang.String], defined in toucan2.pipeline (toucan2/pipeline.clj:137)

  It has the following documentation:

  Compile a string query. Default impl wraps the string in a vector and recursively calls compile.

• [:default :default clojure.lang.IPersistentMap], defined in toucan2.honeysql2 (toucan2/honeysql2.clj:277)

  It has the following documentation:

  Compile a Honey SQL 2 map to [sql & args].

compile* clj

(compile* built-query)

(compile* query-type built-query)

(compile* query-type model built-query)

Helper for compiling a built-query to something that can be executed natively.

default-rf clj

(default-rf query-type)

The default reducing function for queries of query-type. Used for non-reducible operations like toucan2.select/select or toucan2.execute/query.

default-rf is defined in toucan2.pipeline (toucan2/pipeline.clj:329).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :toucan.result-type/update-count, defined in toucan2.pipeline (toucan2/pipeline.clj:337)

  It has the following documentation:

  The reducing function for queries returning an update count. Sums all numbers passed in.

• :toucan.result-type/pks, defined in toucan2.pipeline (toucan2/pipeline.clj:344)

  It has the following documentation:

  The reducing function for queries returning PKs. Presumably these will come back as a map, but that map doesn't need to be realized. This needs to be combined with a transducer like map toucan2.model/select-pks-fn to get the PKs themselves.

• :toucan.result-type/*, defined in toucan2.pipeline (toucan2/pipeline.clj:351)

  It has the following documentation:

  The default reducing function for all query types unless otherwise specified. Returns realized maps (by default, Toucan 2 instances).

resolve clj

(resolve query-type₁ model₂ queryable₃)

Resolve a queryable to an actual query, e.g. resolve a named query defined by toucan2.tools.named-query to an actual Honey SQL map.

resolve is defined in toucan2.pipeline (toucan2/pipeline.clj:200).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:208)

  It has the following documentation:

  The default implementation considers a query to already be resolved, and returns it as-is.

results-transform clj

(results-transform query-type₁ model₂)

The transducer that should be applied to the reducing function executed when running a query of query-type (see toucan2.types) for model (nil if the query is ran without a model, e.g. with toucan2.execute/query). The default implementation returns identity; add your own implementations as desired to apply additional results transforms.

Be sure to comp the transform from next-method:

(m/defmethod t2/results-transform [:toucan.query-type/select.* :my-model]
  [query-type model]
  (comp (next-method query-type model)
        (map (fn [instance]
               (assoc instance :num-cans 2)))))

It's probably better to put the transducer returned by next-method first in the call to comp, because cond works like -> when composing transducers, and since next-method is by definition the less-specific method, it makes sense to call that transform before we apply our own. This means our own transforms will get to see the results of the previous stage, rather than vice-versa.

results-transform is defined in toucan2.pipeline (toucan2/pipeline.clj:80).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {[:toucan.result-type/instances :toucan2.tools.after-select/after-select] #{[:toucan.result-type/instances :toucan2.tools.after/model]}}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:105)

• [:toucan.result-type/instances :toucan2.tools.after/model], defined in toucan2.tools.after (toucan2/tools/after.clj:38)

• [:toucan.result-type/instances :toucan2.tools.after-select/after-select], defined in toucan2.tools.after-select (toucan2/tools/after_select.clj:20)

transduce-execute-with-connection clj

(transduce-execute-with-connection rf conn₁ query-type₂ model₃ compiled-query)

The final stage of the Toucan 2 query execution pipeline. Execute a compiled query (as returned by compile) with a database connection, e.g. a java.sql.Connection, and transduce results with reducing function rf.

The only reason you should need to implement this method is if you are writing a new query execution backend.

transduce-execute-with-connection is defined in toucan2.pipeline (toucan2/pipeline.clj:43).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {[:toucan2.jdbc.mysql-mariadb/connection :toucan.query-type/insert.pks :default] #{[java.sql.Connection :toucan.result-type/pks :default]}, [:toucan2.jdbc.mysql-mariadb/connection :toucan.query-type/update.pks :default] #{[java.sql.Connection :toucan.result-type/pks :default]}}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• [java.sql.Connection :default :default], defined in [[toucan2.jdbc.pipeline]] (toucan2/jdbc/pipeline.clj:9)

  It has the following documentation:

  Default impl for the JDBC query execution backend.

• [java.sql.Connection :toucan.result-type/pks :default], defined in [[toucan2.jdbc.pipeline]] (toucan2/jdbc/pipeline.clj:22)

  It has the following documentation:

  JDBC query execution backend for executing queries that return PKs (:toucan.result-type/pks).

  Applies transducer to call toucan2.model/select-pks-fn on each result row.

• [java.sql.Connection :toucan2.jdbc.pipeline/DML-queries-returning-instances :default], defined in [[toucan2.jdbc.pipeline]] (toucan2/jdbc/pipeline.clj:52)

  It has the following documentation:

  DML queries like UPDATE or INSERT don't usually support returning instances, at least not with JDBC. So for these situations we'll fake it by first running an equivalent query returning inserted/affected PKs, and then do a subsequent SELECT to get those rows. Then we'll reduce the rows with the original reducing function.

• [:toucan2.jdbc.mysql-mariadb/connection :toucan.query-type/insert.pks :default], defined in [[toucan2.jdbc.mysql-mariadb]] (toucan2/jdbc/mysql_mariadb.clj:46)

  It has the following documentation:

  Apparently RETURN_GENERATED_KEYS doesn't work for MySQL/MariaDB if:

  1. Values for the primary key are specified in the INSERT itself, and

  2. The primary key is not an integer.

  So to work around this we will look at the rows we're inserting: if every rows specifies the primary key column(s) (including nil values), we'll transduce those specified values rather than what JDBC returns.

  This seems like it won't work if these values were arbitrary Honey SQL expressions. I suppose we could work around THAT problem by running the primary key values thru another SELECT query... but that just seems like too much. I guess we can cross that bridge when we get there.

• [:toucan2.jdbc.mysql-mariadb/connection :toucan.query-type/update.pks :default], defined in [[toucan2.jdbc.mysql-mariadb]] (toucan2/jdbc/mysql_mariadb.clj:98)

  It has the following documentation:

  MySQL and MariaDB don't support returning PKs for UPDATE. Execute a SELECT query to capture the PKs of the rows that will be affected BEFORE performing the UPDATE. We need to capture PKs for both :toucan.query-type/update.pks and for :toucan.query-type/update.instances, since ultimately the latter is implemented on top of the former.

These aux methods are known:

:before methods:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:57)

  It has the following documentation:

  Count all queries that are executed by calling [[call-count-thunk]] if bound.

transduce-query clj

(transduce-query rf query-type₁ model₂ parsed-args resolved-query₃)

One of the primary customization points for the Toucan 2 query execution pipeline. build and compile a resolved-query, then open a connection, execute the query, and transduce the results with transduce-execute-with-connection (using the results-transform).

You can implement this method to introduce custom behavior that should happen before a query is built or compiled, e.g. transformations to the parsed-args or other shenanigans like changing underlying query type being executed (e.g. toucan2.tools.after, which 'upgrades' queries returning update counts or PKs to ones returning instances so toucan2.tools.after-update and toucan2.tools.after-insert can be applied to affected rows).

transduce-query is defined in toucan2.pipeline (toucan2/pipeline.clj:254).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:269)

• [:toucan2.tools.after/query-type :toucan2.tools.after/model :default], defined in toucan2.tools.after (toucan2/tools/after.clj:84)

  It has the following documentation:

  'Upgrade' a query so that it returns instances, and run the upgraded query so that we can apply [[each-row-fn]] to the results. Then apply [[result-type-rf]] to the results of the original expected type are ultimately returned.

• [:toucan.query-type/delete.* :toucan2.tools.before-delete/before-delete :default], defined in toucan2.tools.before-delete (toucan2/tools/before_delete.clj:43)

  It has the following documentation:

  Do a recursive SELECT query with the args passed to delete!; apply [[before-delete]] to all matching rows. Then call the next-method. This is all done inside of a transaction.

These aux methods are known:

:around methods:

• [:toucan.query-type/insert.* :toucan2.tools.before-insert/before-insert :default], defined in toucan2.tools.before-insert (toucan2/tools/before_insert.clj:31)

  It has the following documentation:

  Execute [[before-insert]] methods and the INSERT query inside a transaction.