toucan2.pipeline — io.github.camsaul/toucan2 1.0.513

This is a low-level namespace implementing our query execution pipeline. Most of the stuff you'd use on a regular basis are implemented on top of stuff here.

Pipeline order is

1. toucan2.query/parse-args (entrypoint fn: [[transduce-unparsed]])
2. toucan2.model/resolve-model (entrypoint fn: [[transduce-parsed]])
3. resolve
4. transduce-query
5. build
6. compile
7. results-transform
8. transduce-execute-with-connection

The main pipeline entrypoint is [[transduce-unparsed]].

*build* clj

(*build* query-type model parsed-args resolved-query)

The function to use when building a query. Normally build, but you can bind this to intercept build behavior to do something different.

*compile* clj

(*compile* query-type model built-query)

The function to use when compiling a query. Normally compile, but you can bind this to intercept normal compilation behavior to do something different.

*parsed-args* clj

The parsed args seen at the beginning of the pipeline. This is bound in case methods in later stages of the pipeline, such as results-transform, need it for one reason or another. (See for example toucan2.tools.default-fields, which applies different behavior if a query was initiated with [model & columns] syntax vs. if it was not.)

*resolved-query* clj

The query after it has been resolved. This is bound in case methods in the later stages of the pipeline need it for one reason or another.

build clj

(build query-type₁ model₂ parsed-args resolved-query₃)

Build a query by applying parsed-args to resolved-query into something that can be compiled by compile, e.g. build a Honey SQL query by applying parsed-args to an initial resolved-query map.

You should implement this method when writing a custom map backend; see toucan2.map-backend for more information.

In addition to dispatching on query-type and model, this dispatches on the type of resolved-query, in a special way: for plain maps this will dispatch on the current [[map/backend]].

build is defined in toucan2.pipeline (toucan2/pipeline.clj:173).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• [:default :default java.lang.String], defined in toucan2.pipeline (toucan2/pipeline.clj:217)

  It has the following documentation:

  Default implementation for plain strings. Wrap the string in a vector and recurse.

• [:default :default nil], defined in toucan2.pipeline (toucan2/pipeline.clj:193)

  It has the following documentation:

  Something like (select my-model nil) should basically mean SELECT * FROM my_model WHERE id IS NULL

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:189)

• [:default :default java.lang.Integer], defined in toucan2.pipeline (toucan2/pipeline.clj:200)

  It has the following documentation:

  Treat lone integers as queries to select an integer primary key.

• [:default :default java.lang.Long], defined in toucan2.pipeline (toucan2/pipeline.clj:205)

  It has the following documentation:

  Treat lone integers as queries to select an integer primary key.

• [:toucan.query-type/delete.* :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:213)

  It has the following documentation:

  Build a Honey SQL 2 DELETE query.

  If the table for model should not be aliased (i.e., toucan2.model/namespace returns nil), builds a query that compiles to something like:

  DELETE FROM my_table
  WHERE ...
  

  If the table is aliased, this looks like

  DELETE FROM my_table AS t1
  WHERE ...
  

  for Postgres/H2/etc., or like

  DELETE t1
  FROM my_table AS t1
  WHERE ...
  

  for MySQL/MariaDB. MySQL/MariaDB does not seem to support aliases in DELETE FROM, so we need to use this alternative syntax; H2 doesn't support it however. So it has to be compiled differently based on the DB.

• [:toucan.query-type/select.instances.from-update :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:193)

  It has the following documentation:

  Treat the resolved query as a conditions map but otherwise behave the same as the :toucan.query-type/select.instances impl.

• [:toucan.query-type/insert.* :default :default], defined in toucan2.insert (toucan2/insert.clj:60)

  It has the following documentation:

  Default INSERT query method. No-ops if there are no :rows to insert in either parsed-args or resolved-query.

• [:default :default clojure.lang.Sequential], defined in toucan2.pipeline (toucan2/pipeline.clj:222)

  It has the following documentation:

  Default implementation of vector [query & args] queries.

• [:toucan.query-type/insert.* :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:139)

  It has the following documentation:

  Build a Honey SQL 2 INSERT query.

  if rows is just a single empty row then insert it with

  INSERT INTO table DEFAULT VALUES
  

  (Postgres/H2/etc.)

  or

  INSERT INTO table () VALUES ()
  

  (MySQL/MariaDB)

• [:default :default :toucan.map-backend/*], defined in toucan2.pipeline (toucan2/pipeline.clj:210)

  It has the following documentation:

  Base map backend implementation. Applies the :kv-args in parsed-args using query/apply-kv-args, and ignores other parsed args.

• [:toucan.query-type/select.* :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:113)

  It has the following documentation:

  Build a Honey SQL 2 SELECT query.

• [:toucan.query-type/update.* :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:172)

  It has the following documentation:

  Build a Honey SQL 2 UPDATE query.

compile clj

(compile query-type₁ model₂ built-query₃)

Compile a built-query to something that can be executed natively by the query execution backend, e.g. compile a Honey SQL map to a [sql & args] vector.

You should implement this method when writing a custom map backend; see toucan2.map-backend for more information.

In addition to dispatching on query-type and model, this dispatches on the type of built-query, in a special way: for plain maps this will dispatch on the current [[map/backend]].

compile is defined in toucan2.pipeline (toucan2/pipeline.clj:137).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:153)

  It has the following documentation:

  Default implementation: return query as-is (i.e., consider it to already be compiled). Check that the query is non-nil and, if it is a collection, non-empty. Everything else is fair game.

• [:default :default java.lang.String], defined in toucan2.pipeline (toucan2/pipeline.clj:167)

  It has the following documentation:

  Compile a string query. Default impl wraps the string in a vector and recursively calls compile.

• [:default :default :toucan.map-backend/honeysql2], defined in toucan2.map-backend.honeysql2 (toucan2/map_backend/honeysql2.clj:253)

  It has the following documentation:

  Compile a Honey SQL 2 map to [sql & args].

compile* clj

(compile* built-query)

(compile* query-type built-query)

(compile* query-type model built-query)

Helper for compiling a built-query to something that can be executed natively.

default-rf clj

(default-rf query-type)

The default reducing function for queries of query-type. Used for non-reducible operations like toucan2.select/select or toucan2.execute/query.

default-rf is defined in toucan2.pipeline (toucan2/pipeline.clj:369).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :toucan.result-type/update-count, defined in toucan2.pipeline (toucan2/pipeline.clj:377)

  It has the following documentation:

  The reducing function for queries returning an update count. Sums all numbers passed in.

• :toucan.result-type/pks, defined in toucan2.pipeline (toucan2/pipeline.clj:384)

  It has the following documentation:

  The reducing function for queries returning PKs. Presumably these will come back as a map, but that map doesn't need to be realized. This needs to be combined with a transducer like map toucan2.model/select-pks-fn to get the PKs themselves.

• :toucan.result-type/*, defined in toucan2.pipeline (toucan2/pipeline.clj:391)

  It has the following documentation:

  The default reducing function for all query types unless otherwise specified. Returns realized maps (by default, Toucan 2 instances).

query-dispatch-value clj

(query-dispatch-value query)

Dispatch value for a resolved or built query, e.g. a Honey SQL map. Dispatch value is determined in this order:

1. If the query is a map, but not a record type:

   1. Dispatch on :type metadata if present;

   2. otherwise dispatch on the current [[map/backend]].

2. If query is not a plain map, dispatch on [[protocols/dispatch-value]]. For a keyword, this is itself; otherwise it is normally the result of type.

This lets us dispatch on the default [[map/backend]] when we encounter bare Clojure maps with no :type metadata.

resolve clj

(resolve query-type₁ model₂ queryable₃)

Resolve a queryable to an actual query, e.g. resolve a named query defined by toucan2.tools.named-query to an actual Honey SQL map.

resolve is defined in toucan2.pipeline (toucan2/pipeline.clj:234).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:242)

  It has the following documentation:

  The default implementation considers a query to already be resolved, and returns it as-is.

results-transform clj

(results-transform query-type₁ model₂)

The transducer that should be applied to the reducing function executed when running a query of query-type (see toucan2.types) for model (nil if the query is ran without a model, e.g. with toucan2.execute/query). The default implementation returns identity; add your own implementations as desired to apply additional results transforms.

Be sure to comp the transform from next-method:

(m/defmethod t2/results-transform [:toucan.query-type/select.* :my-model]
  [query-type model]
  (comp (next-method query-type model)
        (map (fn [instance]
               (assoc instance :num-cans 2)))))

It's probably better to put the transducer returned by next-method first in the call to comp, because cond works like -> when composing transducers, and since next-method is by definition the less-specific method, it makes sense to call that transform before we apply our own. This means our own transforms will get to see the results of the previous stage, rather than vice-versa.

results-transform is defined in toucan2.pipeline (toucan2/pipeline.clj:88).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:113)

transduce-execute-with-connection clj

(transduce-execute-with-connection rf conn₁ query-type₂ model₃ compiled-query)

The final stage of the Toucan 2 query execution pipeline. Execute a compiled query (as returned by compile) with a database connection, e.g. a java.sql.Connection, and transduce results with reducing function rf.

The only reason you should need to implement this method is if you are writing a new query execution backend. If needed, you can specify initialization logic for your query execution backend by implementing toucan2.query-execution-backend/load-backend-if-needed for your connection class.

transduce-execute-with-connection is defined in toucan2.pipeline (toucan2/pipeline.clj:46).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These aux methods are known:

:before methods:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:63)

  It has the following documentation:

  Count all queries that are executed by calling [[call-count-thunk]] if bound.

transduce-query clj

(transduce-query rf query-type₁ model₂ parsed-args resolved-query₃)

One of the primary customization points for the Toucan 2 query execution pipeline. build and compile a resolved-query, then open a connection, execute the query, and transduce the results with transduce-execute-with-connection (using the results-transform).

You can implement this method to introduce custom behavior that should happen before a query is built or compiled, e.g. transformations to the parsed-args or other shenanigans like changing underlying query type being executed (e.g. toucan2.tools.after, which 'upgrades' queries returning update counts or PKs to ones returning instances so toucan2.tools.after-update and toucan2.tools.after-insert can be applied to affected rows).

As with build and compile, the dispatch value of resolved-query uses special rules, and is calculated with query-dispatch-value rather than [[protocols/dispatch-value]]; map queries with no :type metadata will dispatch on the [[map/backend]] rather than on clojure.lang.IPersistentMap (or similar).

transduce-query is defined in toucan2.pipeline (toucan2/pipeline.clj:288).

It caches methods using a methodical.impl.cache.watching.WatchingCache.

It uses the method combination methodical.impl.combo.threaded.ThreadingMethodCombination with the threading strategy :thread-last.

It uses the dispatcher methodical.impl.dispatcher.multi_default.MultiDefaultDispatcher with hierarchy #'clojure.core/global-hierarchy and prefs {}.

The default value is :default.

It uses the method table methodical.impl.method_table.standard.StandardMethodTable.

These primary methods are known:

• :default, defined in toucan2.pipeline (toucan2/pipeline.clj:309)