Middleware for adding remapping and other dimension related projections. This remaps Fields that have a corresponding
Dimension object (which defines a remapping) in two different ways, depending on the :type
attribute of the
Dimension:
external
type Dimensions mean the Field's values will be replaced with corresponding values from a column on a
different table, joined via a foreign key. A common use-case would be to replace FK IDs with the name of whatever it
references, for example replacing a values of venue.category_id
with values of category.name
. Actual replacement
of values happens on the frontend, so this middleware simply adds the column to be used for replacement (e.g.
category.name
) to the :fields
clause in pre-processing, so the Field will be fetched. Recall that Fields
referenced via with :fk->
clauses imply that JOINs will take place, which are automatically handled later in the
Query Processor pipeline. Additionally, this middleware will swap out and :order-by
clauses referencing the
original Field with ones referencing the remapped Field (for example, so we would sort by category.name
instead of
category_id
).
internal
type Dimensions mean the Field's values are replaced by a user-defined map of values, stored in the
human_readable_values
column of a corresponding FieldValues
object. A common use-case for this scenario would be
to replace integer enum values with something more descriptive, for example replacing values of an enum can_type
-- 0
becomes Toucan
, 1
becomes Pelican
, and so forth. This is handled exclusively in post-processing by
adding extra columns and values to the results.
In both cases, to accomplish values replacement on the frontend, the post-processing part of this middleware adds
appropriate :remapped_from
and :remapped_to
attributes in the result :cols
in post-processing.
:remapped_from
and :remapped_to
are the names of the columns, e.g. category_id
is :remapped_to
name
, and
name
is :remapped_from
:category_id
.
Middleware for adding remapping and other dimension related projections. This remaps Fields that have a corresponding Dimension object (which defines a remapping) in two different ways, depending on the `:type` attribute of the Dimension: `external` type Dimensions mean the Field's values will be replaced with corresponding values from a column on a different table, joined via a foreign key. A common use-case would be to replace FK IDs with the name of whatever it references, for example replacing a values of `venue.category_id` with values of `category.name`. Actual replacement of values happens on the frontend, so this middleware simply adds the column to be used for replacement (e.g. `category.name`) to the `:fields` clause in pre-processing, so the Field will be fetched. Recall that Fields referenced via with `:fk->` clauses imply that JOINs will take place, which are automatically handled later in the Query Processor pipeline. Additionally, this middleware will swap out and `:order-by` clauses referencing the original Field with ones referencing the remapped Field (for example, so we would sort by `category.name` instead of `category_id`). `internal` type Dimensions mean the Field's values are replaced by a user-defined map of values, stored in the `human_readable_values` column of a corresponding `FieldValues` object. A common use-case for this scenario would be to replace integer enum values with something more descriptive, for example replacing values of an enum `can_type` -- `0` becomes `Toucan`, `1` becomes `Pelican`, and so forth. This is handled exclusively in post-processing by adding extra columns and values to the results. In both cases, to accomplish values replacement on the frontend, the post-processing part of this middleware adds appropriate `:remapped_from` and `:remapped_to` attributes in the result `:cols` in post-processing. `:remapped_from` and `:remapped_to` are the names of the columns, e.g. `category_id` is `:remapped_to` `name`, and `name` is `:remapped_from` `:category_id`.
Middlware for adding an implicit :fields
and :order-by
clauses to certain queries.
Middlware for adding an implicit `:fields` and `:order-by` clauses to certain queries.
Middleware that creates corresponding :joins
for Tables referred to by :fk->
clauses and replaces those clauses
with :joined-field
clauses.
Middleware that creates corresponding `:joins` for Tables referred to by `:fk->` clauses and replaces those clauses with `:joined-field` clauses.
Middleware for adding :row_count
and :status
info to QP results.
Middleware for adding `:row_count` and `:status` info to QP results.
Middleware for adding a :settings
map to a query before it is processed.
Middleware for adding a `:settings` map to a query before it is processed.
Middleware for annotating (adding type information to) the results of a query, under the :cols
column.
Middleware for annotating (adding type information to) the results of a query, under the `:cols` column.
Middleware for implementing async QP behavior.
Middleware for implementing async QP behavior.
Middleware that limits the number of concurrent queries for each database.
Each connected database is limited to a maximum of 15 simultaneous queries (configurable) using these methods; any additional queries will park the thread. Super-useful for writing high-performance API endpoints. Prefer these methods to the old-school synchronous versions.
How is this achieved? For each Database, we'll maintain a thread pool executor to limit the number of simultaneous queries.
Middleware that limits the number of concurrent queries for each database. Each connected database is limited to a maximum of 15 simultaneous queries (configurable) using these methods; any additional queries will park the thread. Super-useful for writing high-performance API endpoints. Prefer these methods to the old-school synchronous versions. How is this achieved? For each Database, we'll maintain a thread pool executor to limit the number of simultaneous queries.
Middleware for automatically bucketing unbucketed :type/DateTime
(but not :type/Time
) Fields with :day
bucketing. Applies to any unbucketed Field in a breakout, or fields in a filter clause being compared against
yyyy-MM-dd
format datetime strings.
Middleware for automatically bucketing unbucketed `:type/DateTime` (but not `:type/Time`) Fields with `:day` bucketing. Applies to any unbucketed Field in a breakout, or fields in a filter clause being compared against `yyyy-MM-dd` format datetime strings.
Middleware that handles binning-strategy
Field clauses. This adds a resolved-options
map to every
binning-strategy
clause that contains the information query processors will need in order to perform binning.
Middleware that handles `binning-strategy` Field clauses. This adds a `resolved-options` map to every `binning-strategy` clause that contains the information query processors will need in order to perform binning.
Middleware that returns cached results for queries when applicable.
If caching is enabled (enable-query-caching
is true
) cached results will be returned for Cards if possible.
There's a global default TTL defined by the setting query-caching-default-ttl
, but individual Cards can override
this value with custom TTLs with a value for :cache_ttl
.
For all other queries, caching is skipped.
Various caching backends are defined in metabase.query-processor.middleware.cache-backend
namespaces. The default
backend is db
, which uses the application database; this value can be changed by setting the env var
MB_QP_CACHE_BACKEND
.
Refer to metabase.query-processor.middleware.cache-backend.interface
for more details about how the cache
backends themselves.
Middleware that returns cached results for queries when applicable. If caching is enabled (`enable-query-caching` is `true`) cached results will be returned for Cards if possible. There's a global default TTL defined by the setting `query-caching-default-ttl`, but individual Cards can override this value with custom TTLs with a value for `:cache_ttl`. For all other queries, caching is skipped. Various caching backends are defined in `metabase.query-processor.middleware.cache-backend` namespaces. The default backend is `db`, which uses the application database; this value can be changed by setting the env var `MB_QP_CACHE_BACKEND`. Refer to `metabase.query-processor.middleware.cache-backend.interface` for more details about how the cache backends themselves.
Interface used to define different Query Processor cache backends. Defining a backend is straightforward: define a new namespace with the pattern
metabase.query-processor.middleware.cache-backend.<backend>
Where backend is a key representing the backend, e.g. db
, redis
, or memcached
.
In that namespace, create an object that reifies (or otherwise implements) IQueryProcessorCacheBackend
.
This object must be stored in a var called instance
.
That's it. See metabase.query-processor.middleware.cache-backend.db
for a complete example of how this is done.
Interface used to define different Query Processor cache backends. Defining a backend is straightforward: define a new namespace with the pattern metabase.query-processor.middleware.cache-backend.<backend> Where backend is a key representing the backend, e.g. `db`, `redis`, or `memcached`. In that namespace, create an object that reifies (or otherwise implements) `IQueryProcessorCacheBackend`. This object *must* be stored in a var called `instance`. That's it. See `metabase.query-processor.middleware.cache-backend.db` for a complete example of how this is done.
Middleware for catching exceptions thrown by the query processor and returning them in a friendlier format.
Middleware for catching exceptions thrown by the query processor and returning them in a friendlier format.
Middleware that adds default constraints to limit the maximum number of rows returned to queries that specify the
:add-default-userland-constraints?
:middleware
option.
Middleware that adds default constraints to limit the maximum number of rows returned to queries that specify the `:add-default-userland-constraints?` `:middleware` option.
Middlware for handling cumulative count and cumulative sum aggregations.
Middlware for handling cumulative count and cumulative sum aggregations.
Middleware that's only active in dev and test scenarios. These middleware functions do additional checks of query processor behavior that are undesirable in normal production use.
Middleware that's only active in dev and test scenarios. These middleware functions do additional checks of query processor behavior that are undesirable in normal production use.
Middleware that hands off to a driver's implementation of process-query-in-context
, if any.
If implemented, this effectively lets one inject custom driver-specific middleware for the QP.
Drivers can use it to different things like rewrite queries as needed or perform special permissions checks.
Middleware that hands off to a driver's implementation of `process-query-in-context`, if any. If implemented, this effectively lets one inject custom driver-specific middleware for the QP. Drivers can use it to different things like rewrite queries as needed or perform special permissions checks.
Middleware for expanding :metric
and :segment
'macros' in unexpanded MBQL queries.
(:metric
forms are expanded into aggregations and sometimes filter clauses, while :segment
forms are expanded
into filter clauses.)
TODO - this namespace is ancient and written with MBQL '95 in mind, e.g. it is case-sensitive. At some point this ought to be reworked to be case-insensitive and cleaned up.
Middleware for expanding `:metric` and `:segment` 'macros' in *unexpanded* MBQL queries. (`:metric` forms are expanded into aggregations and sometimes filter clauses, while `:segment` forms are expanded into filter clauses.) TODO - this namespace is ancient and written with MBQL '95 in mind, e.g. it is case-sensitive. At some point this ought to be reworked to be case-insensitive and cleaned up.
Middleware responsible for 'hydrating' the source query for queries that use another query as their source. This middleware looks for MBQL queries like
{:source-table "card__1" ; Shorthand for using Card 1 as source query ...}
and resolves the referenced source query, transforming the query to look like the following:
{:source-query {...} ; Query for Card 1 :source-metadata [...] ; metadata about columns in Card 1 ...}
This middleware resolves Card ID :source-table
s at all levels of the query, but the top-level query often uses the
so-called virtual-id
, because the frontend client might not know the original Database; this middleware will
replace that ID with the approiate ID, e.g.
{:database <virtual-id>, :type :query, :query {:source-table "card__1"}} -> {:database 1, :type :query, :query {:source-query {...}, :source-metadata {...}}}
TODO - consider renaming this namespace to metabase.query-processor.middleware.resolve-card-id-source-tables
Middleware responsible for 'hydrating' the source query for queries that use another query as their source. This middleware looks for MBQL queries like {:source-table "card__1" ; Shorthand for using Card 1 as source query ...} and resolves the referenced source query, transforming the query to look like the following: {:source-query {...} ; Query for Card 1 :source-metadata [...] ; metadata about columns in Card 1 ...} This middleware resolves Card ID `:source-table`s at all levels of the query, but the top-level query often uses the so-called `virtual-id`, because the frontend client might not know the original Database; this middleware will replace that ID with the approiate ID, e.g. {:database <virtual-id>, :type :query, :query {:source-table "card__1"}} -> {:database 1, :type :query, :query {:source-query {...}, :source-metadata {...}}} TODO - consider renaming this namespace to `metabase.query-processor.middleware.resolve-card-id-source-tables`
Middleware that formats the results of a query. Currently, the only thing this does is convert datetime types to ISO-8601 strings in the appropriate timezone.
Middleware that formats the results of a query. Currently, the only thing this does is convert datetime types to ISO-8601 strings in the appropriate timezone.
Middleware that handles limiting the maximum number of rows returned by a query.
Middleware that handles limiting the maximum number of rows returned by a query.
Middleware for logging a query before it is processed. (Various other middleware functions log the query as well in different stages.)
Middleware for logging a query before it is processed. (Various other middleware functions log the query as well in different stages.)
Middleware responsible for converting MBQL queries to native queries (by calling the driver's QP methods) so the query can then be executed.
Middleware responsible for converting MBQL queries to native queries (by calling the driver's QP methods) so the query can then be executed.
Middleware that converts a query into a normalized, canonical form.
Middleware that converts a query into a normalized, canonical form.
Middleware for substituting parameters in queries.
Middleware for substituting parameters in queries.
Shared code for handling datetime parameters, used by both MBQL and native params implementations.
Shared code for handling datetime parameters, used by both MBQL and native params implementations.
Code for handling parameter substitution in MBQL queries.
Code for handling parameter substitution in MBQL queries.
Param substitution for SQL queries. This is a new implementation, fondly referred to as 'SQL parameters 2.0', written for v0.23.0. The new implementation uses prepared statement args instead of substituting them directly into the query, and is much better-organized and better-documented.
Param substitution for *SQL* queries. This is a new implementation, fondly referred to as 'SQL parameters 2.0', written for v0.23.0. The new implementation uses prepared statement args instead of substituting them directly into the query, and is much better-organized and better-documented.
Middleware for checking that the current user has permissions to run the current query.
Middleware for checking that the current user has permissions to run the current query.
Middleware related to doing extra steps for queries that are ran via API endpoints (i.e., most of them -- as opposed to queries ran internally e.g. as part of the sync process). These include things like saving QueryExecutions and formatting the results.
Middleware related to doing extra steps for queries that are ran via API endpoints (i.e., most of them -- as opposed to queries ran internally e.g. as part of the sync process). These include things like saving QueryExecutions and formatting the results.
SQL places restrictions when using a GROUP BY
clause (MBQL :breakout
) in combination with an ORDER BY
clause (MBQL :order-by
) -- columns that appear in the ORDER BY
must appear in the GROUP BY
. When we apply
datetime or binning bucketing in a breakout, for example cast(x AS DATE)
(MBQL :datetime-field
clause), we need
to apply the same bucketing to instances of that Field in the order-by
clause. In other words:
Bad:
SELECT count(*) FROM table GROUP BY CAST(x AS date) ORDER BY x ASC
(MBQL)
{:source-table 1 :breakout [[:datetime-field [:field-id 1] :day]] :order-by [[:asc [:field-id 1]]]}
Good:
SELECT count(*) FROM table GROUP BY CAST(x AS date) ORDER BY CAST(x AS date) ASC
(MBQL)
{:source-table 1 :breakout [[:datetime-field [:field-id 1] :day]] :order-by [[:asc [:datetime-field [:field-id 1] :day]]]}
The frontend, on the rare occasion it generates a query that explicitly specifies an order-by
clause, usually will
generate one that directly corresponds to the bad example above. This middleware finds these cases and rewrites the
query to look like the good example.
SQL places restrictions when using a `GROUP BY` clause (MBQL `:breakout`) in combination with an `ORDER BY` clause (MBQL `:order-by`) -- columns that appear in the `ORDER BY` must appear in the `GROUP BY`. When we apply datetime or binning bucketing in a breakout, for example `cast(x AS DATE)` (MBQL `:datetime-field` clause), we need to apply the same bucketing to instances of that Field in the `order-by` clause. In other words: Bad: SELECT count(*) FROM table GROUP BY CAST(x AS date) ORDER BY x ASC (MBQL) {:source-table 1 :breakout [[:datetime-field [:field-id 1] :day]] :order-by [[:asc [:field-id 1]]]} Good: SELECT count(*) FROM table GROUP BY CAST(x AS date) ORDER BY CAST(x AS date) ASC (MBQL) {:source-table 1 :breakout [[:datetime-field [:field-id 1] :day]] :order-by [[:asc [:datetime-field [:field-id 1] :day]]]} The frontend, on the rare occasion it generates a query that explicitly specifies an `order-by` clause, usually will generate one that directly corresponds to the bad example above. This middleware finds these cases and rewrites the query to look like the good example.
Middleware for resolving the appropriate driver to use for processing a query.
Middleware for resolving the appropriate driver to use for processing a query.
Middleware that resolves the Fields referenced by a query.
Middleware that resolves the Fields referenced by a query.
Middleware that fetches tables that will need to be joined, referred to by fk->
clauses, and adds information to
the query about what joins should be done and how they should be performed.
Middleware that fetches tables that will need to be joined, referred to by `fk->` clauses, and adds information to the query about what joins should be done and how they should be performed.
Fetches Tables corresponding to any :source-table
IDs anywhere in the query.
Fetches Tables corresponding to any `:source-table` IDs anywhere in the query.
Middleware that stores metadata about results column types after running a query for a Card, and returns that metadata (which can be passed back to the backend when saving a Card) as well as a checksum in the API response.
Middleware that stores metadata about results column types after running a query for a Card, and returns that metadata (which can be passed *back* to the backend when saving a Card) as well as a checksum in the API response.
The store middleware is responsible for initializing a fresh QP Store, which caches resolved objects for the duration
of a query execution. See metabase.query-processor.store
for more details.
The store middleware is responsible for initializing a fresh QP Store, which caches resolved objects for the duration of a query execution. See `metabase.query-processor.store` for more details.
Middleware for checking that a normalized query is valid.
Middleware for checking that a normalized query is valid.
Middleware that wraps value literals in value
/absolute-datetime
/etc. clauses containing relevant type
information; parses datetime string literals when appropriate.
Middleware that wraps value literals in `value`/`absolute-datetime`/etc. clauses containing relevant type information; parses datetime string literals when appropriate.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close