Main entrypoint
PuppetDB consists of several, cooperating components:
Command processing
PuppetDB uses a CQRS pattern for making changes to its domain objects (facts, catalogs, etc). Instead of simply submitting data to PuppetDB and having it figure out the intent, the intent needs to explicitly be codified as part of the operation. This is known as a "command" (e.g. "replace the current facts for node X").
Commands are processed asynchronously, however we try to do our best to ensure that once a command has been accepted, it will eventually be executed. Ordering is also preserved. To do this, all incoming commands are placed in a message queue which the command processing subsystem reads from in FIFO order.
Refer to puppetlabs.puppetdb.command for more details.
Message queue
We use stockpile to durably store commands. The "in memory" representation of that queue is a core.async channel.
REST interface
All interaction with PuppetDB is conducted via its REST API. We embed an instance of Jetty to handle web server duties. Commands that come in via REST are relayed to the message queue. Read-only requests are serviced synchronously.
Database sweeper
As catalogs are modified, unused records may accumulate and stale data may linger in the database. We periodically sweep the database, compacting it and performing regular cleanup so we can maintain acceptable performance.
Main entrypoint PuppetDB consists of several, cooperating components: * Command processing PuppetDB uses a CQRS pattern for making changes to its domain objects (facts, catalogs, etc). Instead of simply submitting data to PuppetDB and having it figure out the intent, the intent needs to explicitly be codified as part of the operation. This is known as a "command" (e.g. "replace the current facts for node X"). Commands are processed asynchronously, however we try to do our best to ensure that once a command has been accepted, it will eventually be executed. Ordering is also preserved. To do this, all incoming commands are placed in a message queue which the command processing subsystem reads from in FIFO order. Refer to `puppetlabs.puppetdb.command` for more details. * Message queue We use stockpile to durably store commands. The "in memory" representation of that queue is a core.async channel. * REST interface All interaction with PuppetDB is conducted via its REST API. We embed an instance of Jetty to handle web server duties. Commands that come in via REST are relayed to the message queue. Read-only requests are serviced synchronously. * Database sweeper As catalogs are modified, unused records may accumulate and stale data may linger in the database. We periodically sweep the database, compacting it and performing regular cleanup so we can maintain acceptable performance.
(auto-expire-nodes! node-ttl db)Expire nodes which haven't had any activity (catalog/fact submission)
for more than node-ttl.
Expire nodes which haven't had any activity (catalog/fact submission) for more than `node-ttl`.
(clean-up db
lock
{:keys [node-ttl node-purge-ttl report-ttl resource-events-ttl]}
db-lock-status
request
incremental?)Inputs: [db lock :- ReentrantLock {:keys [node-ttl node-purge-ttl report-ttl resource-events-ttl]} :- {:node-ttl Period, :node-purge-ttl Period, :report-ttl Period, :resource-events-ttl Period, s/Keyword s/Any} db-lock-status :- clojure.lang.Atom request :- clean-request-schema incremental?]
Cleans up the resources specified by request, or everything if request is empty?.
Inputs: [db lock :- ReentrantLock {:keys [node-ttl node-purge-ttl report-ttl resource-events-ttl]} :- {:node-ttl Period, :node-purge-ttl Period, :report-ttl Period, :resource-events-ttl Period, s/Keyword s/Any} db-lock-status :- clojure.lang.Atom request :- clean-request-schema incremental?]
Cleans up the resources specified by request, or everything if
request is empty?.(cli args)(cli args opts)Runs the services command as directed by the command line args and returns an appropriate exit status. When the :upgrade-and-exit? opt is true, upgrades the database and returns.
Runs the services command as directed by the command line args and returns an appropriate exit status. When the :upgrade-and-exit? opt is true, upgrades the database and returns.
(collect-garbage db clean-lock config db-lock-status clean-request)(collect-garbage db clean-lock config db-lock-status clean-request incremental?)(db-config->clean-request config)Given a database config, creates a list of tuples that are the list of database clean requests to perform and their scheduled interval
Given a database config, creates a list of tuples that are the list of database clean requests to perform and their scheduled interval
(db-unsupported-msg current oldest)Returns a message describing which databases are supported.
Returns a message describing which databases are supported.
(garbage-collect! db)Perform garbage collection on db, which means deleting any orphaned data.
This basically just wraps the corresponding scf.storage function with some
logging and other ceremony. Exceptions are logged but otherwise ignored.
Perform garbage collection on `db`, which means deleting any orphaned data. This basically just wraps the corresponding scf.storage function with some logging and other ceremony. Exceptions are logged but otherwise ignored.
(gc-resource-events! db
{:keys [incremental? resource-events-ttl db-lock-status]})Delete resource-events entries which are older than than resource-events-ttl.
Delete resource-events entries which are older than than `resource-events-ttl`.
(init-with-db db-name db-config read-user)Performs all initialization operations requiring a database connection using a transient connection pool derived from the db-config. Blocks until the initialization is complete, and then returns an unspecified value. Throws exceptions on errors. Throws {:kind ::unsupported-database :current version :oldest version} if the current database is not supported. Throws {:kind ::invalid-database-configuration :failed-validation failed-map} if the database contains a disallowed setting.
Performs all initialization operations requiring a database
connection using a transient connection pool derived from the
db-config. Blocks until the initialization is complete, and
then returns an unspecified value. Throws exceptions on errors.
Throws {:kind ::unsupported-database :current version :oldest
version} if the current database is not supported. Throws
{:kind ::invalid-database-configuration :failed-validation failed-map}
if the database contains a disallowed setting.(invoke-periodic-gc db cfg request shutdown-for-ex clean-lock lock-status)(maybe-send-cmd-event! emit-cmd-events? cmd-event-ch cmdref kind)Put a map with identifying information about an enqueued command on the cmd-event-chan. Valid :kind values are: ::command/ingested ::command/processed
Put a map with identifying information about an enqueued command on the cmd-event-chan. Valid :kind values are: ::command/ingested ::command/processed
(prep-db {:keys [username migrator-username migrate min-required-version]}
read-user)(provide-services args)(provide-services args {:keys [upgrade-and-exit?]})Starts PuppetDB as a service via Trapperkeeper. Augments TK's normal config parsing a bit.
Starts PuppetDB as a service via Trapperkeeper. Augments TK's normal config parsing a bit.
Defines a trapperkeeper service for PuppetDB; this service is responsible for initializing all of the PuppetDB subsystems and registering shutdown hooks that trapperkeeper will call on exit.
Defines a trapperkeeper service for PuppetDB; this service is responsible for initializing all of the PuppetDB subsystems and registering shutdown hooks that trapperkeeper will call on exit.
(clean this)(clean this what)Performs maintenance. If specified, what requests a subset of the normal operations, and must itself be a subset of #{"expire_nodes" "purge_nodes" "purge_reports" "other"}. If what is not specified or is empty, performs all maintenance. Returns false if some kind of maintenance was already in progress, true otherwise. Although the latter does not imply that all of the operations were successful; consult the logs for more information.
Performs maintenance. If specified, what requests a subset of
the normal operations, and must itself be a subset of
#{"expire_nodes" "purge_nodes" "purge_reports" "other"}.
If what is not specified or is empty, performs all maintenance.
Returns false if some kind of maintenance was already in progress,
true otherwise. Although the latter does not imply that all of
the operations were successful; consult the logs for more
information.(cmd-event-mult this)Returns a core.async mult to which {:kind <::ingested|::processed|::queue/queue-loaded>} maps are sent when emit-cmd-events? is set to true. In the case of {:kind <::ingested|::processed>} the map will contain :entity, :certname, and :producer-ts entries for each command PDB has either ingested or finished processing. The {:kind ::queue/queue-loaded} map is used to indicate that the queue/message-loader has finished loading any existing commands from the stockpile queue when PDB starts up.
Returns a core.async mult to which
{:kind <::ingested|::processed|::queue/queue-loaded>} maps are sent
when emit-cmd-events? is set to true. In the case of
{:kind <::ingested|::processed>} the map will contain :entity, :certname,
and :producer-ts entries for each command PDB has either ingested or
finished processing. The {:kind ::queue/queue-loaded} map is used to
indicate that the queue/message-loader has finished loading any existing
commands from the stockpile queue when PDB starts up.(delete-node this certname)Immediately delete all data for the provided certname
Immediately delete all data for the provided certname
(query this version query-expr paging-options row-callback-fn)(query this
version
query-expr
paging-options
row-callback-fn
{:keys [timeout-ns]})Call row-callback-fn' for matching rows. Thepaging-options' should
be a map containing :order_by, :offset, and/or :limit.
Call `row-callback-fn' for matching rows. The `paging-options' should be a map containing :order_by, :offset, and/or :limit.
(set-url-prefix this url-prefix)(shared-globals this)(purge-nodes! node-purge-ttl opts db)Inputs: [node-purge-ttl :- (s/pred period?) opts :- (s/if map? purge-nodes-opts-schema (s/eq true)) db :- (s/pred map?)]
Deletes nodes which have been deactivated or expired longer than node-purge-ttl. Deletes at most batch_limit nodes if opts is a map, all relevant nodes otherwise.
Inputs: [node-purge-ttl :- (s/pred period?) opts :- (s/if map? purge-nodes-opts-schema (s/eq true)) db :- (s/pred map?)] Deletes nodes which have been *deactivated or expired* longer than node-purge-ttl. Deletes at most batch_limit nodes if opts is a map, all relevant nodes otherwise.
(reduce-clean-request request)Converts the incoming vector of requests to a map of requests to their options, where the last one of each kind wins. e.g. ["purge_nodes" "purge_reports" {"purge_nodes" {...}}] becomes #{{"purge_reports" true} {"purge_nodes" {...}}}.
Converts the incoming vector of requests to a map of requests to
their options, where the last one of each kind wins.
e.g. ["purge_nodes" "purge_reports" {"purge_nodes" {...}}]
becomes #{{"purge_reports" true} {"purge_nodes" {...}}}.(shutdown-requestor request-shutdown service)Returns a shim for TK's request-shutdown that also records the shutdown request in the service context as :shutdown-request.
Returns a shim for TK's request-shutdown that also records the shutdown request in the service context as :shutdown-request.
(start-garbage-collection {:keys [clean-lock] :as _context}
sched
db-configs
db-pools
db-lock-statuses
shutdown-for-ex)Starts garbage collection of the databases represented in db-configs
Starts garbage collection of the databases represented in db-configs
(start-puppetdb context
config
service
_get-registered-endpoints
request-shutdown
upgrade-and-exit?)Throws {:kind ::unsupported-database :current version :oldest version} if the current database is not supported. If a database setting is configured incorrectly, throws {:kind ::invalid-database-configuration :failed-validation failed-map}
Throws {:kind ::unsupported-database :current version :oldest version} if
the current database is not supported. If a database setting is configured
incorrectly, throws {:kind ::invalid-database-configuration :failed-validation failed-map}(start-puppetdb-or-shutdown context
config
service
get-registered-endpoints
request-shutdown)(start-schema-checks _context
service
job-pool
request-shutdown
db-configs
db-pools
shutdown-for-ex)(stop-puppetdb context request-shutdown)Shuts down PuppetDB, releasing resources when possible.
Shuts down PuppetDB, releasing resources when possible.
(stop-unless-valid-schema-version desired-version db _service request-shutdown)(sweep-reports! db
{:keys [incremental? report-ttl resource-events-ttl
db-lock-status]})Deletes reports older than than report-ttl. When resource-events-ttl is also provided, deletes events that are older than that ttl (rounded to the day) or the report-ttl, whichever is newer, allowing this function to act as a merged reports and resource-events gc. When incremental? is true, only deletes the oldest partition(s) when there are multiple candidates.
Deletes reports older than than report-ttl. When resource-events-ttl is also provided, deletes events that are older than that ttl (rounded to the day) or the report-ttl, whichever is newer, allowing this function to act as a merged reports and resource-events gc. When incremental? is true, only deletes the oldest partition(s) when there are multiple candidates.
(validate-read-only-user read-db read-user)Check that the postgres user supplied in the read-database config has the appropriate read-only permissions. Log ERROR level warnings if this user is not configured properly
Check that the postgres user supplied in the read-database config has the appropriate read-only permissions. Log ERROR level warnings if this user is not configured properly
(verify-database-setting m [setting value])Given m, a map of database settings, verify 'expected', which has the form [setting-name expected-value].
Given m, a map of database settings, verify 'expected', which has the form [setting-name expected-value].
(verify-database-settings database-settings)Ensure the database configuration does not have any settings known to break PuppetDB. If an invalid database configuration is found, throws {:kind ::invalid-database-configuration :failed-validation failed-map}
Ensure the database configuration does not have any settings known
to break PuppetDB. If an invalid database configuration is found, throws
{:kind ::invalid-database-configuration :failed-validation failed-map}(verify-database-version min-version-override)Verifies that the available database version is acceptable. Throws {:kind ::unsupported-database :current version :oldest version} if the current database is not supported.
Verifies that the available database version is acceptable. Throws
{:kind ::unsupported-database :current version :oldest version} if
the current database is not supported.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |