Catalog persistence
Catalogs are persisted in a relational database. Roughly speaking, the schema looks like this:
resource_parameters are associated 0 to N catalog_resources (they are deduped across catalogs). It's possible for a resource_param to exist in the database, yet not be associated with a catalog. This is done as a performance optimization.
edges are associated with a single catalog
catalogs are associated with a single certname
facts are associated with a single certname
The standard set of operations on information in the database will
likely result in dangling resources and catalogs; to clean these
up, it's important to run garbage-collect!.
Catalog persistence Catalogs are persisted in a relational database. Roughly speaking, the schema looks like this: * resource_parameters are associated 0 to N catalog_resources (they are deduped across catalogs). It's possible for a resource_param to exist in the database, yet not be associated with a catalog. This is done as a performance optimization. * edges are associated with a single catalog * catalogs are associated with a single certname * facts are associated with a single certname The standard set of operations on information in the database will likely result in dangling resources and catalogs; to clean these up, it's important to run `garbage-collect!`.
(acquire-locks! table-modes)Acquires a lock on the specified table.
Acquires a lock on the specified table.
(add-catalog-metadata! hash {:keys [certname] :as catalog} received-timestamp)Stores catalog metadata for certname.
Stores catalog metadata for certname.
(add-certname! certname)Inputs: [certname :- String]
Add the given host to the db
Inputs: [certname :- String] Add the given host to the db
(add-certnames certnames)Inserts the collection of certnames.
Inserts the collection of certnames.
(add-facts! {:keys [certname values environment timestamp producer_timestamp
producer package_inventory]
:as fact-data})Inputs: [{:keys [certname values environment timestamp producer_timestamp producer package_inventory], :as fact-data} :- facts-schema]
Given a certname and a map of fact names to values, store records for those facts associated with the certname.
Inputs: [{:keys [certname values environment timestamp producer_timestamp producer package_inventory], :as fact-data} :- facts-schema]
Given a certname and a map of fact names to values, store records for those
facts associated with the certname.(add-new-catalog certname-id hash catalog refs-to-hashes received-timestamp)Inputs: [certname-id :- Long hash :- String catalog :- catalog-schema refs-to-hashes :- {resource-ref-schema String} received-timestamp :- pls/Timestamp]
Creates new catalog metadata and adds the proper associations for the edges and resources
Inputs: [certname-id :- Long hash :- String catalog :- catalog-schema refs-to-hashes :- {resource-ref-schema String} received-timestamp :- pls/Timestamp]
Creates new catalog metadata and adds the proper associations for the edges and resources(add-params! refs-to-resources refs-to-hashes)Inputs: [refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- resource-ref->hash]
Persists the new parameters found in refs-to-resources and populates the
resource_params_cache.
Inputs: [refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- resource-ref->hash] Persists the new parameters found in `refs-to-resources` and populates the resource_params_cache.
(add-report! report received-timestamp db conn-status)(add-report! {:keys [certname producer_timestamp] :as report}
received-timestamp
db
conn-status
options-config)Inputs: ([report received-timestamp db conn-status] [{:keys [certname producer_timestamp], :as report} :- reports/report-wireformat-schema received-timestamp :- pls/Timestamp db conn-status options-config]) Returns: processing-status-schema
Add a report and all of the associated events to the database.
Inputs: ([report received-timestamp db conn-status] [{:keys [certname producer_timestamp], :as report} :- reports/report-wireformat-schema received-timestamp :- pls/Timestamp db conn-status options-config])
Returns: processing-status-schema
Add a report and all of the associated events to the database.(add-report!* orig-report received-timestamp save-event? options-config)Inputs: [orig-report :- reports/report-wireformat-schema received-timestamp :- pls/WireTimestamp save-event? :- s/Bool options-config] Returns: processing-status-schema
Helper function for adding a report. Accepts an extra parameter, update-latest-report?, which
is used to determine whether or not the update-latest-report! function will be called as part of
the transaction. This should always be set to true, except during some very specific testing
scenarios.
Inputs: [orig-report :- reports/report-wireformat-schema received-timestamp :- pls/WireTimestamp save-event? :- s/Bool options-config] Returns: processing-status-schema Helper function for adding a report. Accepts an extra parameter, `update-latest-report?`, which is used to determine whether or not the `update-latest-report!` function will be called as part of the transaction. This should always be set to `true`, except during some very specific testing scenarios.
(add-resources! certname-id certname refs-to-resources refs-to-hashes)Inputs: [certname-id :- Long certname :- String refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- {resource-ref-schema String}]
Persist the given resource and associate it with the given catalog.
Inputs: [certname-id :- Long certname :- String refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- {resource-ref-schema String}]
Persist the given resource and associate it with the given catalog.(basic-diff left right)Inputs: [left right]
Basic diffing that returns only the keys/values of right whose values don't match those of left.
This is different from clojure.data/diff in that it treats non-equal sets as completely different
(rather than returning only the differing items of the set) and only returns differences from right.
Inputs: [left right] Basic diffing that returns only the keys/values of `right` whose values don't match those of `left`. This is different from clojure.data/diff in that it treats non-equal sets as completely different (rather than returning only the differing items of the set) and only returns differences from `right`.
(catalog-edges-map certname)Inputs: [certname :- String]
Return all edges for a given catalog id as a map
Inputs: [certname :- String] Return all edges for a given catalog id as a map
(catalog-hash-for-certname certname)Returns the hash for the certname catalog
Returns the hash for the `certname` catalog
(catalog-resources certname-id)Inputs: [certname-id :- Long]
Returns the resource hashes keyed by resource reference
Inputs: [certname-id :- Long] Returns the resource hashes keyed by resource reference
(catalog-row-map hash
{:keys [version code_id job_id transaction_uuid catalog_uuid
environment producer_timestamp producer]}
received-timestamp)Inputs: [hash {:keys [version code_id job_id transaction_uuid catalog_uuid environment producer_timestamp producer]} :- catalog-schema received-timestamp :- pls/Timestamp]
Creates a row map for the catalogs table, optionally adding envrionment when it was found
Inputs: [hash {:keys [version code_id job_id transaction_uuid catalog_uuid environment producer_timestamp producer]} :- catalog-schema received-timestamp :- pls/Timestamp]
Creates a row map for the catalogs table, optionally adding envrionment when it was foundThis is a bit of a hack to make a more restrictive schema in the storage layer. Moving the more restrictive resource/edge schemas into puppetdb.catalogs is TODO. Upstream code needs to assume a map of resources (not a vector) and tests need to be update to adhere to the new format.
This is a bit of a hack to make a more restrictive schema in the storage layer. Moving the more restrictive resource/edge schemas into puppetdb.catalogs is TODO. Upstream code needs to assume a map of resources (not a vector) and tests need to be update to adhere to the new format.
(certname-factset-metadata certname)Inputs: [certname :- s/Str] Returns: {:package_hash (s/maybe s/Str), :factset_id s/Int, :certname_id s/Int, :stable_hash (s/maybe (s/pred shash/sha1-bytes?))}
Given a certname, return the factset id, hash and certname id.
Inputs: [certname :- s/Str]
Returns: {:package_hash (s/maybe s/Str), :factset_id s/Int, :certname_id s/Int, :stable_hash (s/maybe (s/pred shash/sha1-bytes?))}
Given a certname, return the factset id, hash and certname id.(clean-catalog-data db)Delete unassociated resource parameters and environments.
Delete unassociated resource parameters and environments.
(deactivate-node! certname)(deactivate-node! certname timestamp)Inputs: ([certname :- String] [certname :- String timestamp :- pls/Timestamp])
Deactivate the given host, recording the current time. If the node is currently inactive, no change is made.
Inputs: ([certname :- String] [certname :- String timestamp :- pls/Timestamp]) Deactivate the given host, recording the current time. If the node is currently inactive, no change is made.
(delete-catalog-resources! certname-id certname old-resources refs-to-delete)Inputs: [certname-id :- Long certname :- String old-resources refs-to-delete]
Inputs: [certname-id :- Long certname :- String old-resources refs-to-delete]
(delete-certname! certname)Delete the given host from the db
Delete the given host from the db
(delete-certnames! certnames)Delete the list of hosts from the db
Delete the list of hosts from the db
(delete-edges! certname edges)Inputs: [certname :- String edges :- edge-db-schema]
Delete edges for a given certname.
Edges must be either nil or a collection of lists containing each element of an edge, e.g.:
[[<source> <target> <type>] ...]
Inputs: [certname :- String edges :- edge-db-schema] Delete edges for a given certname. Edges must be either nil or a collection of lists containing each element of an edge, e.g.: [[<source> <target> <type>] ...]
(delete-reports-older-than!
{:keys [report-ttl resource-events-ttl incremental? update-lock-status]
:or {incremental? false update-lock-status (constantly true)}})Delete all reports in the database which have a producer-timestamp that is prior to the specified report-time. When event-time is specified, delete all the events that are older than whichever time is more recent.
Delete all reports in the database which have a producer-timestamp that is prior to the specified report-time. When event-time is specified, delete all the events that are older than whichever time is more recent.
(delete-reports-older-than-in-pg-11! report-ttl
resource-events-ttl
incremental?
update-lock-status)PostgreSQL 11 workflow for deleting all reports in the database which have a producer-timestamp that is prior to the specified report-time. When event-time is specified, delete all the events that are older than whichever time is more recent.
PostgreSQL 11 workflow for deleting all reports in the database which have a producer-timestamp that is prior to the specified report-time. When event-time is specified, delete all the events that are older than whichever time is more recent.
(delete-resource-events-older-than! date incremental? update-lock-status)Delete all resource events in the database by dropping any partition older than the day of the year of the given date. Note: this ignores the time in the given timestamp, rounding to the day.
Delete all resource events in the database by dropping any partition older than the day of the year of the given date. Note: this ignores the time in the given timestamp, rounding to the day.
(delete-unassociated-environments!)Remove any environments that aren't associated with a catalog, report or factset
Remove any environments that aren't associated with a catalog, report or factset
(delete-unassociated-params!)Remove any resources that aren't associated with a catalog
Remove any resources that aren't associated with a catalog
(delete-unused-fact-paths)Deletes paths from fact_paths that are no longer needed by any factset. In the unusual case where a path changes type, the previous version will linger. This requires a parent transaction with at least a repeatable-read isolation level (and may or may not need postgres' "stronger than the standard" repeatable-read behavior). Otherwise paths could be added to fact_paths elsewhere during the gc, not be noticed, and then be deleted at the end.
Deletes paths from fact_paths that are no longer needed by any factset. In the unusual case where a path changes type, the previous version will linger. This requires a parent transaction with at least a repeatable-read isolation level (and may or may not need postgres' "stronger than the standard" repeatable-read behavior). Otherwise paths could be added to fact_paths elsewhere during the gc, not be noticed, and then be deleted at the end.
(detach-daily-partitions parent-table
date
incremental?
update-lock-status
status-key)Detaches declarative partitions concurrently in PG14+, which runs with a less restrictive SHARE UPDATE EXCLUSIVE lock instead of ACCESS EXCLUSIVE. Once it's detached, it should no longer participate in queries to the parent table and the subsequent drop should be able to acquire an ACCESS EXCLUSIVE lock more quickly.
NOTE: This must be run outside of a transaction.
Detaches declarative partitions concurrently in PG14+, which runs with a less restrictive SHARE UPDATE EXCLUSIVE lock instead of ACCESS EXCLUSIVE. Once it's detached, it should no longer participate in queries to the parent table and the subsequent drop should be able to acquire an ACCESS EXCLUSIVE lock more quickly. NOTE: This must be run outside of a transaction.
(detach-partitions-concurrently?)True if Postgresql server supports ALTER TABLE DETACH PARTITION CONCURRENTLY. (PG 14+)
True if Postgresql server supports ALTER TABLE DETACH PARTITION CONCURRENTLY. (PG 14+)
(diff-resources-metadata left right certname)Inputs: [left right certname]
Return resource references with values that are only the key/values that from right that
are different from those of the left. The keys/values here are suitable for issuing update
statements that will update resources to the correct (new) values.
Inputs: [left right certname] Return resource references with values that are only the key/values that from `right` that are different from those of the `left`. The keys/values here are suitable for issuing update statements that will update resources to the correct (new) values.
(drop-partition-tables! old-partition-tables update-lock-status status-key)Drops the given set of tables. Will throw an SQLException termination if the operation takes much longer than PDB_GC_DAILY_PARTITION_DROP_LOCK_TIMEOUT_MS.
Drops the given set of tables. Will throw an SQLException termination if the operation takes much longer than PDB_GC_DAILY_PARTITION_DROP_LOCK_TIMEOUT_MS.
(ensure-certname certname)Inputs: [certname :- String]
Adds the given host to the db iff it isn't already there.
Inputs: [certname :- String] Adds the given host to the db iff it isn't already there.
(ensure-named-row table col name)Returns the id (primary key) of the named row, creating it if necessary.
Returns the id (primary key) of the named row, creating it if necessary.
(expire-stale-nodes horizon)Inputs: [horizon :- Period]
Expires nodes with no activity within the provided horizon (prior to now) and returns a collection of the affected certnames.
Inputs: [horizon :- Period] Expires nodes with no activity within the provided horizon (prior to now) and returns a collection of the affected certnames.
(finalize-pending-detach parent)Finalize a previously failed detach operation. A partitioned table can only have one partition pending detachment at any time.
Finalize a previously failed detach operation. A partitioned table can only have one partition pending detachment at any time.
(find-containing-class containment-path)Given a containment path from Puppet, find the outermost 'class'.
Given a containment path from Puppet, find the outermost 'class'.
(find-stranded-partitions child-format)Identify tables that match the child format of a partitioned table (like reports_historical) that are not present in the pg_inherits table. These partitions have been detached, but failed to be deleted.
Tables that are not partitioned will also not be in the pg_inherits table, so you MUST write a child-format that does not match any non-partitioned tables.
Returns a list of strings. Each string is a stranded partition that should be removed.
Identify tables that match the child format of a partitioned table (like reports_historical) that are not present in the pg_inherits table. These partitions have been detached, but failed to be deleted. Tables that are not partitioned will also not be in the pg_inherits table, so you MUST write a child-format that does not match any non-partitioned tables. Returns a list of strings. Each string is a stranded partition that should be removed.
(garbage-collect! db)Delete any lingering, unassociated data in the database
Delete any lingering, unassociated data in the database
(handle-resource-insert-sql-ex ex certname row-or-rows)Handles a SQLException encountered while inserting a resource. This may occur when the inserted resource has a value too big for a postgres btree index.
Handles a SQLException encountered while inserting a resource. This may occur when the inserted resource has a value too big for a postgres btree index.
(have-newer-record-for-certname? certname timestamp)Inputs: [certname :- String timestamp :- Timestamp]
Returns a truthy value indicating whether a record exists that has a producer_timestamp newer than the given timestamp.
Inputs: [certname :- String timestamp :- Timestamp] Returns a truthy value indicating whether a record exists that has a producer_timestamp newer than the given timestamp.
(insert-catalog-resources! certname-id
certname
refs-to-hashes
refs-to-resources
refs-to-insert)Inputs: [certname-id :- Long certname :- String refs-to-hashes :- {resource-ref-schema String} refs-to-resources :- resource-ref->resource-schema refs-to-insert]
Inputs: [certname-id :- Long certname :- String refs-to-hashes :- {resource-ref-schema String} refs-to-resources :- resource-ref->resource-schema refs-to-insert]
(insert-edges! certname edges)Inputs: [certname :- String edges :- edge-db-schema]
Insert edges for a given certname.
Edges must be either nil or a collection of lists containing each element of an edge, eg:
[[<source> <target> <type>] ...]
Inputs: [certname :- String edges :- edge-db-schema] Insert edges for a given certname. Edges must be either nil or a collection of lists containing each element of an edge, eg: [[<source> <target> <type>] ...]
(insert-missing-packages existing-hashes-map new-hashed-package-tuples)(insert-resource-events certname cert-id report-id resource-events ttl latest)(latest-catalog-metadata certname)Returns the id, hash, certname_id and producer_timestamp of certname's catalog.
Returns the id, hash, certname_id and producer_timestamp of certname's catalog.
(latest-report-metadata certname)Return the latest report timestamp for the given certname
Return the latest report timestamp for the given certname
(maybe-activate-node! certname time)Inputs: [certname :- String time :- pls/Timestamp]
Reactivate the given host, only if it was deactivated or expired before
time. Returns true if the node is activated, or if it was already active.
Adds the host to the database if it was not already present.
Inputs: [certname :- String time :- pls/Timestamp] Reactivate the given host, only if it was deactivated or expired before `time`. Returns true if the node is activated, or if it was already active. Adds the host to the database if it was not already present.
(maybe-log-query-termination result)Takes a seq of maps containing information about PostgreSQL pids which the query-bulldozer attempted to call pg_terminate_backend(<pid>) on and logs information about the termination attempt. Example input: [{:pg_terminate_backend t/f :pid <pid>}]
Takes a seq of maps containing information about PostgreSQL pids
which the query-bulldozer attempted to call pg_terminate_backend(<pid>)
on and logs information about the termination attempt.
Example input: [{:pg_terminate_backend t/f :pid <pid>}](merge-resource-hash refs-to-hashes refs-to-resources)Assoc each hash from refs-to-hashes as :resource on refs-to-resources
Assoc each hash from `refs-to-hashes` as :resource on `refs-to-resources`
(named-row-id table col name)Returns the id (primary key) from the table for the row with col = name.
Returns the id (primary key) from the table for the row with col = name.
(new-params-only persisted-params refs-to-resources refs-to-hashes)Inputs: [persisted-params :- #{String} refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- resource-ref->hash]
Returns a map of not persisted parameters, keyed by hash
Inputs: [persisted-params :- #{String} refs-to-resources :- resource-ref->resource-schema refs-to-hashes :- resource-ref->hash]
Returns a map of not persisted parameters, keyed by hash(node-deactivated-time certname)Returns the time the node specified by certname was deactivated, or nil if
the node is currently active.
Returns the time the node specified by `certname` was deactivated, or nil if the node is currently active.
(node-expired-time certname)Returns the time the node specified by certname expired, or nil if
the node is currently active.
Returns the time the node specified by `certname` expired, or nil if the node is currently active.
(normalize-report {:keys [resources] :as report})Prep the report for comparison/computation of a hash
Prep the report for comparison/computation of a hash
(normalize-resource-event event)Prep event for comparison/computation of a hash
Prep `event` for comparison/computation of a hash
(prune-daily-partitions parent-table
date
incremental?
update-lock-status
status-key)(prune-daily-partitions parent-table
date
incremental?
update-lock-status
status-key
just-detach?)Either detaches or drops obsolete day-oriented partitions older than the date. Deletes or detaches only the oldest such candidate if incremental? is true. Will throw an SQLException termination if the operation takes much longer than PDB_GC_DAILY_PARTITION_DROP_LOCK_TIMEOUT_MS.
Either detaches or drops obsolete day-oriented partitions older than the date. Deletes or detaches only the oldest such candidate if incremental? is true. Will throw an SQLException termination if the operation takes much longer than PDB_GC_DAILY_PARTITION_DROP_LOCK_TIMEOUT_MS.
(purge-deactivated-and-expired-nodes! horizon)(purge-deactivated-and-expired-nodes! horizon batch-limit)Delete nodes from the database which were deactivated before horizon.
Delete nodes from the database which were deactivated before horizon.
(query-bulldozer db gc-pid connected gc-finished?)Creates a thread which will loop and wait for the gc-pid to get blocked waiting on an AccessExclusiveLock. Once this thread detects that GC is blocked it will terminate all running queries from the pdb user against the pdb database which have been granted locks with the exception of queries from the gc-pid or the bulldozer's pid. This should clear the way for GC to grab the lock it's requesting in the main GC thread. This loop repeats until the bulldozer thread is interrupted or receives confirmation from the GC thread via the gc-finished? atom that the GC thread has dropped the partition.
Creates a thread which will loop and wait for the gc-pid to get blocked waiting on an AccessExclusiveLock. Once this thread detects that GC is blocked it will terminate all running queries from the pdb user against the pdb database which have been granted locks with the exception of queries from the gc-pid or the bulldozer's pid. This should clear the way for GC to grab the lock it's requesting in the main GC thread. This loop repeats until the bulldozer thread is interrupted or receives confirmation from the GC thread via the gc-finished? atom that the GC thread has dropped the partition.
(realize-paths pathmaps)(realize-paths pathmaps notice-pathmap)Ensures that every path in the pathmaps has a corresponding row in fact_paths.
Ensures that every path in the pathmaps has a corresponding row in fact_paths.
(replace-catalog! catalog)(replace-catalog! {:keys [producer_timestamp resources certname] :as catalog}
received-timestamp)Inputs: ([catalog :- catalog-schema] [{:keys [producer_timestamp resources certname], :as catalog} :- catalog-schema received-timestamp :- pls/Timestamp]) Returns: processing-status-schema
Persist the supplied catalog in the database, returning its similarity hash.
Inputs: ([catalog :- catalog-schema] [{:keys [producer_timestamp resources certname], :as catalog} :- catalog-schema received-timestamp :- pls/Timestamp])
Returns: processing-status-schema
Persist the supplied catalog in the database, returning its
similarity hash.(replace-catalog-inputs! certname catalog_uuid inputs updated)Inputs: [certname :- s/Str catalog_uuid :- s/Str inputs :- [[s/Str]] updated :- pls/Timestamp] Returns: processing-status-schema
Inputs: [certname :- s/Str catalog_uuid :- s/Str inputs :- [[s/Str]] updated :- pls/Timestamp] Returns: processing-status-schema
(replace-edges! certname edges refs-to-hashes)Inputs: [certname :- String edges :- #{edge-schema} refs-to-hashes :- {resource-ref-schema String}]
Persist the given edges in the database
Each edge is looked up in the supplied resources map to find a resource object that corresponds to the edge. We then use that resource's hash for persistence purposes.
For example, if the source of an edge is {'type' 'Foo' 'title' 'bar'}, then we'll lookup a resource with that key and use its hash.
Inputs: [certname :- String edges :- #{edge-schema} refs-to-hashes :- {resource-ref-schema String}]
Persist the given edges in the database
Each edge is looked up in the supplied resources map to find a
resource object that corresponds to the edge. We then use that
resource's hash for persistence purposes.
For example, if the source of an edge is {'type' 'Foo' 'title' 'bar'},
then we'll lookup a resource with that key and use its hash.(replace-existing-catalog certname-id
catalog-id
hash
catalog
refs-to-hashes
received-timestamp)Inputs: [certname-id :- Long catalog-id :- Long hash :- String catalog :- catalog-schema refs-to-hashes :- {resource-ref-schema String} received-timestamp :- pls/Timestamp]
New catalogs for a given certname needs to have their metadata, resources and edges updated.
Inputs: [certname-id :- Long catalog-id :- Long hash :- String catalog :- catalog-schema refs-to-hashes :- {resource-ref-schema String} received-timestamp :- pls/Timestamp]
New catalogs for a given certname needs to have their metadata, resources and
edges updated.(replace-facts! {:keys [certname producer_timestamp] :as fact-data})Inputs: [{:keys [certname producer_timestamp], :as fact-data} :- facts-schema] Returns: processing-status-schema
Updates the facts of an existing node, if the facts are newer than the current set of facts. Adds all new facts if no existing facts are found. Invoking this function under the umbrella of a repeatable read or serializable transaction enforces only one update to the facts of a certname can happen at a time. The first to start the transaction wins. Subsequent transactions will fail as the factsets will have changed while the transaction was in-flight.
Inputs: [{:keys [certname producer_timestamp], :as fact-data} :- facts-schema]
Returns: processing-status-schema
Updates the facts of an existing node, if the facts are newer than the current set of facts.
Adds all new facts if no existing facts are found. Invoking this function under the umbrella of
a repeatable read or serializable transaction enforces only one update to the facts of a certname
can happen at a time. The first to start the transaction wins. Subsequent transactions will fail
as the factsets will have changed while the transaction was in-flight.Returns true of the map is a resource reference
Returns true of the map is a resource reference
(resources-exist? resource-hashes)Inputs: [resource-hashes :- #{String}] Returns: #{String}
Given a collection of resource-hashes, return the subset that already exist in the database.
Inputs: [resource-hashes :- #{String}]
Returns: #{String}
Given a collection of resource-hashes, return the subset that
already exist in the database.(set-certname-facts-expiration certname expire? updated)Inputs: [certname :- s/Str expire? :- s/Bool updated :- pls/Timestamp] Returns: processing-status-schema
Inputs: [certname :- s/Str expire? :- s/Bool updated :- pls/Timestamp] Returns: processing-status-schema
(strip-params resource)Remove params from the resource as it is stored (and hashed) separately from the resource metadata
Remove params from the resource as it is stored (and hashed) separately from the resource metadata
This provides a partial ordering of tables, to ensure that we always lock them in a consistent order.
This provides a partial ordering of tables, to ensure that we always lock them in a consistent order.
(update-catalog-associations! certname-id
{:keys [resources edges certname]}
refs-to-hashes)Inputs: [certname-id :- Long {:keys [resources edges certname]} :- catalog-schema refs-to-hashes :- {resource-ref-schema String}]
Adds/updates/deletes the edges and resources for the given certname
Inputs: [certname-id :- Long {:keys [resources edges certname]} :- catalog-schema refs-to-hashes :- {resource-ref-schema String}]
Adds/updates/deletes the edges and resources for the given certname(update-catalog-input-metadata! certid catalog-uuid last-updated inputs-hash)(update-catalog-metadata! id hash catalog received-timestamp)Inputs: [id :- Long hash :- String catalog :- catalog-schema received-timestamp :- pls/Timestamp]
Given some catalog metadata, update the db
Inputs: [id :- Long hash :- String catalog :- catalog-schema received-timestamp :- pls/Timestamp] Given some catalog metadata, update the db
(update-catalog-resources! certname-id
certname
refs-to-hashes
refs-to-resources
old-resources
maybe-updated-refs)Inputs: [certname-id :- Long certname :- String refs-to-hashes :- {resource-ref-schema String} refs-to-resources old-resources maybe-updated-refs]
Inputs: [certname-id :- Long certname :- String refs-to-hashes :- {resource-ref-schema String} refs-to-resources old-resources maybe-updated-refs]
(update-existing-catalog catalog-id hash catalog received-timestamp)Inputs: [catalog-id :- Long hash :- String catalog :- catalog-schema received-timestamp :- pls/Timestamp]
When a new incoming catalog has the same hash as an existing catalog, update storage-metrics and the transaction id for the new catalog
Inputs: [catalog-id :- Long hash :- String catalog :- catalog-schema received-timestamp :- pls/Timestamp] When a new incoming catalog has the same hash as an existing catalog, update storage-metrics and the transaction id for the new catalog
(update-facts! {:keys [certname values environment timestamp producer_timestamp
producer package_inventory]
:as fact-data})Inputs: [{:keys [certname values environment timestamp producer_timestamp producer package_inventory], :as fact-data} :- facts-schema]
Given a certname, querys the DB for existing facts for that certname and will update, delete or insert the facts as necessary to match the facts argument. (cf. add-facts!)
Inputs: [{:keys [certname values environment timestamp producer_timestamp producer package_inventory], :as fact-data} :- facts-schema]
Given a certname, querys the DB for existing facts for that
certname and will update, delete or insert the facts as necessary
to match the facts argument. (cf. add-facts!)(update-packages certname_id package_hash inventory)Inputs: [certname_id :- s/Int package_hash :- (s/maybe s/Str) inventory :- [pkg-util/package-tuple]]
Compares inventory to the stored package inventory for
certname. Differences will result in updates to the database
Inputs: [certname_id :- s/Int package_hash :- (s/maybe s/Str) inventory :- [pkg-util/package-tuple]] Compares `inventory` to the stored package inventory for `certname`. Differences will result in updates to the database
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |