Schema migrations
The initialize-schema function can be used to prepare the
database, applying all the pending migrations to the database, in
ascending order of schema version. Pending is defined as having a
schema version greater than the current version in the database.
A migration is specified by defining a function of arity 0 and adding it to
the migrations map, along with its schema version. To apply the migration,
the migration function will be invoked, and the schema version and current
time will be recorded in the schema_migrations table.
A migration function can return a map with ::vacuum-analyze to indicate what tables need to be analyzed post-migration.
NOTE: in order to support bug-fix schema changes to older branches without breaking the ability to upgrade, it is possible to define a sequence of migrations with non-sequential integers. e.g., if the 1.0.x branch contains migrations 1-5, and the 2.0.x branch contains schema migrations 1-10, and then a bugfix schema change (such as creating or adding an index) is identified, this migration can be defined as #11 in both branches. Code in the 1.0.x branch should happily apply #11 even though it does not have a definition for 6-10. Then when a 1.0.x user upgrades to 2.0.x, migrations 6-10 will be applied, and 11 will be skipped because it's already been run. Because of this, it is crucial to be extremely careful about numbering new migrations if they are going into multiple branches. It's also crucial to be absolutely certain that the schema change in question is compatible with both branches and that the migrations missing from the earlier branch can reasonably and safely be applied after the bugfix migration, because that is what will happen for upgrading users.
In short, here are some guidelines re: applying schema changes to multiple branches:
TODO: consider using multimethods for migration funcs
Schema migrations
The `initialize-schema` function can be used to prepare the
database, applying all the pending migrations to the database, in
ascending order of schema version. Pending is defined as having a
schema version greater than the current version in the database.
A migration is specified by defining a function of arity 0 and adding it to
the `migrations` map, along with its schema version. To apply the migration,
the migration function will be invoked, and the schema version and current
time will be recorded in the schema_migrations table.
A migration function can return a map with ::vacuum-analyze to indicate what tables
need to be analyzed post-migration.
NOTE: in order to support bug-fix schema changes to older branches without
breaking the ability to upgrade, it is possible to define a sequence of
migrations with non-sequential integers. e.g., if the 1.0.x branch
contains migrations 1-5, and the 2.0.x branch contains schema migrations
1-10, and then a bugfix schema change (such as creating or adding an index)
is identified, this migration can be defined as #11 in both branches. Code
in the 1.0.x branch should happily apply #11 even though it does not have
a definition for 6-10. Then when a 1.0.x user upgrades to 2.0.x, migrations
6-10 will be applied, and 11 will be skipped because it's already been run.
Because of this, it is crucial to be extremely careful about numbering new
migrations if they are going into multiple branches. It's also crucial to
be absolutely certain that the schema change in question is compatible
with both branches and that the migrations missing from the earlier branch
can reasonably and safely be applied *after* the bugfix migration, because
that is what will happen for upgrading users.
In short, here are some guidelines re: applying schema changes to multiple
branches:
1. If at all possible, avoid it.
2. Seriously, are you sure you need to do this? :)
3. OK, if you really must do it, make sure that the schema change in question
is as independent as humanly possible. For example, things like creating
or dropping an index on a table should be fairly self-contained. You should
think long and hard about any change more complex than that.
4. Determine what the latest version of the schema is in each of the two branches.
5. Examine every migration that exists in the newer branch but not the older
branch, and make sure that your new schema change will not conflict with
*any* of those migrations. Your change must be able to execute successfully
regardless of whether it is applied BEFORE all of those migrations or AFTER
them.
6. If you're certain you've met the conditions described above, choose the next
available integer from the *newer* branch and add your migration to both
branches using this integer. This will result in a gap between the integers
in the migrations array in the old branch, but that is not a problem.
_TODO: consider using multimethods for migration funcs_(add-expired-to-certnames)Add a 'expired' column to the 'certnames' table, to track which nodes have been automatically expired because of inactivity.
Add a 'expired' column to the 'certnames' table, to track which nodes have been automatically expired because of inactivity.
(add-indexes-for-reports-summary-query)This aggregate and function is used by PE, not puppetdb
This aggregate and function is used by PE, not puppetdb
(add-producer-to-reports-catalogs-and-factsets)(add-report-partition-indexes-on-certname-end-time)(add-report-type-to-reports)Add a 'report_type' column to the 'reports' table, to indicate the source of the report. Valid values at the time of this migration are 'agent' or 'plan'.
Add a 'report_type' column to the 'reports' table, to indicate the source of the report. Valid values at the time of this migration are 'agent' or 'plan'.
(applied-migrations)Returns a collection of migrations that have been run against this database already, ordered from oldest to latest.
Returns a collection of migrations that have been run against this database already, ordered from oldest to latest.
(autovacuum-vacuum-scale-factor-factsets-catalogs-certnames)(call-with-connections-blocked-during-migration db-name users f)(call-with-connections-blocked-during-migration db-name [user & others] users f)(create-original-partition base-table date-column date constraint-fn index-fn)Inputs: [base-table :- s/Str date-column :- s/Str date :- (s/cond-pre LocalDate LocalDateTime ZonedDateTime Instant java.sql.Timestamp) constraint-fn :- (s/fn-schema (s/fn :- [s/Str] [_iso-year-week :- s/Str])) index-fn :- (s/fn-schema (s/fn :- [s/Str] [_full-table-name :- s/Str _iso-year-week :- s/Str]))]
Creates an inheritance partition for the historical state of migrations #73 and #74
Inputs: [base-table :- s/Str date-column :- s/Str date :- (s/cond-pre LocalDate LocalDateTime ZonedDateTime Instant java.sql.Timestamp) constraint-fn :- (s/fn-schema (s/fn :- [s/Str] [_iso-year-week :- s/Str])) index-fn :- (s/fn-schema (s/fn :- [s/Str] [_full-table-name :- s/Str _iso-year-week :- s/Str]))] Creates an inheritance partition for the historical state of migrations #73 and #74
(create-original-reports-partition date)Creates an inheritance partition in the reports table for migration #74
Creates an inheritance partition in the reports table for migration #74
(create-original-resource-events-partition date)Creates an inheritance partition in the resource_events table for migration #73
Creates an inheritance partition in the resource_events table for migration #73
(desired-schema-version)The newest migration this PuppetDB instance knows about. Anything newer is considered invalid as far as this instance is concerned.
The newest migration this PuppetDB instance knows about. Anything newer is considered invalid as far as this instance is concerned.
(drop-child-partition-constraints date-column child-table-maps)Drops the redundant constraints previously placed on the child tables that were enforcing the partitioning for the inheritance partitions.
Drops the redundant constraints previously placed on the child tables that were enforcing the partitioning for the inheritance partitions.
(drop-foreign-key-constraints child-table-maps)Drops the foreign keys previously placed on the child tables
Drops the foreign keys previously placed on the child tables
(initialize-schema)(initialize-schema write-user read-user db-name)Ensures the database is migrated to the latest version, and returns true if and only if migrations were run. Assumes the database status, version, etc. have already been validated.
Ensures the database is migrated to the latest version, and returns true if and only if migrations were run. Assumes the database status, version, etc. have already been validated.
(migrate-resource-events-to-declarative-partitioning)The available migrations, as a map from migration version to migration function.
The available migrations, as a map from migration version to migration function.
(note-migrations-finished)Currently just a hook used during testing.
Currently just a hook used during testing.
(pending-migrations)Returns a collection of pending migrations, ordered from oldest to latest.
Returns a collection of pending migrations, ordered from oldest to latest.
(reattach-partitions table child-table-maps)Detaches partition children from PostgreSQL inheritance and attaches them to the new partition table (under its placeholder name).
Detaches partition children from PostgreSQL inheritance and attaches them to the new partition table (under its placeholder name).
(record-migration! version)Records a migration by storing its version in the schema_migrations table, along with the time at which the migration was performed.
Records a migration by storing its version in the schema_migrations table, along with the time at which the migration was performed.
(remove-catalog-resources-file-trgm-index)Drops the expensive catalog_resources_file_trgm index, and resets the catalog_resources autovacuum_analyze_scale_factor storage parameter to default.
Drops the expensive catalog_resources_file_trgm index, and resets the catalog_resources autovacuum_analyze_scale_factor storage parameter to default.
(require-previously-optional-trigram-indexes)Create trgm indexes if they do not currently exist.
Create trgm indexes if they do not currently exist.
(require-valid-schema)Returns true if the database is ready for use, otherwise throws.
Returns true if the database is ready for use, otherwise throws.
(resource-events-partitioning)(resource-events-partitioning batch-size)(run-migrations migrations)Runs the requested migrations. Returns a set of tables that should be analyzed if there were any migrations.
Runs the requested migrations. Returns a set of tables that should be analyzed if there were any migrations.
(split-certnames-table)Split the certnames table. Avoid dirtying buffer cache pages for critical deactivated/expired informaton.
Split the certnames table. Avoid dirtying buffer cache pages for critical deactivated/expired informaton.
(unrecognized-migrations applied-migrations)Returns a set of migrations, likely created by a future version of PuppetDB
Returns a set of migrations, likely created by a future version of PuppetDB
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |