metabase.sync.util — metabase-core 1.0.0-SNAPSHOT

Utility functions and macros to abstract away some common patterns and operations across the sync processes, such as logging start/end messages.

calculate-duration-str clj

(calculate-duration-str begin-time end-time)

Inputs: [begin-time :- (s/protocol tcoerce/ICoerce) end-time :- (s/protocol tcoerce/ICoerce)] Returns: s/Str

Given two datetimes, caculate the time between them, return the result as a string

calculate-hash clj

(calculate-hash clj-data)

Calculate a cryptographic hash on clj-data and return that hash as a string

create-sync-step clj

(create-sync-step step-name sync-fn)

(create-sync-step step-name sync-fn log-summary-fn)

Creates and returns a step suitable for run-step-with-metadata. See StepDefinition for more info.

db->sync-tables clj

(db->sync-tables database-or-id)

Return all the Tables that should go through the sync processes for DATABASE-OR-ID.

do-sync-operation clj

(do-sync-operation operation database message f)

Internal implementation of sync-operation; use that instead of calling this directly.

do-with-error-handling clj

(do-with-error-handling f)

(do-with-error-handling message f)

Internal implementation of with-error-handling; use that instead of calling this directly.

emoji-progress-bar clj

(emoji-progress-bar completed total log-every-n)

Create a string that shows progress for something, e.g. a database sync process.

(emoji-progress-bar 10 40) -> "[************······································] 😒 25%

LogSummaryFunction clj

A log summary function takes a StepRunMetadata and returns a string with a step-specific log message

name-for-logging clj

(name-for-logging this)

Return an appropriate string for logging an object in sync logging messages. Should be something like "postgres Database 'test-data'"

run-step-with-metadata clj

(run-step-with-metadata database
                        {:keys [step-name sync-fn log-summary-fn] :as step})

Inputs: [database :- i/DatabaseInstance {:keys [step-name sync-fn log-summary-fn], :as step} :- StepDefinition] Returns: StepNameWithMetadata

Runs step on `database returning metadata from the run

run-sync-operation clj

(run-sync-operation operation database sync-steps)

Inputs: [operation :- s/Str database :- i/DatabaseInstance sync-steps :- [StepDefinition]]

Run sync-steps and log a summary message

StepDefinition clj

Defines a step. :sync-fn runs the step, returns a map that contains step specific metadata. log-summary-fn takes that metadata and turns it into a string for logging

StepNameWithMetadata clj

Pair with the step name and metadata about the completed step run

StepRunMetadata clj

Map with metadata about the step. Contains both generic information like start-time and end-time and step specific information

StepSpecificMetadata clj

A step function can return any metadata and is used by the related LogSummaryFunction to provide step-specific details about run

sum-for clj macro

(sum-for [item-binding coll & more-for-bindings] & body)

Basically the same as for, but sums the results of each iteration of body that returned a number. See also sum-numbers.

As an added bonus, unlike normal for, this wraps body in an implicit do, so you can have more than one form inside the loop. Nice

sum-for* clj

(sum-for* results)

Impl for sum-for macro; see its docstring;

sum-numbers clj

(sum-numbers f coll)

Similar to a 2-arg call to map, but will add all numbers that result from the invocations of f. Used mainly for logging purposes, such as to count and log the number of Fields updated by a sync operation. See also sum-for, a for-style macro version.

sync-operation clj macro

(sync-operation operation database message & body)

Perform the operations in BODY as a sync operation, which wraps the code in several special macros that do things like error handling, logging, duplicate operation prevention, and event publishing. Intended for use with the various top-level sync operations, such as sync-metadata or analyze.

SyncOperationMetadata clj

Timing and step information for the entire sync or analyze run

with-duplicate-ops-prevented clj

(with-duplicate-ops-prevented operation database-or-id f)

Run F in a way that will prevent it from simultaneously being ran more for a single database more than once for a given OPERATION. This prevents duplicate sync-like operations from taking place for a given DB, e.g. if a user hits the Sync button in the admin panel multiple times.

;; Only one sync-db! for database-id will be allowed at any given moment; duplicates will be ignored (with-duplicate-ops-prevented :sync database-id #(sync-db! database-id))

with-emoji-progress-bar clj macro

(with-emoji-progress-bar [emoji-progress-fn-binding total-count] & body)

Run BODY with access to a function that makes using our amazing emoji-progress-bar easy like Sunday morning. Calling the function will return the approprate string output for logging and automatically increment an internal counter as needed.

(with-emoji-progress-bar [progress-bar 10] (dotimes [i 10] (println (progress-bar))))

with-error-handling clj macro

(with-error-handling message & body)

Execute BODY in a way that catches and logs any Exceptions thrown, and returns nil if they do so. Pass a MESSAGE to help provide information about what failed for the log message.

with-start-and-finish-debug-logging clj

(with-start-and-finish-debug-logging message f)

Similar to with-start-and-finish-logging except invokesf` and returns its result and logs at the debug level