dynamo-utils.core — dynamo-utils 0.3.1

estimate clj

(estimate {:keys [item-count item-kbs process-time concurrency-level batch-size
                  delay-per-item delay-per-batch current-partition-count]})

estimate-partitions clj

(estimate-partitions total-item-kbs read-capacity write-capacity)

Estimates the number of partitions dynamodb will allocate for a given dataset + capacity.

Interesting behaviors about AWS partitioning rules:

• The minimum number of partitions would be max of
• by at least 10 GB chuncks
  • ceiling(data-size / 10 GB)
• by ( readCapacityUnits / 3,000 ) + ( writeCapacityUnits / 1,000 ) = partitions (rounded up)
  • Capacity can only be allocated by doubling the initial num of partitions. So the smallest base-2 that covers the minimum number of partitions will be used. (Technically, this only applies to reallocation of capacity), but this is an estimation.

NOTE: partitions never decrease once allocated. If a previous capacity allocation increased partitions to 32, then it was always be at least 32 partitions.

See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.Partitions

estimate-single-consumption clj

(estimate-single-consumption item-count total-item-kbs)

lazy-scan clj

(lazy-scan dynamo-config table batch-size batch-delay-ms)

(lazy-scan dynamo-config
           table
           batch-size
           batch-delay-ms
           {:keys [last-prim-kvs] :as scan-options})

lazy-scan-parallel clj

(lazy-scan-parallel dynamo-config
                    table
                    batch-size
                    batch-delay-ms
                    total-segments
                    scan-options)

sanitize clj

TODO: remove me when we update faraday after 1.7.1