mesomatic.allocator — twosigma/mesomatic 1.0.1-2018.08.28-df2c2ca

Resource Allocation

accept-offer clj

(accept-offer offer task pos)

Associate a task with an offer's corresponding slave. When allocating multiple instances of a task (for daemons) and the task-id field is a vector get the appropriate member of the vector

accept-ports clj

(accept-ports resources ports)

Eat up as many ports as specified in the first port range. Notice that only the first port range is considered for now.

adjust-offer clj

(adjust-offer offer task)

If an offer has been accepted, decrease its available resources.

adjust-ports clj

(adjust-ports ranges want)

Adjust port range. Notice that only the first range is considered.

adjustor clj

(adjustor cpus mem ports)

Yield a function which will adjust a resource record when appropriate.

allocate-naively clj

(allocate-naively offers tasks)

Cycle through all tasks, sorted by biggest to smallest in terms of resource needs and try to satisfy requirements.

allocate-task-naively clj

(allocate-task-naively acc {:keys [instances] :or {instances 1} :as task})

Allocate a task's worth of necessary resources. Tasks may ask for a specific count of instances and a maximum collocation factor to avoid behind hosted on a single slave.

Allocation technique

Let's assume the following workload and available offers:

[ T1(1,1,1,1) T2(4,8,2,1) T3(1,1,4,2) ] [ O1(4,8) O2(8,16) O3(2,2) ]

We first sort our offers:

[ T2(4,8,2,1) T3(1,1,4,2) T1(1,1,1,1) ] [ O1(8,16) O2(4,8) O3(2,2) ]

We can now step through tasks and allocate appropriately:

[ T3(1,1,4,2) T1(1,1,1,1) ] [ O1(4,8) O3(2,2) ] [ T1:(O1,O2) ]

[ T1(1,1,1,1) ] [ O1(2,6) ] [ T2:(O1,O2) T3(O1,O3) ]

[ ] [ O1(1,5) ] [ T2:(O1,O2) T3(O1,O3) T1(O1) ]

get-ports clj

(get-ports rlist)

get-ranges clj

(get-ranges rlist name)

get-scalar clj

(get-scalar rlist name)

IAllocator clj protocol

Allocator abstraction to allow for pluggable allocation techniques.

allocate clj

(allocate this offers tasks)

Given a list of offers and tasks to start, figure out the best possible arrangement. Will yield a list of tasks containing an offerid to select or a nil if no way to fully fulfill requirements was found. Allocation should not yield partial results.

Resources taken into account are:

• cpus
• mem
• port ranges

tasks is a list of mesomatic.types/TaskInfo offers is a list of mesomatic.types/Offer

map-ports clj

(map-ports port-mappings ports)

naive-allocator clj

(naive-allocator)

Conform to IAllocator by handing off decisions to allocate-naively

offer-matches? clj

(offer-matches? offer task)

Predicate to validate an offer can satisfy a task's requirements

resource-cmp clj

(resource-cmp r1 r2)

Comparator for resources. Will sort in decreasing order.

resource-factor clj

(resource-factor {:keys [resources]})

Compute an integral resource factor to help sort offers