benchgecko.core — org.clojars.dropthe/benchgecko 0.1.0

Clojure SDK for BenchGecko — compare LLM benchmarks, estimate inference costs, and explore AI model performance data across providers.

Models are represented as maps with the following keys: :name - Model identifier (e.g., "gpt-4o") :provider - Provider name (e.g., "OpenAI") :context-window - Maximum context in tokens (optional) :scores - Map of benchmark category keywords to scores (0-100) :pricing - Map with :input-per-mtok and :output-per-mtok in USD (optional)

Benchmark categories: :reasoning :coding :knowledge :instruction :multilingual :safety :long-context :vision :agentic

average-score clj

(average-score model)

Calculate the average benchmark score for a model. Returns nil if the model has no scores.

benchmark-categories clj

All benchmark categories tracked by BenchGecko.

best-value clj

(best-value models)

Find the model with the best value score (performance per dollar). Returns nil if no models have both pricing and scores.

compare-models clj

(compare-models model-a model-b)

Compare two models across all mutually-scored benchmark categories. Returns a map with: :model-a - First model :model-b - Second model :deltas - Map of category to score difference (positive = A leads) :a-wins - Categories where model A scores higher :b-wins - Categories where model B scores higher :ties - Categories with identical scores :winner - The model with higher average score (ties favor A)

(compare-models gpt4 claude)

estimate-cost clj

(estimate-cost model input-tokens output-tokens)

Estimate inference cost in USD for a given number of input and output tokens. Returns nil if the model has no pricing information.

(estimate-cost gpt4 5000 2000) ;=> 0.0325

estimate-monthly clj

(estimate-monthly model daily-requests avg-input-tokens avg-output-tokens)

Estimate monthly cost assuming a daily request volume.

(estimate-monthly gpt4 1000 3000 1000) ;=> monthly USD cost

filter-by-tier clj

(filter-by-tier models tier)

Filter models by performance tier.

(filter-by-tier models :S) ;=> sequence of S-tier models

get-model clj

(get-model models name)

Look up a model by name from a collection of models. Returns the first model matching the given name, or nil.

make-model clj

(make-model name provider & {:keys [context-window scores pricing]})

Create a model map with required name and provider, plus optional fields.

(make-model "gpt-4o" "OpenAI" :context-window 128000 :scores {:reasoning 92.3 :coding 89.1} :pricing {:input-per-mtok 2.50 :output-per-mtok 10.00})

model-summary clj

(model-summary model)

Generate a human-readable summary string for a model.

(model-summary gpt4) ;=> "gpt-4o (OpenAI) [S-Tier] avg=90.0 value=7.2"

model-tier clj

(model-tier model)

Classify a model into a performance tier (:S :A :B :C :D). Returns nil if the model has no scores.

Tier	Average Score	Description
:S	90+	Elite frontier
:A	80-89	Strong general
:B	70-79	Capable mid-range
:C	60-69	Budget / older gen
:D	<60	Entry-level / legacy

rank-by-category clj

(rank-by-category models category)

Rank models by score in a specific benchmark category (descending). Models without a score in that category are excluded.

(rank-by-category models :coding) ;=> [{:model gpt4 :score 89.1} {:model claude :score 93.7}]

value-score clj

(value-score model)

Compute value score: average benchmark performance per dollar of blended token price. Higher is better. Returns nil if pricing or scores are missing.