lazy-elasticsearch-scroll

Clojure library to use the Elasticsearch Scroll API as a lazy sequence.

Use Cases

The purpose of the library is to have an interface the consume all or some part the data from Elasticsearch. Why would you need to do that:

• One-off data transfer between Elasticsearch clusters (e.g. production -> staging);
• One-off query replay from Elasticsearch logs cluster with slow queries back to the production Elasticsearch cluster;
• If your enriched documents goes directly to the production Elasticsearch, and you want to play with the enriched data on your laptop.
• etc...

Latest Version

The library is uploaded to Clojars, so you can just:

{:deps {lazy-elasticsearch-scroll {:mvn/version "1.0.6"}}}

If you want to use code straight from Github then:

{:deps {lazy-elasticsearch-scroll {:git/url "https://github.com/dainiusjocas/lazy-elasticsearch-scroll.git"
                                   :sha "447d9656b7ca0fd655e0c3207e62e76b310f22ad"}}}

Quickstart

(require '[scroll :as scroll])

(scroll/hits
  {:es-host    "http://localhost:9200"
   :index-name ".kibana"
   :query      {:query {:match_all {}}}})
;; =>
({:_id "space:default",
  :_type "_doc",
  :_score 1.0,
  :_index ".kibana_1",
  :_source {:space {:description "This is your default space!",
                    :color "#00bfb3",
                    :name "Default",
                    :_reserved true,
                    :disabledFeatures []},
            :migrationVersion {:space "6.6.0"},
            :type "space",
            :references [],
            :updated_at "2020-02-12T14:16:18.621Z"}}
 {:_id "config:7.6.0",
  :_type "_doc",
  :_score 1.0,
  :_index ".kibana_1",
  :_source {:config {:buildNum 29000}, :type "config", :references [], :updated_at "2020-02-12T14:16:20.526Z"}})

Examples

;; Scroll through all the documents:
(scroll/hits {:es-host "http://localhost:9200"})

;; Fetch at most 10 docs:
(take 10 (scroll/hits
           {:es-host    "http://localhost:9200"
            :index-name ".kibana"
            :query      {:query {:match_all {}}}}))

;; Do not keywordize keys
(scroll/hits
  {:es-host    "http://localhost:9200"
   :opts       {:keywordize?  false}})
;; =>
({"_score" nil,
  "_type" "_doc",
  "sort" [0],
  "_source" {"space" {"disabledFeatures" [],
                      "name" "Default",
                      "_reserved" true,
                      "color" "#00bfb3",
                      "description" "This is your default space!"},
             "references" [],
             "updated_at" "2020-02-12T14:16:18.621Z",
             "type" "space",
             "migrationVersion" {"space" "6.6.0"}},
  "_id" "space:default",
  "_index" ".kibana_1"}
 {"_score" nil, "_type" "_doc", "sort" [0], "_source" {"value" 0}, "_id" "0", "_index" "scroll-test-index"})

Supported Elasticsearch versions

• 7.6.x
• 7.5.x
• 6.8.x
• 5.6.X

Development

Run the development environment make run-dev-env. This will start a docker-compose cluster with Elasticsearch and Kibana on exposed ports 9200 and 5601 respectively.

Run integration tests locally make run-integration-tests. This will start a docker-compose in which the integration tests will be run.

License

Copyright © 2020 Dainius Jocas.

Distributed under the The Apache License, Version 2.0.