Liking cljdoc? Tell your friends :D
Clojure only.

datasplash.es

read-esclj

(read-es hosts index type p)
(read-es hosts index type options p)

Read from elasticsearch.

See https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.html

Examples:

(es/read-es ["http://127.0.0.1:9200"] "my-index" "my-type" {:batch-size 100 :keep-alive "5m"} pcoll)

Available options:

  • :batch-size => Specify the scroll size (number of document by page). Default to 100. Maximum is 10 000. If documents are small, increasing batch size might improve read performance. If documents are big, you might need to decrease batch-size
  • :checkpoint => Given a path, will store the resulting pcoll at this path in edn to facilitate dev/debug.
  • :coder => Uses a specific Coder for the results of this transform. Usually defaults to some form of nippy-coder.
  • :key-fn => Can be either true (to coerce keys to keywords),false to leave them as strings, or a function to provide custom coercion.
  • :keystore-password => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  • :keystore-path => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  • :name => Adds a name to the Transform.
  • :password => password
  • :query => Provide a query used while reading from Elasticsearch.
  • :retry-configuration => Creates RetryConfiguration for ElasticsearchIO with provided max-attempts, max-durations and exponential backoff based retries
  • :scroll-keep-alive => Provide a scroll keepalive. See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-request-scroll.html . Default is "5m". Change this only if you get "No search context found" errors.
  • :username => username
Read from elasticsearch.

See https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.html

Examples:
```
(es/read-es ["http://127.0.0.1:9200"] "my-index" "my-type" {:batch-size 100 :keep-alive "5m"} pcoll)
```

Available options:

  - :batch-size => Specify the scroll size (number of document by page). Default to 100. Maximum is 10 000. If documents are small, increasing batch size might improve read performance. If documents are big, you might need to decrease batch-size
  - :checkpoint => Given a path, will store the resulting pcoll at this path in edn to facilitate dev/debug.
  - :coder => Uses a specific Coder for the results of this transform. Usually defaults to some form of nippy-coder.
  - :key-fn => Can be either true (to coerce keys to keywords),false to leave them as strings, or a function to provide custom coercion.
  - :keystore-password => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  - :keystore-path => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  - :name => Adds a name to the Transform.
  - :password => password
  - :query => Provide a query used while reading from Elasticsearch.
  - :retry-configuration => Creates RetryConfiguration for ElasticsearchIO with provided max-attempts, max-durations and exponential backoff based retries
  - :scroll-keep-alive => Provide a scroll keepalive. See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-request-scroll.html . Default is "5m". Change this only if you get "No search context found" errors.
  - :username => username
sourceraw docstring

write-esclj

(write-es hosts index type pcoll)
(write-es hosts index type options pcoll)

Write to elasticsearch.

See https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.html

Examples:

(es/write-es ["http://127.0.0.1:9200"] "my-index" "my-type" {:id-fn (fn [x] (get x "id")) :max-batch-size-bytes 50000000 :name write-es})

Available options:

  • :checkpoint => Given a path, will store the resulting pcoll at this path in edn to facilitate dev/debug.
  • :coder => Uses a specific Coder for the results of this transform. Usually defaults to some form of nippy-coder.
  • :id-fn =>
  • :index-fn =>
  • :keystore-password => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  • :keystore-path => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  • :max-batch-size => Specify the max number of documents in a bulk. Default to 1000
  • :max-batch-size-bytes => Specify the max number of bytes in a bulk. Default to 5MB
  • :name => Adds a name to the Transform.
  • :password => password
  • :retry-configuration => Creates RetryConfiguration for ElasticsearchIO with provided max-attempts, max-durations and exponential backoff based retries
  • :type-fn => Provide a function to extract the target type from the document allowing for dynamic document routing.
  • :use-partial-update => Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.
  • :username => username
Write to elasticsearch.

See https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.html

Examples:
```
(es/write-es ["http://127.0.0.1:9200"] "my-index" "my-type" {:id-fn (fn [x] (get x "id")) :max-batch-size-bytes 50000000 :name write-es})
```

Available options:

  - :checkpoint => Given a path, will store the resulting pcoll at this path in edn to facilitate dev/debug.
  - :coder => Uses a specific Coder for the results of this transform. Usually defaults to some form of nippy-coder.
  - :id-fn => 
  - :index-fn => 
  - :keystore-password => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  - :keystore-path => If Elasticsearch uses SSL/TLS with mutual authentication (via shield), provide the password to open the client keystore.
  - :max-batch-size => Specify the max number of documents in a bulk. Default to 1000
  - :max-batch-size-bytes => Specify the max number of bytes in a bulk. Default to 5MB
  - :name => Adds a name to the Transform.
  - :password => password
  - :retry-configuration => Creates RetryConfiguration for ElasticsearchIO with provided max-attempts, max-durations and exponential backoff based retries
  - :type-fn => Provide a function to extract the target type from the document allowing for dynamic document routing.
  - :use-partial-update => Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.
  - :username => username
sourceraw docstring

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close