zero-one.geni.core.data-sources — zero.one/geni 0.0.39

->kebab-columns clj

(->kebab-columns dataset)

Returns a new Dataset with all columns renamed to kebab cases.

create-global-temp-view! clj

(create-global-temp-view! dataframe view-name)

Creates a global temporary view using the given name.

Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

create-or-replace-global-temp-view! clj

(create-or-replace-global-temp-view! dataframe view-name)

Creates or replaces a global temporary view using the given name.

Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

create-or-replace-temp-view! clj

(create-or-replace-temp-view! dataframe view-name)

Creates or replaces a local temporary view using the given name.

The lifetime of this temporary view is tied to the SparkSession that was used to create this Dataset.

create-temp-view! clj

(create-temp-view! dataframe view-name)

Creates a local temporary view using the given name.

Local temporary view is session-scoped. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

default-options clj

Default DataFrameReader options.

read-avro! clj multimethod

Loads an Avro file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-binary! clj multimethod

Loads a binary file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources-binaryFile.html

read-csv! clj multimethod

Loads a CSV file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-edn! clj multimethod

Loads an EDN file and returns the results as a DataFrame.

read-jdbc! clj

(read-jdbc! options)

(read-jdbc! spark options)

Loads a database table and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-json! clj multimethod

Loads a JSON file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-libsvm! clj multimethod

Loads a LIBSVM file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-parquet! clj multimethod

Loads a Parquet file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources-parquet.html

read-table! clj multimethod

Reads a managed (hive) table and returns the result as a DataFrame.

read-text! clj multimethod

Loads a text file and returns the results as a DataFrame.

Spark's DataFrameReader options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

read-xlsx! clj multimethod

Loads an Excel file and returns the results as a DataFrame.

Example options:

{:header true :sheet "Sheet2"}

write-avro! clj

(write-avro! dataframe path)

(write-avro! dataframe path options)

Writes an Avro file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

write-csv! clj

(write-csv! dataframe path)

(write-csv! dataframe path options)

Writes a CSV file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

write-edn! clj

(write-edn! dataframe path)

(write-edn! dataframe path options)

Writes an EDN file at the specified path.

write-jdbc! clj

(write-jdbc! dataframe options)

Writes a database table.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

write-json! clj

(write-json! dataframe path)

(write-json! dataframe path options)

Writes a JSON file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources-json.html

write-libsvm! clj

(write-libsvm! dataframe path)

(write-libsvm! dataframe path options)

Writes a LIBSVM file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

write-parquet! clj

(write-parquet! dataframe path)

(write-parquet! dataframe path options)

Writes a Parquet file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources-parquet.html

write-table! clj

(write-table! dataframe table-name)

(write-table! dataframe table-name options)

Writes the dataset to a managed (hive) table.

write-text! clj

(write-text! dataframe path)

(write-text! dataframe path options)

Writes a text file at the specified path.

Spark's DataFrameWriter options may be passed in as a map of options.

See: https://spark.apache.org/docs/latest/sql-data-sources.html

write-xlsx! clj

(write-xlsx! dataframe path)

(write-xlsx! dataframe path options)

Writes an Excel file at the specified path.