sparq-yoots.core — org.clojars.nfcharles/sparq-yoots 0.3.2

Main utility functions

load-dataframe clj

(load-dataframe sql-ctx
                path
                colspecs
                &
                {:keys [fmt] :or {fmt (:default sparq.const/read)}})

Loads dataframe from specification.

load-dataframe-from-schema clj

(load-dataframe-from-schema sql-ctx path schema fmt)

Loads dataframe with defined schema.

load-dataframe-ns clj

(load-dataframe-ns sql-ctx
                   path
                   &
                   {:keys [fmt] :or {fmt (:default sparq.const/read)}})

Loads dataframe with defined schema.

load-dataframe-sources clj

(load-dataframe-sources
  sql-ctx
  path
  &
  {:keys [schema fmt paths]
   :or {schema nil fmt (:default sparq.const/read) paths nil}})

Loads dataframe with defined schema.

load-row-rdd clj

(load-row-rdd sql-ctx
              path
              &
              {:keys [fmt] :or {fmt (:default sparq.const/read)}})

Loads row rdd

optional-conf clj

(optional-conf obj action predicate)

rdd->dataframe clj

(rdd->dataframe sql-ctx rdd struct-type)

seq->dataframe clj

(seq->dataframe session rows colspecs)

variable-conf clj

(variable-conf obj a1 a2 predicate)

write clj

(write df
       output
       &
       {:keys [save-mode with-partitions]
        :or {save-mode nil with-partitions nil}})

Persists dataframe in parquet format.