(! expr)
Params: (e: Column) Result: Column Inversion of boolean expression, i.e. NOT. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.985Z
Params: (e: Column) Result: Column Inversion of boolean expression, i.e. NOT. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.985Z
(** base exponent)
Params: (l: Column, r: Column) Result: Column Returns the value of the first argument raised to the power of the second argument.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.009Z
Params: (l: Column, r: Column) Result: Column Returns the value of the first argument raised to the power of the second argument. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.009Z
(->date-col expr)
(->date-col expr date-format)
Params: (e: Column) Result: Column Converts the column into DateType by casting rules to DateType.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.115Z
Params: (e: Column) Result: Column Converts the column into DateType by casting rules to DateType. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.115Z
(->timestamp-col expr)
(->timestamp-col expr date-format)
Params: (s: Column) Result: Column Converts to a timestamp by casting rules to TimestampType.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.123Z
Params: (s: Column) Result: Column Converts to a timestamp by casting rules to TimestampType. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.123Z
(->utc-timestamp expr)
Params: (ts: Column, tz: String) Result: Column Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.125Z
Params: (ts: Column, tz: String) Result: Column Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.125Z
(abs expr)
Params: (e: Column) Result: Column Computes the absolute value of a numeric value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.577Z
Params: (e: Column) Result: Column Computes the absolute value of a numeric value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.577Z
(acos expr)
Params: (e: Column) Result: Column inverse cosine of e in radians, as if computed by java.lang.Math.acos Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.580Z
Params: (e: Column) Result: Column inverse cosine of e in radians, as if computed by java.lang.Math.acos Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.580Z
(add-months expr months)
Params: (startDate: Column, numMonths: Int) Result: Column Returns the date that is numMonths after startDate.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.583Z
Params: (startDate: Column, numMonths: Int) Result: Column Returns the date that is numMonths after startDate. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.583Z
(aggregate expr init merge-fn)
(aggregate expr init merge-fn finish-fn)
Params: (expr: Column, initialValue: Column, merge: (Column, Column) ⇒ Column, finish: (Column) ⇒ Column) Result: Column Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.587Z
Params: (expr: Column, initialValue: Column, merge: (Column, Column) ⇒ Column, finish: (Column) ⇒ Column) Result: Column Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state. The final state is converted into the final result by applying a finish function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.587Z
(approx-count-distinct expr)
(approx-count-distinct expr rsd)
Params: (e: Column) Result: Column (Since version 2.1.0) Use approx_count_distinct Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.231Z
Params: (e: Column) Result: Column (Since version 2.1.0) Use approx_count_distinct Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.231Z
(array & exprs)
Params: (cols: Column*) Result: Column Creates a new array column. The input columns must all have the same data type.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.594Z
Params: (cols: Column*) Result: Column Creates a new array column. The input columns must all have the same data type. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.594Z
(array-contains expr value)
Params: (column: Column, value: Any) Result: Column Returns null if the array is null, true if the array contains value, and false otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.595Z
Params: (column: Column, value: Any) Result: Column Returns null if the array is null, true if the array contains value, and false otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.595Z
(array-distinct expr)
Params: (e: Column) Result: Column Removes duplicate values from the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.596Z
Params: (e: Column) Result: Column Removes duplicate values from the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.596Z
(array-except left right)
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the first array but not in the second array, without duplicates. The order of elements in the result is not determined
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.597Z
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the first array but not in the second array, without duplicates. The order of elements in the result is not determined Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.597Z
(array-intersect left right)
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the intersection of the given two arrays, without duplicates.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.598Z
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the intersection of the given two arrays, without duplicates. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.598Z
(array-join expr delimiter)
(array-join expr delimiter null-replacement)
Params: (column: Column, delimiter: String, nullReplacement: String) Result: Column Concatenates the elements of column using the delimiter. Null values are replaced with nullReplacement. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.601Z
Params: (column: Column, delimiter: String, nullReplacement: String) Result: Column Concatenates the elements of column using the delimiter. Null values are replaced with nullReplacement. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.601Z
(array-max expr)
Params: (e: Column) Result: Column Returns the maximum value in the array.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.602Z
Params: (e: Column) Result: Column Returns the maximum value in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.602Z
(array-min expr)
Params: (e: Column) Result: Column Returns the minimum value in the array.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.603Z
Params: (e: Column) Result: Column Returns the minimum value in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.603Z
(array-position expr value)
Params: (column: Column, value: Any) Result: Column Locates the position of the first occurrence of the value in the given array as long. Returns null if either of the arguments are null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.604Z
Params: (column: Column, value: Any) Result: Column Locates the position of the first occurrence of the value in the given array as long. Returns null if either of the arguments are null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.604Z
(array-remove expr element)
Params: (column: Column, element: Any) Result: Column Remove all elements that equal to element from the given array.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.606Z
Params: (column: Column, element: Any) Result: Column Remove all elements that equal to element from the given array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.606Z
(array-repeat left right)
Params: (left: Column, right: Column) Result: Column Creates an array containing the left argument repeated the number of times given by the right argument.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.608Z
Params: (left: Column, right: Column) Result: Column Creates an array containing the left argument repeated the number of times given by the right argument. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.608Z
(array-sort expr)
Params: (e: Column) Result: Column Sorts the input array in ascending order. The elements of the input array must be orderable. Null elements will be placed at the end of the returned array.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.609Z
Params: (e: Column) Result: Column Sorts the input array in ascending order. The elements of the input array must be orderable. Null elements will be placed at the end of the returned array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.609Z
(array-union left right)
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the union of the given two arrays, without duplicates.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.610Z
Params: (col1: Column, col2: Column) Result: Column Returns an array of the elements in the union of the given two arrays, without duplicates. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.610Z
(arrays-overlap left right)
Params: (a1: Column, a2: Column) Result: Column Returns true if a1 and a2 have at least one non-null element in common. If not and both the arrays are non-empty and any of them contains a null, it returns null. It returns false otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.616Z
Params: (a1: Column, a2: Column) Result: Column Returns true if a1 and a2 have at least one non-null element in common. If not and both the arrays are non-empty and any of them contains a null, it returns null. It returns false otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.616Z
(arrays-zip & exprs)
Params: (e: Column*) Result: Column Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.617Z
Params: (e: Column*) Result: Column Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.617Z
(ascii expr)
Params: (e: Column) Result: Column Computes the numeric value of the first character of the string column, and returns the result as an int column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.623Z
Params: (e: Column) Result: Column Computes the numeric value of the first character of the string column, and returns the result as an int column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.623Z
(asin expr)
Params: (e: Column) Result: Column inverse sine of e in radians, as if computed by java.lang.Math.asin Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.626Z
Params: (e: Column) Result: Column inverse sine of e in radians, as if computed by java.lang.Math.asin Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.626Z
(atan expr)
Params: (e: Column) Result: Column inverse tangent of e, as if computed by java.lang.Math.atan Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.628Z
Params: (e: Column) Result: Column inverse tangent of e, as if computed by java.lang.Math.atan Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.628Z
(atan-2 expr-x expr-y)
Params: (y: Column, x: Column) Result: Column coordinate on y-axis Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.642Z
Params: (y: Column, x: Column) Result: Column coordinate on y-axis Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.642Z
(atan2 expr-x expr-y)
Params: (y: Column, x: Column) Result: Column coordinate on y-axis Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.642Z
Params: (y: Column, x: Column) Result: Column coordinate on y-axis Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.642Z
(base-64 expr)
Params: (e: Column) Result: Column Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.646Z
Params: (e: Column) Result: Column Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.646Z
(base64 expr)
Params: (e: Column) Result: Column Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.646Z
Params: (e: Column) Result: Column Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.646Z
(bin expr)
Params: (e: Column) Result: Column An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.648Z
Params: (e: Column) Result: Column An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100". Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.648Z
(bitwise-not expr)
Params: (e: Column) Result: Column Computes bitwise NOT (~) of a number.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.649Z
Params: (e: Column) Result: Column Computes bitwise NOT (~) of a number. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.649Z
(broadcast dataframe)
Params: (df: Dataset[T]) Result: Dataset[T] Marks a DataFrame as small enough for use in broadcast joins. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.650Z
Params: (df: Dataset[T]) Result: Dataset[T] Marks a DataFrame as small enough for use in broadcast joins. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.650Z
(bround expr)
Params: (e: Column) Result: Column Returns the value of the column e rounded to 0 decimal places with HALF_EVEN round mode.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.653Z
Params: (e: Column) Result: Column Returns the value of the column e rounded to 0 decimal places with HALF_EVEN round mode. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.653Z
(cbrt expr)
Params: (e: Column) Result: Column Computes the cube-root of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.663Z
Params: (e: Column) Result: Column Computes the cube-root of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.663Z
(ceil expr)
Params: (e: Column) Result: Column Computes the ceiling of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.665Z
Params: (e: Column) Result: Column Computes the ceiling of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.665Z
(collect-list expr)
Params: (e: Column) Result: Column Aggregate function: returns a list of objects with duplicates.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.680Z
Params: (e: Column) Result: Column Aggregate function: returns a list of objects with duplicates. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.680Z
(collect-set expr)
Params: (e: Column) Result: Column Aggregate function: returns a set of objects with duplicate elements eliminated.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.682Z
Params: (e: Column) Result: Column Aggregate function: returns a set of objects with duplicate elements eliminated. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.682Z
(concat & exprs)
Params: (exprs: Column*) Result: Column Concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.686Z
Params: (exprs: Column*) Result: Column Concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.686Z
(concat-ws sep & exprs)
Params: (sep: String, exprs: Column*) Result: Column Concatenates multiple input string columns together into a single string column, using the given separator.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.687Z
Params: (sep: String, exprs: Column*) Result: Column Concatenates multiple input string columns together into a single string column, using the given separator. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.687Z
(conv expr from-base to-base)
Params: (num: Column, fromBase: Int, toBase: Int) Result: Column Convert a number in a string column from one base to another.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.688Z
Params: (num: Column, fromBase: Int, toBase: Int) Result: Column Convert a number in a string column from one base to another. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.688Z
(cos expr)
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.695Z
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.695Z
(cosh expr)
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.699Z
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.699Z
(count-distinct & exprs)
Params: (expr: Column, exprs: Column*) Result: Column Aggregate function: returns the number of distinct items in a group.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.706Z
Params: (expr: Column, exprs: Column*) Result: Column Aggregate function: returns the number of distinct items in a group. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.706Z
(covar l-expr r-expr)
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the sample covariance for two columns.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.714Z
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the sample covariance for two columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.714Z
(covar-pop l-expr r-expr)
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the population covariance for two columns.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.710Z
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the population covariance for two columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.710Z
(covar-samp l-expr r-expr)
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the sample covariance for two columns.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.714Z
Params: (column1: Column, column2: Column) Result: Column Aggregate function: returns the sample covariance for two columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.714Z
(crc-32 expr)
Params: (e: Column) Result: Column Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.717Z
Params: (e: Column) Result: Column Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.717Z
(crc32 expr)
Params: (e: Column) Result: Column Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.717Z
Params: (e: Column) Result: Column Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.717Z
(cube-root expr)
Params: (e: Column) Result: Column Computes the cube-root of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.663Z
Params: (e: Column) Result: Column Computes the cube-root of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.663Z
(cume-dist)
Params: () Result: Column Window function: returns the cumulative distribution of values within a window partition, i.e. the fraction of rows that are below the current row. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.719Z
Params: () Result: Column Window function: returns the cumulative distribution of values within a window partition, i.e. the fraction of rows that are below the current row. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.719Z
(current-date)
Params: () Result: Column Returns the current date as a date column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.720Z
Params: () Result: Column Returns the current date as a date column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.720Z
(current-timestamp)
Params: () Result: Column Returns the current timestamp as a timestamp column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.722Z
Params: () Result: Column Returns the current timestamp as a timestamp column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.722Z
(date-add expr days)
Params: (start: Column, days: Int) Result: Column Returns the date that is days days after start
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.735Z
Params: (start: Column, days: Int) Result: Column Returns the date that is days days after start Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.735Z
(date-diff l-expr r-expr)
Params: (end: Column, start: Column) Result: Column Returns the number of days from start to end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.747Z
Params: (end: Column, start: Column) Result: Column Returns the number of days from start to end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.747Z
(date-format expr date-fmt)
Params: (dateExpr: Column, format: String) Result: Column Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.739Z
Params: (dateExpr: Column, format: String) Result: Column Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.739Z
(date-sub expr days)
Params: (start: Column, days: Int) Result: Column Returns the date that is days days before start
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.742Z
Params: (start: Column, days: Int) Result: Column Returns the date that is days days before start Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.742Z
(date-trunc fmt expr)
Params: (format: String, timestamp: Column) Result: Column Returns timestamp truncated to the unit specified by the format. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.744Z
Params: (format: String, timestamp: Column) Result: Column Returns timestamp truncated to the unit specified by the format. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.744Z
(datediff l-expr r-expr)
Params: (end: Column, start: Column) Result: Column Returns the number of days from start to end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.747Z
Params: (end: Column, start: Column) Result: Column Returns the number of days from start to end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.747Z
(day-of-month expr)
Params: (e: Column) Result: Column Extracts the day of the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.749Z
Params: (e: Column) Result: Column Extracts the day of the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.749Z
(day-of-week expr)
Params: (e: Column) Result: Column Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.751Z
Params: (e: Column) Result: Column Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.751Z
(day-of-year expr)
Params: (e: Column) Result: Column Extracts the day of the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.752Z
Params: (e: Column) Result: Column Extracts the day of the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.752Z
(dayofmonth expr)
Params: (e: Column) Result: Column Extracts the day of the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.749Z
Params: (e: Column) Result: Column Extracts the day of the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.749Z
(dayofweek expr)
Params: (e: Column) Result: Column Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.751Z
Params: (e: Column) Result: Column Extracts the day of the week as an integer from a given date/timestamp/string. Ranges from 1 for a Sunday through to 7 for a Saturday Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.751Z
(dayofyear expr)
Params: (e: Column) Result: Column Extracts the day of the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.752Z
Params: (e: Column) Result: Column Extracts the day of the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.752Z
(decode expr charset)
Params: (value: Column, charset: String) Result: Column Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.756Z
Params: (value: Column, charset: String) Result: Column Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.756Z
(degrees expr)
Params: (e: Column) Result: Column Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.759Z
Params: (e: Column) Result: Column Converts an angle measured in radians to an approximately equivalent angle measured in degrees. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.759Z
(dense-rank)
Params: () Result: Column Window function: returns the rank of rows within a window partition, without any gaps. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.760Z
Params: () Result: Column Window function: returns the rank of rows within a window partition, without any gaps. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.760Z
(element-at expr value)
Params: (column: Column, value: Any) Result: Column Returns element of array at given index in value if column is array. Returns value for the given key in value if column is map.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.765Z
Params: (column: Column, value: Any) Result: Column Returns element of array at given index in value if column is array. Returns value for the given key in value if column is map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.765Z
(encode expr charset)
Params: (value: Column, charset: String) Result: Column Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.767Z
Params: (value: Column, charset: String) Result: Column Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.767Z
(exists expr predicate)
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns whether a predicate holds for one or more elements in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.770Z
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns whether a predicate holds for one or more elements in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.770Z
(exp expr)
Params: (e: Column) Result: Column Computes the exponential of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.773Z
Params: (e: Column) Result: Column Computes the exponential of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.773Z
(explode expr)
Params: (e: Column) Result: Column Creates a new row for each element in the given array or map column. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.774Z
Params: (e: Column) Result: Column Creates a new row for each element in the given array or map column. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.774Z
(explode-outer expr)
Params: (e: Column) Result: Column Creates a new row for each element in the given array or map column. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.774Z
Params: (e: Column) Result: Column Creates a new row for each element in the given array or map column. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.774Z
(expm-1 expr)
Params: (e: Column) Result: Column Computes the exponential of the given value minus one.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.777Z
Params: (e: Column) Result: Column Computes the exponential of the given value minus one. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.777Z
(expm1 expr)
Params: (e: Column) Result: Column Computes the exponential of the given value minus one.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.777Z
Params: (e: Column) Result: Column Computes the exponential of the given value minus one. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.777Z
(expr s)
Params: (expr: String) Result: Column Parses the expression string into the column that it represents, similar to Dataset#selectExpr. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.779Z
Params: (expr: String) Result: Column Parses the expression string into the column that it represents, similar to Dataset#selectExpr. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.779Z
(factorial expr)
Params: (e: Column) Result: Column Computes the factorial of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.780Z
Params: (e: Column) Result: Column Computes the factorial of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.780Z
(flatten expr)
Params: (e: Column) Result: Column Creates a single array from an array of arrays. If a structure of nested arrays is deeper than two levels, only one level of nesting is removed. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.796Z
Params: (e: Column) Result: Column Creates a single array from an array of arrays. If a structure of nested arrays is deeper than two levels, only one level of nesting is removed. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.796Z
(floor expr)
Params: (e: Column) Result: Column Computes the floor of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.798Z
Params: (e: Column) Result: Column Computes the floor of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.798Z
(forall expr predicate)
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns whether a predicate holds for every element in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.800Z
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns whether a predicate holds for every element in the array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.800Z
(format-number expr decimal-places)
Params: (x: Column, d: Int) Result: Column Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.802Z
Params: (x: Column, d: Int) Result: Column Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.802Z
(format-string fmt & exprs)
Params: (format: String, arguments: Column*) Result: Column Formats the arguments in printf-style and returns the result as a string column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.803Z
Params: (format: String, arguments: Column*) Result: Column Formats the arguments in printf-style and returns the result as a string column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.803Z
(from-csv expr schema)
(from-csv expr schema options)
Params: (e: Column, schema: StructType, options: Map[String, String]) Result: Column Parses a column containing a CSV string into a StructType with the specified schema. Returns null, in the case of an unparseable string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.807Z
Params: (e: Column, schema: StructType, options: Map[String, String]) Result: Column Parses a column containing a CSV string into a StructType with the specified schema. Returns null, in the case of an unparseable string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.807Z
(from-json expr schema)
(from-json expr schema options)
Params: (e: Column, schema: StructType, options: Map[String, String]) Result: Column (Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema. Returns null, in the case of an unparseable string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.826Z
Params: (e: Column, schema: StructType, options: Map[String, String]) Result: Column (Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema. Returns null, in the case of an unparseable string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.826Z
(from-unixtime expr)
(from-unixtime expr fmt)
Params: (ut: Column) Result: Column Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.830Z
Params: (ut: Column) Result: Column Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.830Z
(greatest & exprs)
Params: (exprs: Column*) Result: Column Returns the greatest value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.839Z
Params: (exprs: Column*) Result: Column Returns the greatest value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.839Z
(grouping expr)
Params: (e: Column) Result: Column Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.845Z
Params: (e: Column) Result: Column Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.845Z
(grouping-id & exprs)
Params: (cols: Column*) Result: Column Aggregate function: returns the level of grouping, equals to Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.848Z
Params: (cols: Column*) Result: Column Aggregate function: returns the level of grouping, equals to Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.848Z
(hash & exprs)
Params: (cols: Column*) Result: Column Calculates the hash code of given columns, and returns the result as an int column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.849Z
Params: (cols: Column*) Result: Column Calculates the hash code of given columns, and returns the result as an int column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.849Z
(hex expr)
Params: (column: Column) Result: Column Computes hex value of the given column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.851Z
Params: (column: Column) Result: Column Computes hex value of the given column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.851Z
(hour expr)
Params: (e: Column) Result: Column Extracts the hours as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.852Z
Params: (e: Column) Result: Column Extracts the hours as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.852Z
(hypot left-expr right-expr)
Params: (l: Column, r: Column) Result: Column Computes sqrt(a2 + b2) without intermediate overflow or underflow.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.865Z
Params: (l: Column, r: Column) Result: Column Computes sqrt(a2 + b2) without intermediate overflow or underflow. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.865Z
(initcap expr)
Params: (e: Column) Result: Column Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.866Z
Params: (e: Column) Result: Column Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.866Z
(input-file-name)
Params: () Result: Column Creates a string column for the file name of the current Spark task.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.867Z
Params: () Result: Column Creates a string column for the file name of the current Spark task. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.867Z
(instr expr substr)
Params: (str: Column, substring: String) Result: Column Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.869Z
Params: (str: Column, substring: String) Result: Column Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.869Z
(kurtosis expr)
Params: (e: Column) Result: Column Aggregate function: returns the kurtosis of the values in a group.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.894Z
Params: (e: Column) Result: Column Aggregate function: returns the kurtosis of the values in a group. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.894Z
(lag expr offset)
(lag expr offset default)
Params: (e: Column, offset: Int) Result: Column Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.900Z
Params: (e: Column, offset: Int) Result: Column Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.900Z
(last-day expr)
Params: (e: Column) Result: Column Returns the last day of the month which the given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.918Z
Params: (e: Column) Result: Column Returns the last day of the month which the given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.918Z
(lead expr offset)
(lead expr offset default)
Params: (columnName: String, offset: Int) Result: Column Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.924Z
Params: (columnName: String, offset: Int) Result: Column Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.924Z
(least & exprs)
Params: (exprs: Column*) Result: Column Returns the least value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.927Z
Params: (exprs: Column*) Result: Column Returns the least value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.927Z
(length expr)
Params: (e: Column) Result: Column Computes the character length of a given string or number of bytes of a binary string. The length of character strings include the trailing spaces. The length of binary strings includes binary zeros.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.928Z
Params: (e: Column) Result: Column Computes the character length of a given string or number of bytes of a binary string. The length of character strings include the trailing spaces. The length of binary strings includes binary zeros. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.928Z
(levenshtein left-expr right-expr)
Params: (l: Column, r: Column) Result: Column Computes the Levenshtein distance of the two given string columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.929Z
Params: (l: Column, r: Column) Result: Column Computes the Levenshtein distance of the two given string columns. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.929Z
(locate substr expr)
Params: (substr: String, str: Column) Result: Column Locate the position of the first occurrence of substr.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.933Z
Params: (substr: String, str: Column) Result: Column Locate the position of the first occurrence of substr. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.933Z
(log expr)
Params: (e: Column) Result: Column Computes the natural logarithm of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.937Z
Params: (e: Column) Result: Column Computes the natural logarithm of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.937Z
(log-10 expr)
Params: (e: Column) Result: Column Computes the logarithm of the given value in base 10.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.939Z
Params: (e: Column) Result: Column Computes the logarithm of the given value in base 10. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.939Z
(log-1p expr)
Params: (e: Column) Result: Column Computes the natural logarithm of the given value plus one.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.941Z
Params: (e: Column) Result: Column Computes the natural logarithm of the given value plus one. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.941Z
(log-2 expr)
Params: (expr: Column) Result: Column Computes the logarithm of the given column in base 2.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.943Z
Params: (expr: Column) Result: Column Computes the logarithm of the given column in base 2. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.943Z
(log10 expr)
Params: (e: Column) Result: Column Computes the logarithm of the given value in base 10.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.939Z
Params: (e: Column) Result: Column Computes the logarithm of the given value in base 10. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.939Z
(log1p expr)
Params: (e: Column) Result: Column Computes the natural logarithm of the given value plus one.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.941Z
Params: (e: Column) Result: Column Computes the natural logarithm of the given value plus one. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.941Z
(log2 expr)
Params: (expr: Column) Result: Column Computes the logarithm of the given column in base 2.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.943Z
Params: (expr: Column) Result: Column Computes the logarithm of the given column in base 2. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.943Z
(lower expr)
Params: (e: Column) Result: Column Converts a string column to lower case.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.944Z
Params: (e: Column) Result: Column Converts a string column to lower case. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.944Z
(lpad expr length pad)
Params: (str: Column, len: Int, pad: String) Result: Column Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.946Z
Params: (str: Column, len: Int, pad: String) Result: Column Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.946Z
(ltrim expr)
Params: (e: Column) Result: Column Trim the spaces from left end for the specified string value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.948Z
Params: (e: Column) Result: Column Trim the spaces from left end for the specified string value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.948Z
(map & exprs)
Params: (cols: Column*) Result: Column Creates a new map column. The input columns must be grouped as key-value pairs, e.g. (key1, value1, key2, value2, ...). The key columns must all have the same data type, and can't be null. The value columns must all have the same data type.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.949Z
Params: (cols: Column*) Result: Column Creates a new map column. The input columns must be grouped as key-value pairs, e.g. (key1, value1, key2, value2, ...). The key columns must all have the same data type, and can't be null. The value columns must all have the same data type. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.949Z
(map-concat & exprs)
Params: (cols: Column*) Result: Column Returns the union of all the given maps. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.950Z
Params: (cols: Column*) Result: Column Returns the union of all the given maps. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.950Z
(map-entries expr)
Params: (e: Column) Result: Column Returns an unordered array of all entries in the given map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.951Z
Params: (e: Column) Result: Column Returns an unordered array of all entries in the given map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.951Z
(map-filter expr predicate)
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Returns a map whose key-value pairs satisfy a predicate. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.953Z
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Returns a map whose key-value pairs satisfy a predicate. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.953Z
(map-from-arrays key-expr val-expr)
Params: (keys: Column, values: Column) Result: Column Creates a new map column. The array in the first column is used for keys. The array in the second column is used for values. All elements in the array for key should not be null.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.958Z
Params: (keys: Column, values: Column) Result: Column Creates a new map column. The array in the first column is used for keys. The array in the second column is used for values. All elements in the array for key should not be null. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.958Z
(map-from-entries expr)
Params: (e: Column) Result: Column Returns a map created from the given array of entries. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.959Z
Params: (e: Column) Result: Column Returns a map created from the given array of entries. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.959Z
(map-keys expr)
Params: (e: Column) Result: Column Returns an unordered array containing the keys of the map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.960Z
Params: (e: Column) Result: Column Returns an unordered array containing the keys of the map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.960Z
(map-values expr)
Params: (e: Column) Result: Column Returns an unordered array containing the values of the map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.961Z
Params: (e: Column) Result: Column Returns an unordered array containing the values of the map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.961Z
(map-zip-with left right merge-fn)
Params: (left: Column, right: Column, f: (Column, Column, Column) ⇒ Column) Result: Column Merge two given maps, key-wise into a single map using a function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.963Z
Params: (left: Column, right: Column, f: (Column, Column, Column) ⇒ Column) Result: Column Merge two given maps, key-wise into a single map using a function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.963Z
(md-5 expr)
Params: (e: Column) Result: Column Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.966Z
Params: (e: Column) Result: Column Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.966Z
(md5 expr)
Params: (e: Column) Result: Column Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.966Z
Params: (e: Column) Result: Column Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.966Z
(minute expr)
Params: (e: Column) Result: Column Extracts the minutes as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.971Z
Params: (e: Column) Result: Column Extracts the minutes as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.971Z
(monotonically-increasing-id)
Params: () Result: Column A column expression that generates monotonically increasing 64-bit integers. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.233Z
Params: () Result: Column A column expression that generates monotonically increasing 64-bit integers. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.233Z
(month expr)
Params: (e: Column) Result: Column Extracts the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.974Z
Params: (e: Column) Result: Column Extracts the month as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.974Z
(months-between l-expr r-expr)
Params: (end: Column, start: Column) Result: Column Returns number of months between dates start and end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.979Z
Params: (end: Column, start: Column) Result: Column Returns number of months between dates start and end. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.979Z
(nanvl left-expr right-expr)
Params: (col1: Column, col2: Column) Result: Column Returns col1 if it is not NaN, or col2 if col1 is NaN. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.980Z
Params: (col1: Column, col2: Column) Result: Column Returns col1 if it is not NaN, or col2 if col1 is NaN. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.980Z
(negate expr)
Params: (e: Column) Result: Column Unary minus, i.e. negate the expression. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.982Z
Params: (e: Column) Result: Column Unary minus, i.e. negate the expression. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.982Z
(next-day expr day-of-week)
Params: (date: Column, dayOfWeek: String) Result: Column Returns the first date which is later than the value of the date column that is on the specified day of the week. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.984Z
Params: (date: Column, dayOfWeek: String) Result: Column Returns the first date which is later than the value of the date column that is on the specified day of the week. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.984Z
(not expr)
Params: (e: Column) Result: Column Inversion of boolean expression, i.e. NOT. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.985Z
Params: (e: Column) Result: Column Inversion of boolean expression, i.e. NOT. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.985Z
(ntile n)
Params: (n: Int) Result: Column Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition. For example, if n is 4, the first quarter of the rows will get value 1, the second quarter will get 2, the third quarter will get 3, and the last quarter will get 4. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.988Z
Params: (n: Int) Result: Column Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition. For example, if n is 4, the first quarter of the rows will get value 1, the second quarter will get 2, the third quarter will get 3, and the last quarter will get 4. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.988Z
(overlay src rep pos)
(overlay src rep pos len)
Params: (src: Column, replace: Column, pos: Column, len: Column) Result: Column Overlay the specified portion of src with replace, starting from byte position pos of src and proceeding for len bytes.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.991Z
Params: (src: Column, replace: Column, pos: Column, len: Column) Result: Column Overlay the specified portion of src with replace, starting from byte position pos of src and proceeding for len bytes. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.991Z
(percent-rank)
Params: () Result: Column Window function: returns the relative rank (i.e. percentile) of rows within a window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.992Z
Params: () Result: Column Window function: returns the relative rank (i.e. percentile) of rows within a window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.992Z
The double value that is closer than any other to pi, the ratio of the circumference of a circle to its diameter.
The double value that is closer than any other to pi, the ratio of the circumference of a circle to its diameter.
(pmod left-expr right-expr)
Params: (dividend: Column, divisor: Column) Result: Column Returns the positive value of dividend mod divisor.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.994Z
Params: (dividend: Column, divisor: Column) Result: Column Returns the positive value of dividend mod divisor. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.994Z
(posexplode expr)
Params: (e: Column) Result: Column Creates a new row for each element with position in the given array or map column. Uses the default column name pos for position, and col for elements in the array and key and value for elements in the map unless specified otherwise.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.995Z
Params: (e: Column) Result: Column Creates a new row for each element with position in the given array or map column. Uses the default column name pos for position, and col for elements in the array and key and value for elements in the map unless specified otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.995Z
(posexplode-outer expr)
Params: (e: Column) Result: Column Creates a new row for each element with position in the given array or map column. Uses the default column name pos for position, and col for elements in the array and key and value for elements in the map unless specified otherwise.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.995Z
Params: (e: Column) Result: Column Creates a new row for each element with position in the given array or map column. Uses the default column name pos for position, and col for elements in the array and key and value for elements in the map unless specified otherwise. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:40.995Z
(pow base exponent)
Params: (l: Column, r: Column) Result: Column Returns the value of the first argument raised to the power of the second argument.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.009Z
Params: (l: Column, r: Column) Result: Column Returns the value of the first argument raised to the power of the second argument. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.009Z
(quarter expr)
Params: (e: Column) Result: Column Extracts the quarter as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.010Z
Params: (e: Column) Result: Column Extracts the quarter as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.010Z
(radians expr)
Params: (e: Column) Result: Column Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.013Z
Params: (e: Column) Result: Column Converts an angle measured in degrees to an approximately equivalent angle measured in radians. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.013Z
(rand)
(rand seed)
Params: (seed: Long) Result: Column Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0).
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.015Z
Params: (seed: Long) Result: Column Generate a random column with independent and identically distributed (i.i.d.) samples uniformly distributed in [0.0, 1.0). Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.015Z
(randn)
(randn seed)
Params: (seed: Long) Result: Column Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.017Z
Params: (seed: Long) Result: Column Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.017Z
(rank)
Params: () Result: Column Window function: returns the rank of rows within a window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.018Z
Params: () Result: Column Window function: returns the rank of rows within a window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.018Z
(regexp-extract expr regex idx)
Params: (e: Column, exp: String, groupIdx: Int) Result: Column Extract a specific group matched by a Java regex, from the specified string column. If the regex did not match, or the specified group did not match, an empty string is returned.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.019Z
Params: (e: Column, exp: String, groupIdx: Int) Result: Column Extract a specific group matched by a Java regex, from the specified string column. If the regex did not match, or the specified group did not match, an empty string is returned. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.019Z
(regexp-replace expr pattern-expr replacement-expr)
Params: (e: Column, pattern: String, replacement: String) Result: Column Replace all substrings of the specified string value that match regexp with rep.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.022Z
Params: (e: Column, pattern: String, replacement: String) Result: Column Replace all substrings of the specified string value that match regexp with rep. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.022Z
(reverse expr)
Params: (e: Column) Result: Column Returns a reversed string or an array with reverse order of elements. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.024Z
Params: (e: Column) Result: Column Returns a reversed string or an array with reverse order of elements. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.024Z
(rint expr)
Params: (e: Column) Result: Column Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.026Z
Params: (e: Column) Result: Column Returns the double value that is closest in value to the argument and is equal to a mathematical integer. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.026Z
(round expr)
Params: (e: Column) Result: Column Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.028Z
Params: (e: Column) Result: Column Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.028Z
(row-number)
Params: () Result: Column Window function: returns a sequential number starting at 1 within a window partition.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.029Z
Params: () Result: Column Window function: returns a sequential number starting at 1 within a window partition. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.029Z
(rpad expr length pad)
Params: (str: Column, len: Int, pad: String) Result: Column Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.030Z
Params: (str: Column, len: Int, pad: String) Result: Column Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.030Z
(rtrim expr)
Params: (e: Column) Result: Column Trim the spaces from right end for the specified string value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.032Z
Params: (e: Column) Result: Column Trim the spaces from right end for the specified string value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.032Z
(schema-of-csv expr)
(schema-of-csv expr options)
Params: (csv: String) Result: Column Parses a CSV string and infers its schema in DDL format.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.036Z
Params: (csv: String) Result: Column Parses a CSV string and infers its schema in DDL format. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.036Z
(schema-of-json expr)
(schema-of-json expr options)
Params: (json: String) Result: Column Parses a JSON string and infers its schema in DDL format.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.043Z
Params: (json: String) Result: Column Parses a JSON string and infers its schema in DDL format. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.043Z
(second expr)
Params: (e: Column) Result: Column Extracts the seconds as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.045Z
Params: (e: Column) Result: Column Extracts the seconds as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.045Z
(sequence start stop step)
Params: (start: Column, stop: Column, step: Column) Result: Column Generate a sequence of integers from start to stop, incrementing by step.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.047Z
Params: (start: Column, stop: Column, step: Column) Result: Column Generate a sequence of integers from start to stop, incrementing by step. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.047Z
(sha-1 expr)
Params: (e: Column) Result: Column Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.048Z
Params: (e: Column) Result: Column Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.048Z
(sha-2 expr n-bits)
Params: (e: Column, numBits: Int) Result: Column Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.049Z
Params: (e: Column, numBits: Int) Result: Column Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.049Z
(sha1 expr)
Params: (e: Column) Result: Column Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.048Z
Params: (e: Column) Result: Column Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.048Z
(sha2 expr n-bits)
Params: (e: Column, numBits: Int) Result: Column Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.049Z
Params: (e: Column, numBits: Int) Result: Column Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.049Z
(shift-left expr num-bits)
Params: (e: Column, numBits: Int) Result: Column Shift the given value numBits left. If the given value is a long value, this function will return a long value else it will return an integer value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.050Z
Params: (e: Column, numBits: Int) Result: Column Shift the given value numBits left. If the given value is a long value, this function will return a long value else it will return an integer value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.050Z
(shift-right expr num-bits)
Params: (e: Column, numBits: Int) Result: Column (Signed) shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.052Z
Params: (e: Column, numBits: Int) Result: Column (Signed) shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.052Z
(shift-right-unsigned expr num-bits)
Params: (e: Column, numBits: Int) Result: Column Unsigned shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.053Z
Params: (e: Column, numBits: Int) Result: Column Unsigned shift the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.053Z
(sign expr)
Params: (e: Column) Result: Column Computes the signum of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.056Z
Params: (e: Column) Result: Column Computes the signum of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.056Z
(signum expr)
Params: (e: Column) Result: Column Computes the signum of the given value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.056Z
Params: (e: Column) Result: Column Computes the signum of the given value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.056Z
(sin expr)
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.058Z
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.058Z
(sinh expr)
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.060Z
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.060Z
(size expr)
Params: (e: Column) Result: Column Returns length of array or map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.062Z
Params: (e: Column) Result: Column Returns length of array or map. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.062Z
(skewness expr)
Params: (e: Column) Result: Column Aggregate function: returns the skewness of the values in a group.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.064Z
Params: (e: Column) Result: Column Aggregate function: returns the skewness of the values in a group. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.064Z
(slice expr start length)
Params: (x: Column, start: Int, length: Int) Result: Column Returns an array containing all the elements in x from index start (or starting from the end if start is negative) with the specified length.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.065Z
Params: (x: Column, start: Int, length: Int) Result: Column Returns an array containing all the elements in x from index start (or starting from the end if start is negative) with the specified length. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.065Z
(sort-array expr)
(sort-array expr asc)
Params: (e: Column) Result: Column Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements. Null elements will be placed at the beginning of the returned array.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.068Z
Params: (e: Column) Result: Column Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements. Null elements will be placed at the beginning of the returned array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.068Z
(soundex expr)
Params: (e: Column) Result: Column Returns the soundex code for the specified expression.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.069Z
Params: (e: Column) Result: Column Returns the soundex code for the specified expression. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.069Z
(spark-partition-id)
Params: () Result: Column Partition ID.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.070Z
Params: () Result: Column Partition ID. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.070Z
(split expr pattern)
Params: (str: Column, pattern: String) Result: Column Splits str around matches of the given pattern.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.073Z
Params: (str: Column, pattern: String) Result: Column Splits str around matches of the given pattern. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.073Z
(sqr expr)
Returns the value of the first argument raised to the power of two.
Returns the value of the first argument raised to the power of two.
(sqrt expr)
Params: (e: Column) Result: Column Computes the square root of the specified float value.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.075Z
Params: (e: Column) Result: Column Computes the square root of the specified float value. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.075Z
(std expr)
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
(stddev expr)
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
(stddev-pop expr)
Params: (e: Column) Result: Column Aggregate function: returns the population standard deviation of the expression in a group.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.091Z
Params: (e: Column) Result: Column Aggregate function: returns the population standard deviation of the expression in a group. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.091Z
(stddev-samp expr)
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
Params: (e: Column) Result: Column Aggregate function: alias for stddev_samp. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.077Z
(struct & exprs)
Params: (cols: Column*) Result: Column Creates a new struct column. If the input column is a column in a DataFrame, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated as col with a suffix index + 1, i.e. col1, col2, col3, ...
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.096Z
Params: (cols: Column*) Result: Column Creates a new struct column. If the input column is a column in a DataFrame, or a derived column expression that is named (i.e. aliased), its name would be retained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated as col with a suffix index + 1, i.e. col1, col2, col3, ... Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.096Z
(substring expr pos len)
Params: (str: Column, pos: Int, len: Int) Result: Column Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.097Z
Params: (str: Column, pos: Int, len: Int) Result: Column Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.097Z
(substring-index expr delim cnt)
Params: (str: Column, delim: String, count: Int) Result: Column Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.098Z
Params: (str: Column, delim: String, count: Int) Result: Column Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.098Z
(sum-distinct expr)
Params: (e: Column) Result: Column Aggregate function: returns the sum of distinct values in the expression.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.103Z
Params: (e: Column) Result: Column Aggregate function: returns the sum of distinct values in the expression. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.103Z
(tan expr)
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.107Z
Params: (e: Column) Result: Column angle in radians Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.107Z
(tanh expr)
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.109Z
Params: (e: Column) Result: Column hyperbolic angle Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.109Z
(time-window time-expr duration)
(time-window time-expr duration slide)
(time-window time-expr duration slide start)
Params: (timeColumn: Column, windowDuration: String, slideDuration: String, startTime: String) Result: Column Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour: Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.220Z
Params: (timeColumn: Column, windowDuration: String, slideDuration: String, startTime: String) Result: Column Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour: Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.220Z
(to-csv expr)
(to-csv expr options)
Params: (e: Column, options: Map[String, String]) Result: Column (Java-specific) Converts a column containing a StructType into a CSV string with the specified schema. Throws an exception, in the case of an unsupported type.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.112Z
Params: (e: Column, options: Map[String, String]) Result: Column (Java-specific) Converts a column containing a StructType into a CSV string with the specified schema. Throws an exception, in the case of an unsupported type. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.112Z
(to-date expr)
(to-date expr date-format)
Params: (e: Column) Result: Column Converts the column into DateType by casting rules to DateType.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.115Z
Params: (e: Column) Result: Column Converts the column into DateType by casting rules to DateType. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.115Z
(to-timestamp expr)
(to-timestamp expr date-format)
Params: (s: Column) Result: Column Converts to a timestamp by casting rules to TimestampType.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.123Z
Params: (s: Column) Result: Column Converts to a timestamp by casting rules to TimestampType. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.123Z
(to-utc-timestamp expr)
Params: (ts: Column, tz: String) Result: Column Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.125Z
Params: (ts: Column, tz: String) Result: Column Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.125Z
(transform expr xform-fn)
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns an array of elements after applying a transformation to each element in the input array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.128Z
Params: (column: Column, f: (Column) ⇒ Column) Result: Column Returns an array of elements after applying a transformation to each element in the input array. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.128Z
(transform-keys expr key-fn)
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.130Z
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new keys for the pairs. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.130Z
(transform-values expr key-fn)
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.138Z
Params: (expr: Column, f: (Column, Column) ⇒ Column) Result: Column Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.138Z
(translate expr match replacement)
Params: (src: Column, matchingString: String, replaceString: String) Result: Column Translate any character in the src by a character in replaceString. The characters in replaceString correspond to the characters in matchingString. The translate will happen when any character in the string matches the character in the matchingString.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.139Z
Params: (src: Column, matchingString: String, replaceString: String) Result: Column Translate any character in the src by a character in replaceString. The characters in replaceString correspond to the characters in matchingString. The translate will happen when any character in the string matches the character in the matchingString. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.139Z
(trim expr trim-string)
Params: (e: Column) Result: Column Trim the spaces from both ends for the specified string column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.141Z
Params: (e: Column) Result: Column Trim the spaces from both ends for the specified string column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.141Z
(unbase-64 expr)
Params: (e: Column) Result: Column Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.188Z
Params: (e: Column) Result: Column Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.188Z
(unbase64 expr)
Params: (e: Column) Result: Column Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.188Z
Params: (e: Column) Result: Column Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.188Z
(unhex expr)
Params: (column: Column) Result: Column Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to the byte representation of number.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.189Z
Params: (column: Column) Result: Column Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to the byte representation of number. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.189Z
(unix-timestamp)
(unix-timestamp expr)
(unix-timestamp expr pattern)
Params: () Result: Column Returns the current Unix timestamp (in seconds) as a long.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.197Z
Params: () Result: Column Returns the current Unix timestamp (in seconds) as a long. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.197Z
(upper expr)
Params: (e: Column) Result: Column Converts a string column to upper case.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.198Z
Params: (e: Column) Result: Column Converts a string column to upper case. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.198Z
(var-pop expr)
Params: (e: Column) Result: Column Aggregate function: returns the population variance of the values in a group.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.200Z
Params: (e: Column) Result: Column Aggregate function: returns the population variance of the values in a group. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.200Z
(var-samp expr)
Params: (e: Column) Result: Column Aggregate function: alias for var_samp.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.204Z
Params: (e: Column) Result: Column Aggregate function: alias for var_samp. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.204Z
(variance expr)
Params: (e: Column) Result: Column Aggregate function: alias for var_samp.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.204Z
Params: (e: Column) Result: Column Aggregate function: alias for var_samp. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.204Z
(week-of-year expr)
Params: (e: Column) Result: Column Extracts the week number as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.209Z
Params: (e: Column) Result: Column Extracts the week number as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.209Z
(weekofyear expr)
Params: (e: Column) Result: Column Extracts the week number as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.209Z
Params: (e: Column) Result: Column Extracts the week number as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.209Z
(when condition if-expr)
(when condition if-expr else-expr)
Params: (condition: Column, value: Any) Result: Column Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.211Z
Params: (condition: Column, value: Any) Result: Column Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.211Z
(window time-expr duration)
(window time-expr duration slide)
(window time-expr duration slide start)
Params: (timeColumn: Column, windowDuration: String, slideDuration: String, startTime: String) Result: Column Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour: Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.220Z
Params: (timeColumn: Column, windowDuration: String, slideDuration: String, startTime: String) Result: Column Bucketize rows into one or more time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive, e.g. 12:05 will be in the window [12:05,12:10) but not in [12:00,12:05). Windows can support microsecond precision. Windows in the order of months are not supported. The following example takes the average stock price for a one minute window every 10 seconds starting 5 seconds after the hour: Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.220Z
(xxhash-64 & exprs)
Params: (cols: Column*) Result: Column Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.222Z
Params: (cols: Column*) Result: Column Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.222Z
(xxhash64 & exprs)
Params: (cols: Column*) Result: Column Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column.
Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.222Z
Params: (cols: Column*) Result: Column Calculates the hash code of given columns using the 64-bit variant of the xxHash algorithm, and returns the result as a long column. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.222Z
(year expr)
Params: (e: Column) Result: Column Extracts the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.223Z
Params: (e: Column) Result: Column Extracts the year as an integer from a given date/timestamp/string. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.223Z
(zip-with left right merge-fn)
Params: (left: Column, right: Column, f: (Column, Column) ⇒ Column) Result: Column Merge two given arrays, element-wise, into a single array using a function. If one array is shorter, nulls are appended at the end to match the length of the longer array, before applying the function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.226Z
Params: (left: Column, right: Column, f: (Column, Column) ⇒ Column) Result: Column Merge two given arrays, element-wise, into a single array using a function. If one array is shorter, nulls are appended at the end to match the length of the longer array, before applying the function. Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html Timestamp: 2020-10-02T14:21:41.226Z
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close