Skip to content

Commit

Permalink
feat: String functions resolves hablapps#71
Browse files Browse the repository at this point in the history
  • Loading branch information
eruizalo committed Sep 20, 2021
1 parent 21bc687 commit 37edb42
Show file tree
Hide file tree
Showing 8 changed files with 1,606 additions and 12 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Type-safe columns for DataFrames!
![Maven Central](https://img.shields.io/maven-central/v/org.hablapps/doric_2.12)

Doric offers type-safety in DataFrame column expressions at a minimum
cost, without compromising performace. In particular, doric allows you
cost, without compromising performance. In particular, doric allows you
to:

* Get rid of malformed column expressions at compile time
Expand Down
29 changes: 27 additions & 2 deletions src/main/scala/doric/syntax/CommonColumns.scala
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import cats.implicits._
import doric.sem.Location
import doric.types.{Casting, SparkType, UnsafeCasting}

import org.apache.spark.sql.Column
import org.apache.spark.sql.{Column, functions => f}
import org.apache.spark.sql.types.DataType

trait CommonColumns extends ColGetters[DoricColumn] {
Expand All @@ -19,6 +19,21 @@ trait CommonColumns extends ColGetters[DoricColumn] {
*/
@inline def dataType[T: SparkType]: DataType = SparkType[T].dataType

/**
* Returns the first column that is not null, or null if all inputs are null.
*
* For example, `coalesce(a, b, c)` will return a if a is not null, or b if a
* is null and b is not null, or c if both a and b are null but c is not
* null.
*
* @param cols
* the String DoricColumns to coalesce
* @return
* the first column that is not null, or null if all inputs are null.
*/
def coalesce[T](cols: DoricColumn[T]*): DoricColumn[T] =
cols.map(_.elem).toList.sequence.map(f.coalesce(_: _*)).toDC

override protected def constructSide[T](
column: Doric[Column]
): DoricColumn[T] =
Expand Down Expand Up @@ -60,11 +75,21 @@ trait CommonColumns extends ColGetters[DoricColumn] {
* @param other
* the column to compare
* @return
* a reference to a Boolean DoricColumn whit the comparation
* a reference to a Boolean DoricColumn with the comparation
*/
def ===(other: DoricColumn[T]): BooleanColumn =
(column.elem, other.elem).mapN(_ === _).toDC

/**
* Type safe distinct between Columns
* @param other
* the column to compare
* @return
* a reference to a Boolean DoricColumn with the comparation
*/
def =!=(other: DoricColumn[T]): BooleanColumn =
(column.elem, other.elem).mapN(_ =!= _).toDC

/**
* Pipes the column with the provided transformation
* @param f
Expand Down
Loading

0 comments on commit 37edb42

Please sign in to comment.