DataFrame groupBy and concat non-empty strings

jestinm — Wed, 13 Jul 2016 11:57:31 GMT

I want to concatenate non-empty values in a column after grouping by some key.

Eg:

Supposing I have a dataframe:

df.show()

+---+---+----+
| id|num|num2|
+---+---+----+
|  1|  3|   5|
|  2|  3|   4|
|  1|   |   2|
|  1| 10|   0|
+---+---+----+

I want to groupBy "id" and concatenate "num" together. Right now, I have this:

df.groupBy($"id").agg(concat_ws(DELIM, collect_list($"num")))

Which concatenates by key but doesn't exclude empty strings. Is there a way I can specify in the Column argument of concat_ws() or collect_list() to exclude some kind of string?

Thank you!

Re: DataFrame groupBy and concat non-empty strings

qiwang — Sat, 23 Jul 2016 04:11:07 GMT

Could you filter the empty string before the group?

df.where(df("number") !== "").groupBy($"id").agg(concat_ws(DELIM, collect_list($"num")))

question DataFrame groupBy and concat non-empty strings in Archives of Support Questions (Read Only)

DataFrame groupBy and concat non-empty strings

Re: DataFrame groupBy and concat non-empty strings