Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎09-01-2018

Spark Dataset Transformation Chaining

Hi,

 

I am trying to use the "transform" method of the "Dataset". I get an error wrt the second parameter of the transformation function. The code fragment is as below 

 

Function<Dataset<Row>, Dataset<Row>> myTransform1 = (r) -> {
r.select("lob").distinct().show();
};

 

dfInfile.transform(myTransform1);

 

The error message says

 

The method transform(Function1<Dataset<Row>,Dataset<U>>) in the type Dataset<Row> is not applicable for the arguments (Function<Dataset<Row>,Dataset<Row>>)

 

I am not sure how can I make the second argument of the functional interface as an untyped dataset "Dataset<U>".

 

Can somebody help me with this please ?

 

Thanks,

Venkat