Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Dataset Transformation Chaining

Spark Dataset Transformation Chaining

New Contributor

Hi,

 

I am trying to use the "transform" method of the "Dataset". I get an error wrt the second parameter of the transformation function. The code fragment is as below 

 

Function<Dataset<Row>, Dataset<Row>> myTransform1 = (r) -> {
r.select("lob").distinct().show();
};

 

dfInfile.transform(myTransform1);

 

The error message says

 

The method transform(Function1<Dataset<Row>,Dataset<U>>) in the type Dataset<Row> is not applicable for the arguments (Function<Dataset<Row>,Dataset<Row>>)

 

I am not sure how can I make the second argument of the functional interface as an untyped dataset "Dataset<U>".

 

Can somebody help me with this please ?

 

Thanks,

Venkat