I think it won't same performance . Consider below two statements, Let say I have a DataFrame with columns a,b,c,d,e. I want to select only a,b,c 1) df.select("a","b","c") In this , it will only select the required columns and never bothered about other columns 2) df.select.("a","b","c","d","e").drop("d","e") In this First it select all the columns from DataFrame and then drops the unwanted columns. I think in second statement there will be some performance down.. If Iam wrong..please clarify it ... View more

I didn't .. I will go through it.. ... View more

thanks matt.. Fine that works, But the thing is that what if my table has 100 millions rows , whether the ExecuteSQL process pulls the entire rows in single execution or not . if it doesn't pull the all rows in one execution, then I can't use the above suggestion write. is there way to tackle this? ... View more

okay thank you.. one thing is there any way to execute ExecuteSQL Process only once..because it returning duplicate rows ... View more

At the outset, thank you so much for quick reply. We want to build a NiFi job where we will pass table name and it should list all the columns of that table. Further, we will filter few columns and will store required column data to flat file. This process should work for any number of tables irrespective of schema. We are looking to build such generic NiFi Processor. ... View more

Thanks you for the reply.. But Processors related to Databases will give output in AvroFormat write. I have gone through the InferAvroSchema process, for it input should be in json or csv file ..Then I am thinking this is possible or not. ... View more

you can use delete(outputpath,true) method which takes two arguments Example Path path = new Path(outputDirectory); path.getFileSystem(conf).delete(path,true); ... View more