@Lenu K We can do export to Hive ORC as follows: hive> Create table <db_name>.<orc_table_name> stored as orc as select * from <db_name>.<hbase_hive_table>; The above CTAS is generic statement even you can create a partitioned table (or) use distribute by sort by to create files in the directories. ... View more

@vivek jain Could you make hbaseConf.set properties changes directly in Spark hbase-site.xml file instead of setting those property values in spark job and then run spark-submit with newly changed hbase-site.xml? ... View more

@Sandip Dobariya Try with this configurations in ReplaceText processor and this configs, we are only applying nifi expression language replace function on first line of the extracted content not on all lines and this expression replaces spaces in first line. Search Value: (?s)(^[^\n]*)(.*$) Replacement Value: ${'$1':replace(" ","")}$2 Replacement Strategy RegexReplace Evaluation Mode Entire text Input: Date,Location,Name,Manager,Division,Revenue,PVNMS OnLine,NMS LHSC,RMS OnLineRMS LHSC Output: Date,Location,Name,Manager,Division,Revenue,PVNMSOnLine,NMSLHSC,RMSOnLineRMSLHSC ... View more

@Carlton Patterson This is not possible with default save/csv/json functions but using Hadoop API we can rename the filename. Example: >>> df=spark.sql("select int(1)id,string('ll')name") //create a dataframe >>> df.coalesce(1).write.mode("overwrite").csv("/user/shu/test/temp_dir") //writing the df to temp-dir >>> from py4j.java_gateway import java_import >>> java_import(spark._jvm, 'org.apache.hadoop.fs.Path') >>> fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration()) >>> file = fs.globStatus(sc._jvm.Path('/user/shu/test/temp_dir/part*'))[0].getPath().getName() //get the filename of temp_dir >>> fs.rename(sc._jvm.Path('/user/shu/test/temp_dir/' + file),sc._jvm.Path('/user/shu/test/mydata.csv')) //rename the temp directory file with desired filename and directory path >>> fs.delete(sc._jvm.Path('/user/shu/test/temp_dir'), True) //delete the temp directory. - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Nisha Patel If you look into configs of PublishKafkaRecord processor there is Record Reader/Writer controller services, so if your Record Writer is CsvSetWriter then you have configured Include Heder Line property value as true. i.e on each record you are writing the header so when we use Merge Content processor you are going to have header lines included for each record. . To resolve this issue Change the Include Heder Line value to False(now we are not writing header to each record) and then in Merge Content processor keep the Header property value as your header. So by following this way after merging completes then processor adds Header to the file. . Is there a specific reason why are you using PublishKafkaRecord processor? You can even use PublishKafka processor(because as you are splitting each record so there is no need to use Record oriented processors in this case unless if you have some valid reason.) which doesn't require any Record reader/writer controller services, so the message that we published into Kafka topic will be routed to Success relationship. Then use Merge Content processor to Merge all these flowfiles into one and then add the Header to the merged file. Flow: Replace PublishKafkaRecord processor with PublishKafka processor MergeContent processor ... View more