Hi, To convert csv to dataframe we must be aware about the delimiter character at coding time, however in my case we are not aware about the same. The source file will be delimited by some character, our code should be able to infer the delimiter into the file and convert the file into dataframe. As of now i've written a java snippet to check the delimiter character first and tried to read the file. Do we have any predefined function to satisfy my need? Thanks, R ... View more

I've my spark job and deployed on hadoop cluster, In my case i've more than one edge nodes pointing to the same hadoop cluster, now my requirement is if there is some issues for edge node 1 my spark job should get triggered from another available edge node. What is the best way to do it? ... View more

Hi, I am expecting two files, file with abc.txt and file with pqr.txt my next processor should get triggered only if these two files are received. Currently i am using listFile fetchFIle to capture the changes of source directory. Once i have received two mandatory files my next process is to process them. Now i am not sure how should i configure NiFi to hold the file in queue until i got all mandatory files. Do we know how should we achieve this using NiFi? ... View more

Hi, I am validating xml schema against XSD which are coming in the NiFi flow. Schema validation is being done using validateXML processor. I've given XSD file to validate the schema. Now whenever there any change into xml schema file do we need to restart the NiFi cluster or simply stop the validateXML processor and start again will work? ... View more

Hello Friends, @Pierre Villard (Specifically tagging you as like your explanations) Currently i am getting messages from a Queue and i want to trigger out notification email only once for each file type. I've used putEmail processor to do the same but my flow is triggering email notification for every time new flow file came in and that is the expected behavior. I thought to use mergeContent here but again i am not sure how should i configure it. Descriptive requirement: I am getting files for department like computer, mechanical,electronics, etc. Now i want to send notification email only once in a day per file type. Send email as soon as we receive the file for specific department. Thanks for your help, ... View more

Hi, I want to write spark dataframe into hive table.Hive table is partitioned on year and month and file format is parquet. Currently i am writing dataframe into hive table using insertInto() and mode("append") i am able to write the data into hive table but i am not sure that is the correct way to do it? Also while writing i am getting "parquet.hadoop.codec.CompressionCodecNotSupportedException: codec not supported: org.apache.hadoop.io.compress.DefaultCodec" this exception. Could you please help me on this? Thanks for your time, ... View more

My Requirement is as soon as source put the files my spark job should get triggered out and process the files. Currently i am thinking to do like below. 1. Source will push the files in local directory /temp/abc 2. NiFi ListFiles and fetchFile will take care of ingestion of those files into HDFS. 3. On success relationship of putHDFS thinking to setup executeStreamCommand. Could you please suggest is there any best approach to do it? what will be the configuration for executeStreamCommand? Thanks in advance, R ... View more

Hi, I need to create an empty hive bucketed table from spark with parquet file format. Currently spark is spitting exception for conventional cluster by. Thanks ... View more

Hi Folks out there, Currently I have a scenario where i have to get only latest record per id from hive table based on timestamp. I am looking for best approach to do it. My data is in hive internal table with parquet files.Similar to this Using spark+hive. Thanks, ... View more