Q&A

Current setup is using getSftp --> puSFTP to sync data from sftp server to a remote linux host. Problem is with getSFTP i cannot define File age i wish to download, though i found getFile has the min max file age option, however nifi doesn't allow on canvas link between getSFTP and getFile. I was wondering how i can accomplish this, thanks in advance.

NULL

Hi, I have a Process Group which has an internal output port with a large queue. When I exit the Process Group and view the queue from the output of that Process Group to another group it seems to be very small. It seems as though there is a limit/time on the number of FlowFiles being transferred from one port to another even though these ports are both within Nifi without a process in between. Example: Process Group A has an internal output Port ExampleOutputPortA with a queue of XX,XXX FlowFiles. When you exit ExampleA and view the queue into port ExampleInputPortB the queue is XXX which is constantly lower than the queue within ExampleA.

NULL

Hi All, Is there any steps we have to deploy custom nars because i am facing the below errors when deploying custom nars in default lib dir. https://community.hortonworks.com/questions/97610/nifi-javalangnosuchmethoderror.html Please advice.

I am working on a flow getting files from a sftp site, i am not shure what i am during wrong configuring the listSftp processor I this flow ( just startet ) and keep getting this error My configuration in listSftp is current I have validated that the file exits on the nifi server in my directory, even tried to chmod 777 so everybody can do everything on it. ( i did this just for debug purpose ) I have this sence that i am during something basiclly wrong. I have keyfile, hostname, username, a password (key phrase i thing ) there are 100 chars I thing it is strange that nifi cannot validate that the file exits in /home/230960adm/ on the nifi server Any suggestions ?

Hello Everytime i face an error in my nifi workflow, the gethdfs processor recrawls the hdfs directory right from the beginning. I want to keep the files where they are in hdfs (keep source file = true) How can i have the gethdfs processor continue from where it stopped? Thanks

Hi all, There are something strange with data provenance of listHDFS (name listHDFS) processor in process group ExLibris. When I see the data provenance of this processor and i'm suprising that it is talk about the component that is outside of this process group. The data provenance seems get info from process group AutoClean. I think it is more clear with these images : data provenance of listHDFS from process group EXLIBRIS, in the data provenance we can see about component come from AUTOCLEAN dataflow ! process group AUTOCLEAN

Hi All, In mynifi flow i have two processor one is GetFTP and PutS3Object . consider i have one file in FTP a.txt .after the data get into the S3 the a.txt's timestamp is 12:00:00 in S3 after sometime again one file 'b.txt' is put into ftp now the S3 have two files as below but the timestamp in the S3 is changed for both a.txt and b.txt a.txt 12:01:00 b.txt 12:01:00

Checkpointing process is defined in 2 configuration parameters: dfs.namenode.checkpoint.period -> 10 hr which will set the time for every checkpoint. Aslos dfs.namenode.checkpoint.txns is by default set to 1 million. What does this configuration do, in apache wiki website it says this configurations sets the number of uncheckpointed transactions. In the first configuration we are setting it for every 10hrs, in such scenario what is the necessity for setting value for uncheckpointed transactions.

I am trying to set an order to the files Tared in a merge content processor, is this possible somehow? I need the order of the binary bits to be set so that one file (based on filename) can always be first. In this case I need the testfile1.txt to be first and then then testfile2.txt to be second. It seems to ignore my priorityattrbribute prioritizer. Thank you!

The remote process group on insance-1 trying to connect to instance-2 says forbidden. How can i fix this?

My Usecase: Pick log data and send it to PutSplunk. Solution I am trying is: I used TailFile processor which will have details of path of log file. And it will keep fetching the updated data from this file and will put in FlowFile routed to PutSplunk. As in the image attached, its not picking any more data and seems like hung, although the data is queued up. and the processors are in running state. Also added the screen dump of Processor Configuration

Hi all, Do you know how use deleteHDFS to remove empty directories ? thanks

I try to load data from my local system file into HDFS by using Getfile processor in Nifi, but it produces "invalid directory" error

Hi I have the processors unpackContent -> MergeContent. I use this to untar a file and then zip the files. I am using the defragment merge strategy and have been noticing that when MergeContent has to handle many flowfiles at once (flowfile queue builds up before MergeContent) from many different fragments I get "Expected number of fragments is X but only getting Y". Simply routing failures back to merge content or creating a run schedule delay helped solve this but wondering why this would be happening. Thanks,

mergecontent.png I'm new to NIFI. I'm trying to consumeKafka json message and write to hdfs in one file, also the filename should be date of the day. I've tried to use mergecontent and write to hdfs, but it created multiple files. This is my mergecontent looks like:

I have an external table which is created with partitions and i would like to delete/drop few partition along with data as i no longer require it. Appreciate any suggestions.

Team, I have requirement where I need last accessed time of file, I tried to get it from ranger audit db but there are so many tmp files and dirs which are creating any issue. So is there any other way like command or API to get last access time.

How to execute multiple SQL queries in PutSql processor at a time Ex: insert into table_a values('test_1'); insert into table_b values('test_2'); Flow:GetFile -> ReplaceText -> PutSQL -> PutFile In the above flow ReplaceText processor will replace the flow file with above example query. With this SQL flow file PutSql processor getting triggered and failed with error.

Hello All, Can any please explain me the difference between kerberos ticket_lifetime vs renew_lifetime. Thanks Niranjan

Hello, couple of quick questions on Processor group with multiple input ports. 1) within the processor group, how do you distinguish between flowfiles that are coming from the various input ports 2) in data provenance page, is there a way to tell which flowfiles are from which input ports Thanks.

Hi, Just wanted to know how NiFi's MergeContent processor would behave when all 3 of these properties are set - 1) Correlation Attribute Name, 2) Minimum Number of Entries and 3) Max Bin Age; in my use case, the Correlation Attribute Name is the flowfile date (without the time stamp), since I want to merge files that are from the same day; I set the minimum number of entries, because the flowfiles come at varying intervals throughout the day and I wanted to have similar number of flowfiles in each merged file; at midnight, when the day crosses over to next day, the bin may not get filled because of these 2 criteria (same day files and min # of entries); does Max Bin Age act as an override, when the other 2 conditions are not satisfied, and tell MergeContent to create a merged file with whatever files are in the bin ? I'm thinking yes, but I wanted to confirm. Thank you.

Greetings, I am not sure if I understand why you would create multiple input and output ports for a process group (PG). What purpose would the additional ports serve ? I am thinking that if you want to "call" (or send/receive data to/from) the same PG from many different processors in NiFi, then you would use different ports for each processor, to avoid mixing data the PG is getting from the different processors. As an example (please see the image below) if the PG has a MergeContent processor, then I would not want to merge flow files that are coming from different Processors into the PG. Is that one (if any) of the reason for having multiple ports in a PG ?

In this post https://community.hortonworks.com/questions/71481/finding-and-purging-flow-files.html it shows how to purge flowfiles from nifi's ui. is the content purged as well? I assume yes, but would like confirmation.