Solved

My Usecase: Pick log data and send it to PutSplunk. Solution I am trying is: I used TailFile processor which will have details of path of log file. And it will keep fetching the updated data from this file and will put in FlowFile routed to PutSplunk. As in the image attached, its not picking any more data and seems like hung, although the data is queued up. and the processors are in running state. Also added the screen dump of Processor Configuration

Hi all, Do you know how use deleteHDFS to remove empty directories ? thanks

I try to load data from my local system file into HDFS by using Getfile processor in Nifi, but it produces "invalid directory" error

Hi I have the processors unpackContent -> MergeContent. I use this to untar a file and then zip the files. I am using the defragment merge strategy and have been noticing that when MergeContent has to handle many flowfiles at once (flowfile queue builds up before MergeContent) from many different fragments I get "Expected number of fragments is X but only getting Y". Simply routing failures back to merge content or creating a run schedule delay helped solve this but wondering why this would be happening. Thanks,

mergecontent.png I'm new to NIFI. I'm trying to consumeKafka json message and write to hdfs in one file, also the filename should be date of the day. I've tried to use mergecontent and write to hdfs, but it created multiple files. This is my mergecontent looks like:

I have an external table which is created with partitions and i would like to delete/drop few partition along with data as i no longer require it. Appreciate any suggestions.

Team, I have requirement where I need last accessed time of file, I tried to get it from ranger audit db but there are so many tmp files and dirs which are creating any issue. So is there any other way like command or API to get last access time.

How to execute multiple SQL queries in PutSql processor at a time Ex: insert into table_a values('test_1'); insert into table_b values('test_2'); Flow:GetFile -> ReplaceText -> PutSQL -> PutFile In the above flow ReplaceText processor will replace the flow file with above example query. With this SQL flow file PutSql processor getting triggered and failed with error.

Hello All, Can any please explain me the difference between kerberos ticket_lifetime vs renew_lifetime. Thanks Niranjan

Hello, couple of quick questions on Processor group with multiple input ports. 1) within the processor group, how do you distinguish between flowfiles that are coming from the various input ports 2) in data provenance page, is there a way to tell which flowfiles are from which input ports Thanks.

Hi, Just wanted to know how NiFi's MergeContent processor would behave when all 3 of these properties are set - 1) Correlation Attribute Name, 2) Minimum Number of Entries and 3) Max Bin Age; in my use case, the Correlation Attribute Name is the flowfile date (without the time stamp), since I want to merge files that are from the same day; I set the minimum number of entries, because the flowfiles come at varying intervals throughout the day and I wanted to have similar number of flowfiles in each merged file; at midnight, when the day crosses over to next day, the bin may not get filled because of these 2 criteria (same day files and min # of entries); does Max Bin Age act as an override, when the other 2 conditions are not satisfied, and tell MergeContent to create a merged file with whatever files are in the bin ? I'm thinking yes, but I wanted to confirm. Thank you.

Greetings, I am not sure if I understand why you would create multiple input and output ports for a process group (PG). What purpose would the additional ports serve ? I am thinking that if you want to "call" (or send/receive data to/from) the same PG from many different processors in NiFi, then you would use different ports for each processor, to avoid mixing data the PG is getting from the different processors. As an example (please see the image below) if the PG has a MergeContent processor, then I would not want to merge flow files that are coming from different Processors into the PG. Is that one (if any) of the reason for having multiple ports in a PG ?

In this post https://community.hortonworks.com/questions/71481/finding-and-purging-flow-files.html it shows how to purge flowfiles from nifi's ui. is the content purged as well? I assume yes, but would like confirmation.

I cannot get the following query run using PutHiveQL processor, it fails with the permission denied exception. I see that the processor emulates the same behavior as in Beeline. But, I have been able to run the query from Hive CLI and writes to a file as expected. So, we know that Hive shell is an option, but can you let me know if there is any specific settings that causes this behavior in Beeline (Hive2) preventing to write to local filesystem? insert overwrite local directory '/tmp' select current_date from dual Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxxxx] does not have [WRITE] privilege on [/tmp] (state=42000,code=40000)

How to capture the filesize and the filename, which can be found in the details section of the flowfile? I understand that the filename is listed in the attribute section, but is there a way to know the variable name for filesize? kafka-flow-file.jpg

Hello, First time posting here so sorry if this is in the wrong section / wrong format. First, some background : We started a POC using NiFi 1.0.0. We're using a 3 node cluster with limited ressources (this is a POC...). Each of the node has 16 cores, 32gb of ram and 2 volumes : a raid 1 volume for the OS and a Raid 10 volume on 2.5in splindles. I know this is not a recommended setup but the content repo, the provenance repo, the flow files, everything basically, is on the same raid 10 array. The disks are heavily used right now. Content Repo archiving is disabled. Now here's the thing : every 2-3 days, the disk fills up. Right now, the UI reports that we have, in queue : 450 000 (3.21gb). I would expect to have roughly the same amount of data in the nifi/content_repository folder but it's not the case : On one of the node, the content_repo folder is 73gb. I can't tell how big the 2 others nodes are since the "du -h" operation is still running after 10minutes but using "df", I can esti

Hi, I have a case where i have to route incoming flow files to different folders based on the File names. Lets say i have 1000 files that need to send to 5 folders. i thought this would work with Advanced settings of updateattribute where i can write the rules. but the folder name is not being populated.anything wrong i am doing .?? here my test flow..List-->Fetch-->UpdateAttribute-->PutFile and i tried to use the Folder attribute in the putfile ..i tried without single quotes also.. or is there any better approach..?? i need to route 8000 files in to 18 different folders.basically 70 files to 17 folders and remaining 7100+ files to a default folder..(per day 8K files , i need to load data for 18 months) Thanks for your help.. Regards, Sai

Im trying to do an app that takes a json from a web filter the url attribute of the JSON and get the html of this url. My problem is that I can’t set in the property remote url in InvokeHTTP the url that I got in SplitJSON. This is the config of splitJSON (it works, the output is the url that i want) And this is the conf of the InvokeHTTP (it work if I set manually the url) but I dont know how to set a dinamic url.

Hi all, Someone can explain me about "auto Terminate Relationships" in the processor ? thanks

Hello I have a scenario where I am getting confused, - I have Secured 3 node NiFi cluster [Kerberos] - Logged in as user 'admin' trying to do file-based authorization with out Ranger - Created a Process group named Group1 --> 5 processors with in the group - Set modify privilege on just GetSFTP processor for user 'john' in the Group1. - Once I set the privilege for GetSFTP, entire process group is allocated with the modify access for john, not just GetSFTP processor. I am thinking this is not an expected behavior, someone experienced this before? Thanks, Jobin George

Hi,I want to use standalone NiFi 0.6 to ingest data into Kafka within a kerberized HDP2.3.4 cluster.Base config ran smooth and additional config from https://community.hortonworks.com/articles/28180/how-to-configure-hdf-12-to-send-to-and-get-data-fr.html , "Section 2" has been applied.To start playing around I added a GetFile processor and connected it to PutKafka, but after starting the flow, PutKafka fails with the following error:2016-09-20 11:44:38,076 INFO [StandardProcessScheduler Thread-6] o.a.kafka.clients.producer.KafkaProducer Closing the Kafka producer with timeoutMillis = 0 ms.2016-09-20 11:44:38,076 ERROR [StandardProcessScheduler Thread-6] o.apache.nifi.processors.kafka.PutKafka PutKafka[id=9293cf22-d7c2-40a6-ba7e-a6d41942fc30] PutKafka[id=9293cf22-d7c2-40a6-ba7e-a6d41942fc30] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30000 milliseconds: ja

how to copy files from local desktop lo hdfs location using NiFi?

I have a flow that works on my desktop and I would like to move it to a single server that has 16 cores. My question is there anything I need to configure in NiFi to light up all the cores. I see documentation on clustering servers, which doesn't apply to me, or does NiFI use everything available, which is the behavior I'm looking for?

Storage limit of NN and total storage capacity?

In a typical HDFS HA deployment, Zookeeper, ZKFailoverController, DataNodes, NameNodes and JournalNodes, are all installed. What's the best order of starting/stoping these HDFS services to avoid running into services dependency issues?