Solved

Hi , I have stream data (GetHDFS will be running continuosly ) which contains number of lines. e.g: <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. A stream of above lines of data will be in file I have to extract text from above message [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] I tried using a extract text processor and used custom property extract: ([hdfs.*log"]). I tried the above in java regex evaluator, it shows correct text extracted. but when I run the flow, output gets the complete text. expected: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] actual : <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. Please help me to corr

Hello, Just wondering if there is a way to help developers identify in which NiFi UI canvas they are on - Dev, Test, Prod; this is to help prevent developers from accidentally making changes in the wrong environments if they have multiple environments' canvases open; ideally a place like the top left corner where the NiFi logo is would be a good place to identify the environment; I checked the nifi properties file and other than the banner (which is a very small size font) I didn't find any other property where you can have a custom text/image/logo on the canvas. Also, I thought about the Label, but you've to add that inside each process group for it to be useful. Any ideas on how this can be done? Thanks.

Hi All, I want to fetch the data that is stored in HDFS using FetchHDFS processor . The folder structure to store our data is like /MajorData/Location/Year/Month/Day/file1.txt (/MajorData/Location/2017/01/01/file1.txt) As the day changes the folder structure will change to /MajorData/Location/2017/01/02/file2.txt How can I write a Nifi expression which will traverse through all the folders, fetch the data in NiFi?

I have HDP2.6, non-Kerberos cluster. I installed Ranger KMS. The service fails after startup. Can anyone direct me if I am missing anything? 2017-07-13 00:03:21,648 INFO log - ------------------ Ranger KMSWebApp--------------------- 2017-07-13 00:03:21,648 INFO log - provider string = dbks://http@localhost:9292/kms 2017-07-13 00:03:21,649 INFO log - URI = dbks://http@localhost:9292/kms scheme = dbks 2017-07-13 00:03:21,649 INFO log - kmsconf size= 225 kms classname=org.apache.hadoop.conf.Configuration 2017-07-13 00:03:21,649 INFO log - ----------------Instantiating key provider --------------- 2017-07-13 00:03:22,001 INFO RangerKeyStoreProvider - Credential keystore password not applied for KMS; clear text password shall be applicable 2017-07-13 00:03:22,002 ERROR RangerKMSDB - DB Flavor could not be determined 2017-07-13 00:03:23,133 INFO RangerKMSDB - Connected to DB : true 2017-07-13 00:03:23,136 INFO RangerMasterKey - Generating Master Key 2017-07-13 00:03:23,147 INFO AuditProvider

I have a flow in NiFi which splits a file into individual lines, inserts those lines into a database and after those have been inserted updates a control table. The control table only updates after every line has been inserted. To achieve this, the fragment.index is compared to fragment.count - if these are equal, then I know that every line has been processed and we can move on to updating the control table. However recently some of our files failed to update the control table. I have outputted the Attributes of the flow files to disk, and it shows something that confuses me: the number of flow files that comes out of the split text processor is 66430, which matches the number of lines in the file. However, the fragment.count attribute is 66443. Does anybody know why the fragment index would be incorrect, and how I can fix this?

I have a workflow that routes on attribute depending on a type, but it sets a RouteOnAttribute.Route param on a flow that has all of the filters, I don't want that, how can I prevent the processor to do this? Thanks

hi all, I am very new to nifi, i have try to adopt backpressure to control the throughput and i did a test as bleow, setting the highlight link backpressure threshold as screen cap. however, seems the 37.05MB data pass from "ConvertCSVToAvro" to "ConvertAvroToJSON" quickly , feels like backpressure doesn't work. I have read https://community.hortonworks.com/articles/9785/nifihdf-dataflow-optimization-part-2-of-2.html and flow the setting. Would anyone share me tips of using controlrate and backpressure to prevent overwhelm to the system? currently if the input data is large than 500M (CSV), it has OOM error.

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!! I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx). This happening for the MapReduce jobs only!!! When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!   An existing userid (user1) works fine! It seems to me that the issue is permissions-related.   Here are the HDFS permissions of the userids: [root@master ~]# hdfs dfs -ls /tmp/logsFound 5 itemsdrwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfsdrwxrwxrwx&nbs

I'm in the America/New_York timezone and my servers run ntp, and they show the right time. The header of NiFi running on those servers is off by one hour, eg system time says 11:00 and NiFi says 10:00. This question is very similar to https://community.hortonworks.com/questions/76724/change-timezone-in-nifi.html but I've tried that solution and no joy. This feels like a daylight savings problem, as the gui says 10:00:00 EST, and really that should be EDT since at time of writing it is summer. I'm using Oracle Java 1.8.0_71. Any ideas?

1. I am trying to get data from https://support.appsflyer.com/hc/en-us/articles/207034356-Push-APIs-Real-Time-Installation-and-Conversion-Notification-APIs The page say: NOTE: Push API only supports the use of port 80 and 443. I believe I can use NIFI to stream data from a webhook. My question is how to run the HttpListen Processor on port 80 or 443. I tried putting these ports in there and that seems to not work. Secondly, I tried posting to my small workflow in Nifi that has just two components httplistener and putHDFS. It just shows the bytes stuck in the queue and not being written to hdfs. Besides how do I run this on port 80 or 443 as these are the only two ports supported. Are there any other processors I could possibly use? Any help me is appreciated

I want to give all Read access for UI and all new crated processes and process groups by default from coding for each new user. Instead of explicitly go and give the view rights. I think we can do this by making changes in java files. But I am not able to find out the exact flow from java files. Expecting a solution for this. Thanks in advance.

I am running a ListHDFS processor pointing to a directory on hdfs on a timer driven schedule set to execute once per hour. After making sure the state is clear on the processor, I run it and see that it creates a flow file for all but 1 file in the directory. There are 5 files in the directory, and only 4 flowfiles are created. If I add more files and clear the state and attempt to run again, the pattern repeats, always one less flowfile is create, so one file is missed. It is not the same file that is missed with each run. Why is the processor missing 1 file each time? Is this by design? This is in HDF 2.1.0.1 and Apache NiFi - Version 1.1.0.2.1.0.1-1

I am working with nifi cluster. As in tutorial it is suggestion to use remote process group with cluster for load balancing. But to use Remote process group, we have to provide url of other nifi instance which is hardcoded. So what will happen if that nifi instance goes down in between of workflow execution. Is remote process call is working in that situation as load balancing concept is implemented in Remote process call.

Current setup is using getSftp --> puSFTP to sync data from sftp server to a remote linux host. Problem is with getSFTP i cannot define File age i wish to download, though i found getFile has the min max file age option, however nifi doesn't allow on canvas link between getSFTP and getFile. I was wondering how i can accomplish this, thanks in advance.

NULL

Hi, I have a Process Group which has an internal output port with a large queue. When I exit the Process Group and view the queue from the output of that Process Group to another group it seems to be very small. It seems as though there is a limit/time on the number of FlowFiles being transferred from one port to another even though these ports are both within Nifi without a process in between. Example: Process Group A has an internal output Port ExampleOutputPortA with a queue of XX,XXX FlowFiles. When you exit ExampleA and view the queue into port ExampleInputPortB the queue is XXX which is constantly lower than the queue within ExampleA.

NULL

Hi All, Is there any steps we have to deploy custom nars because i am facing the below errors when deploying custom nars in default lib dir. https://community.hortonworks.com/questions/97610/nifi-javalangnosuchmethoderror.html Please advice.

I am working on a flow getting files from a sftp site, i am not shure what i am during wrong configuring the listSftp processor I this flow ( just startet ) and keep getting this error My configuration in listSftp is current I have validated that the file exits on the nifi server in my directory, even tried to chmod 777 so everybody can do everything on it. ( i did this just for debug purpose ) I have this sence that i am during something basiclly wrong. I have keyfile, hostname, username, a password (key phrase i thing ) there are 100 chars I thing it is strange that nifi cannot validate that the file exits in /home/230960adm/ on the nifi server Any suggestions ?

Hello Everytime i face an error in my nifi workflow, the gethdfs processor recrawls the hdfs directory right from the beginning. I want to keep the files where they are in hdfs (keep source file = true) How can i have the gethdfs processor continue from where it stopped? Thanks

Hi all, There are something strange with data provenance of listHDFS (name listHDFS) processor in process group ExLibris. When I see the data provenance of this processor and i'm suprising that it is talk about the component that is outside of this process group. The data provenance seems get info from process group AutoClean. I think it is more clear with these images : data provenance of listHDFS from process group EXLIBRIS, in the data provenance we can see about component come from AUTOCLEAN dataflow ! process group AUTOCLEAN

Hi All, In mynifi flow i have two processor one is GetFTP and PutS3Object . consider i have one file in FTP a.txt .after the data get into the S3 the a.txt's timestamp is 12:00:00 in S3 after sometime again one file 'b.txt' is put into ftp now the S3 have two files as below but the timestamp in the S3 is changed for both a.txt and b.txt a.txt 12:01:00 b.txt 12:01:00

Checkpointing process is defined in 2 configuration parameters: dfs.namenode.checkpoint.period -> 10 hr which will set the time for every checkpoint. Aslos dfs.namenode.checkpoint.txns is by default set to 1 million. What does this configuration do, in apache wiki website it says this configurations sets the number of uncheckpointed transactions. In the first configuration we are setting it for every 10hrs, in such scenario what is the necessity for setting value for uncheckpointed transactions.

I am trying to set an order to the files Tared in a merge content processor, is this possible somehow? I need the order of the binary bits to be set so that one file (based on filename) can always be first. In this case I need the testfile1.txt to be first and then then testfile2.txt to be second. It seems to ignore my priorityattrbribute prioritizer. Thank you!

The remote process group on insance-1 trying to connect to instance-2 says forbidden. How can i fix this?