Solved

Hi All, I have a scenario where i have a 3 node HDF cluster and i am making an API call to get the validation token(valid for 1 hour only) using InvokeHTTP every one hour on primary node only, since its not preferred to make multiple API calls. i am putting this token in Distributed Cache, but since i want to fetch the token from the distributedCache on all nodes, is there a way i can duplicate the flowfile from primary node to all nodes to the processors Downstreams. is there a way i can achieve this ?

I have a processor group that fetches data from ElasticSearch and stores it in mysql. Every time I run this processor group, NiFi will restart automatically after a while, which will cause my 'Scroll Duration=2m' to be disabled in ScrollElasticSearchHttp,and then I will never get the new data from the ES. I checked the log file and found no error message. this is the nifi-bootstrap_2018-10-09.log 2018-10-09 09:43:15,167 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...2018-10-09 09:43:15,343 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 158612018-10-09 09:43:15,373 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 158612018-10-09 09:43:16,401 INFO [NiFi logging handler] org.apache.nifi.StdOut Listening for transport dt_socket at address: 81112018-10-09 09:43:21,932 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and list

Hi, We are planning to use NiFi's Data Provenance on long term to use it for audits. What is the best way to configure NiFi ? If I "simply" backup the Data Provenance disk content, will I be able to use it later? By reinjecting it in a working NiFi? Should I use ReportingTaskProcessor? But again how do you query backuped data later? By keeping it in a dedicated NiFi used only for backup? I also did not understand the management of the FlowFile contents, is it supposed to be stored in the DataProvenance Disk (which would greatly increase its size...)? Or is the "replay button" from Data Provenance UI working only if the content is still fresh and present in the "Content repository"? Or is Data Provenance just not meant to used for long term purpose? Sorry if mess up multiple concepts. Thanks.

Hi, I am using RouteONAttribute Processor and i defined a property and its value contains multiple AND conditions like the following: Property Name: ROUTE1 Property Value: ${text.tag.type:equals('XXX'):and(${txt:contains('AAA')}):and(${txt:contains('BBB')})} The Issue i am facing is that after generating a flow file with attributes text.tag.type = XXX and txt = AAA,BBB it always route to unmatched while it should route to ROUTE1 Thanks ...

Hi, I'm watching 'On Demand' webinars, but saw 'Speed' button missing.This is slightly inconvenience when you want to watch webinar with 1.25x or 1.5x speed. Wondering, if someone from respective department addresses this issue. Sample webinar: https://www.cloudera.com/content/dam/www/marketing/resources/webinars/smarthub-on-cloudera-webinar.png.landing.html  

Hi All, Thanks a lot everyone on this awesome community. I have data flows in PROD now, and I am adding new data flows and testing them, however accidentally I stopped one the flows which was good and running in PROD, reuslting in data loss. Is there any way I can freeze the data flow or dataflow confirms twice before I stop a data flow? Thanks Dheeru

Hi Team, I am using PUTSQL processor to execute copy command in redshift database.This processor usually processes more than 20 flowfiles which are merged files containing json objects.I have configured it this way:(please see attached image). I want to know have i configured run-duration and run schedule correctly? My nifi canvas has more than 2000 processor and i am facing ui slowness issues.Please suggest how to decide these configurations in order to get high throughput and less resource consumption. Thanks in advance!! Sri

I have seen a situation wher some of the jobs suddenly failed with error saying "Cannot create a file. Namenode in safe mode." When I check the cloudera manager, HDFS_CANARY_HEALTH check failed. I also observed that the Active namenode become standaby and standby become active. So, if the HDFS_CANARY_HEALTH test fails then Namenode restarts by itself? Can anyone respond to this.

Hi everyone,   My system have 2 datanode, 2  namenode, 3 journalnode, 3 zookeeper service    I had config cluster namenode ok , when browsing the admin page namenode:50070 , I had see 1 name node status (active) and one namenode status (standby). => OK    When I stop active namenode the other with standby become active .  => OK   But the problem is how start the namenode which I had stop again ?   I do the following :   sudo -u hdfs hdfs namenode -bootstrapStandby -force/etc/init.d/hadoop-hdfs-namenode start With above process sometime namenode start ok with standby mode , but sometime it start with active mode and then I have 2 active node (split brain !!)   So what I have wrong , what is the right process to start a namenode had stop again   Thanks you          

Hi I am trying to send data from one nifi to another nifi (which are working in different nodes) using remote process group. So i created one remote process group and mention the url of other nifi host. Now whenever i am trying to connect a getfile or generate flowfile with this RPG its giving me error that it does not have input ports for RPG. I have configured the nifi.properties file as mentioned in here https://nifi.apache.org/minifi/getting-started.html and mention the site to site port number but it still not working. I am attaching some screenshots regarding my problem. Hoping for a solution. nifi1.png nifi2.png nifi3.png Thanks in Advance. Pratik

Hi, I am performing a basic check to see if a file exists in HDFS or not. I am using hdfs dos -test command for the same. But it doesn't seem to work correctly. In the documentation its mentioned as it would return 0 if the file exists. But I am not getting any output when the command is run. Let me know what need to be done to get this working. Please see the screenshot attached Thanks

We have setup Nifi site-to-site to transfer Snort logs (alert.csv) to a remote Nifi server, using Apache Metron (Hortonworks Cybersecurity). Once Snort is running, it produces a vast amount of logs in CSV format, more than 500 records only in 5 seconds for instance. The problem is that Nifi Tailfile cannot detect and separate all 500 records, it may detect around 20 changes (totally include 500 records). Current configurations are attached to this message.How can we configure the Tailfile processor to detect every single line as a new packet (event)? Any help would be appreciated.

HI, I would like to create data flow,which put file to one directory, and the if it has succeded, put the same file to another directory I use for that Notify/Wait, but I wonder if uuid is a good solution for the signal, Are Wait/Notify properties adequate ? or if there is another way ?

Hi, Our OU in the LDAP connection has changed and we are getting invalid ID,PWD errors when trying to log in to NiFi, I think this is expected. But if i change the old ou=Users and Groups,ou=EU,ou=same,ou=Organizations,dc=comp,dc=com to new ou=Users and Groups,ou=USA,ou=same,ou=Organizations,dc=comp,dc=com do i need to restart nifi.?? is there a way that this could be avoided .?? that is instead of giving the whole string can i just give the top level ou like ou=Users and Groups  

CDH 5.13.1Redhat 6.9We wish to change the number of replications from the default of 3 copies to 2 on one particular folder in hdfs.After running this on one cluster:$ hdfs dfs -setrep -R 2 /backupsand then doing a$ hdfs dfs -du /we saw that it freed the blocks very quickly and the output of fsck shows no "Over-replicated blocks":Status: HEALTHY Total size:    149514016589 B Total dirs:    27440 Total files:    128746 Total symlinks:        0 Total blocks (validated):    126355 (avg. block size 1183285 B) Minimally replicated blocks:    126355 (100.0 %) Over-replicated blocks:    0 (0.0 %) Under-replicated blocks:    0 (0.0 %) Mis-replicated blocks:        0 (0.0 %) Default replication factor:    3 Average block replication:    2.3367577&nb

I have two flow file: 1) $.category = sport 2) $.temp_array = "music" , "sport" , "traveling" I just want to check $.category is exist in $.temp_array or not using in expression language. Ex. ${category:in(${temp_array})} But it doesn't work with array or (variable). I have used this login on RouteOnAttribute processor. Instead of above example if i write like this: ${category:in("music","sport","traveling")} then ii will give you perfect output But this is static data. Instead of this i want to use dynamic. Please help.

Hi, I work on a NiFi flow getting data from a FTP server, sending this data to HDFS. I have to add to this flow the ability of checking data integrity between files fetched on the FTP, and files written on HDFS. To do this, I use the HashContent NiFi processor with MD5 algorithm to compute MD5 hash of flowfiles from start and end of the flow (I can get MD5 hash of each file on FTP, second MD5 is computed after a PutHDFS retrieving files having been written). Finally, I compare both values and consider data integrity is OK if they are equal. Do you have a general advice about this practice? Is this kind of check really useful with NiFi? Thanks, Benjamin

Hi, Is there anyway that we can run Remote Processor Group with multi threading (concurrent tasks) in NiFi Site-2-Site.? Regards, Sai

I have a Hbase cluster of 3 nodes. Below are the hostnames and daemons running on that machines. hbase2 - Hmaster,RegionServer, Datanode and Nodemanager hbase 1 - RegionServer,Namenode,Datanode, ResourceManager, NodeManager hbase 3 - RegionServer, Datanode and Nodemanager The moment I enter the run start-hbase.sh script. It throws the below given error2018-06-13 14:56:52,455 INFO [hbase2:60000.activeMasterManager] retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hbase2/10.94.1.119:8020 after 11 fail over attempts. Trying to fail over after sleeping for 17938ms. java.net.ConnectException: Call From hbase2/10.94.1.119 to hbase2:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused. Hmaster is not running. RegionServers are working fine

Hi, How to move the flowfiles from one queue/relationship to another one through NiFi Web Interface. Thanks,,

HI All, In our cluster NIFI is SSL enabled. Ranger is not SSL enabled. Both NIFI and Ranger are integrated with AD/LDAP. Before enabling NIFI plugin in Ranger, our AD/LDAP users are able to see NIFI UI. But after enabling NIFI plugin in Ranger, our AD/LDAP users are not able to see NIFI UI. We are getting following message on NIFI screen: Insufficient Permissions Untrusted proxy CN=*.test.com, OU=NIFI nifi-user.log shows Authentication success but Untrusted proxy error as follows: 2018-06-07 07:00:13,447 INFO [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://usdf24v0075.test.com:9091/nifi-api/flow/current-user (source ip: 10.23.118.51)2018-06-07 07:00:13,449 INFO [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for test-user2018-06-07 07:00:13,612 INFO [NiFi Web Server-18] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<test-user><CN=*.test.com, OU=NIFI>) GET https://usd

@Matt Burgess I have nifi job to capture cdc from MySQL DB.I am facing one issue with my nifi job in case of any failure. Suppose take an example if there is a failure while inserting record to DB using "PutSQL" processor in attached image because of issue in previous step "UpdateAttribute" processor.How to reprocessed failed records again to DB if I have resolved issue in "UpdateAttribute" processor?Is there any way to reprocessed records in sequence ? Let me give you details the problem I am facing is suppose my sql args for 12 was data type string but while converting it in "UpdateAttribute" I have given it to some other data type, so "PutSQL" will throw error and it will not insert record.Now suppose if I have changed "UpdateAttribute" with correct datatype then how I can reprocessed failed records using same job?

Is FetchSFTP reusing an established connection across individual fetches?I have a case where I only want to fetch datafiles if there exists a <datafile>.sha256 checksum file as well. Idea is to ListSFTP with a filter on .sha256, and then do a FetchSFTP for the found filename, and a fetch with .sha256 removed. Will each file fetched with FetchSFTP open and close the SFTP connection, or is NiFi reusing the connection behind the scenes?Ideas on how to track/notify about datafiles with missing checksum files? Those won't be caught, but someone might want to know about them.

How do I use ListFile to process the files in datetime order , I need to process the files in chronological order, oldest file first ?

Hi! Please advise for our case. We have three separate networks that would perform SNMP integration with devices. The main limitation is that NiFi node in one network can't access to devices in another network. SNMP integration scheme is provided in the picture. In our view, "Extract IP’s From Provided Ranges" is performing only on the Master node and then through Remote Process Groups would pass IP ranges to nodes in separated networks and then execute an extraction logic. Is there a way to send the task to particular node group? Thank you!