Q&A

Hi, Our OU in the LDAP connection has changed and we are getting invalid ID,PWD errors when trying to log in to NiFi, I think this is expected. But if i change the old ou=Users and Groups,ou=EU,ou=same,ou=Organizations,dc=comp,dc=com to new ou=Users and Groups,ou=USA,ou=same,ou=Organizations,dc=comp,dc=com do i need to restart nifi.?? is there a way that this could be avoided .?? that is instead of giving the whole string can i just give the top level ou like ou=Users and Groups  

CDH 5.13.1Redhat 6.9We wish to change the number of replications from the default of 3 copies to 2 on one particular folder in hdfs.After running this on one cluster:$ hdfs dfs -setrep -R 2 /backupsand then doing a$ hdfs dfs -du /we saw that it freed the blocks very quickly and the output of fsck shows no "Over-replicated blocks":Status: HEALTHY Total size:    149514016589 B Total dirs:    27440 Total files:    128746 Total symlinks:        0 Total blocks (validated):    126355 (avg. block size 1183285 B) Minimally replicated blocks:    126355 (100.0 %) Over-replicated blocks:    0 (0.0 %) Under-replicated blocks:    0 (0.0 %) Mis-replicated blocks:        0 (0.0 %) Default replication factor:    3 Average block replication:    2.3367577&nb

I have two flow file: 1) $.category = sport 2) $.temp_array = "music" , "sport" , "traveling" I just want to check $.category is exist in $.temp_array or not using in expression language. Ex. ${category:in(${temp_array})} But it doesn't work with array or (variable). I have used this login on RouteOnAttribute processor. Instead of above example if i write like this: ${category:in("music","sport","traveling")} then ii will give you perfect output But this is static data. Instead of this i want to use dynamic. Please help.

Hi, I work on a NiFi flow getting data from a FTP server, sending this data to HDFS. I have to add to this flow the ability of checking data integrity between files fetched on the FTP, and files written on HDFS. To do this, I use the HashContent NiFi processor with MD5 algorithm to compute MD5 hash of flowfiles from start and end of the flow (I can get MD5 hash of each file on FTP, second MD5 is computed after a PutHDFS retrieving files having been written). Finally, I compare both values and consider data integrity is OK if they are equal. Do you have a general advice about this practice? Is this kind of check really useful with NiFi? Thanks, Benjamin

Hi, Is there anyway that we can run Remote Processor Group with multi threading (concurrent tasks) in NiFi Site-2-Site.? Regards, Sai

I have a Hbase cluster of 3 nodes. Below are the hostnames and daemons running on that machines. hbase2 - Hmaster,RegionServer, Datanode and Nodemanager hbase 1 - RegionServer,Namenode,Datanode, ResourceManager, NodeManager hbase 3 - RegionServer, Datanode and Nodemanager The moment I enter the run start-hbase.sh script. It throws the below given error2018-06-13 14:56:52,455 INFO [hbase2:60000.activeMasterManager] retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hbase2/10.94.1.119:8020 after 11 fail over attempts. Trying to fail over after sleeping for 17938ms. java.net.ConnectException: Call From hbase2/10.94.1.119 to hbase2:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused. Hmaster is not running. RegionServers are working fine

Hi, How to move the flowfiles from one queue/relationship to another one through NiFi Web Interface. Thanks,,

HI All, In our cluster NIFI is SSL enabled. Ranger is not SSL enabled. Both NIFI and Ranger are integrated with AD/LDAP. Before enabling NIFI plugin in Ranger, our AD/LDAP users are able to see NIFI UI. But after enabling NIFI plugin in Ranger, our AD/LDAP users are not able to see NIFI UI. We are getting following message on NIFI screen: Insufficient Permissions Untrusted proxy CN=*.test.com, OU=NIFI nifi-user.log shows Authentication success but Untrusted proxy error as follows: 2018-06-07 07:00:13,447 INFO [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://usdf24v0075.test.com:9091/nifi-api/flow/current-user (source ip: 10.23.118.51)2018-06-07 07:00:13,449 INFO [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for test-user2018-06-07 07:00:13,612 INFO [NiFi Web Server-18] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<test-user><CN=*.test.com, OU=NIFI>) GET https://usd

@Matt Burgess I have nifi job to capture cdc from MySQL DB.I am facing one issue with my nifi job in case of any failure. Suppose take an example if there is a failure while inserting record to DB using "PutSQL" processor in attached image because of issue in previous step "UpdateAttribute" processor.How to reprocessed failed records again to DB if I have resolved issue in "UpdateAttribute" processor?Is there any way to reprocessed records in sequence ? Let me give you details the problem I am facing is suppose my sql args for 12 was data type string but while converting it in "UpdateAttribute" I have given it to some other data type, so "PutSQL" will throw error and it will not insert record.Now suppose if I have changed "UpdateAttribute" with correct datatype then how I can reprocessed failed records using same job?

Is FetchSFTP reusing an established connection across individual fetches?I have a case where I only want to fetch datafiles if there exists a <datafile>.sha256 checksum file as well. Idea is to ListSFTP with a filter on .sha256, and then do a FetchSFTP for the found filename, and a fetch with .sha256 removed. Will each file fetched with FetchSFTP open and close the SFTP connection, or is NiFi reusing the connection behind the scenes?Ideas on how to track/notify about datafiles with missing checksum files? Those won't be caught, but someone might want to know about them.

How do I use ListFile to process the files in datetime order , I need to process the files in chronological order, oldest file first ?

Hi! Please advise for our case. We have three separate networks that would perform SNMP integration with devices. The main limitation is that NiFi node in one network can't access to devices in another network. SNMP integration scheme is provided in the picture. In our view, "Extract IP’s From Provided Ranges" is performing only on the Master node and then through Remote Process Groups would pass IP ranges to nodes in separated networks and then execute an extraction logic. Is there a way to send the task to particular node group? Thank you!

We have set up the archival process in NiFI with below configuration. # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=10 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=/opt/nifi/data/content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=/nifi-content-viewer/ Please refer attached screenshot there is nothing in archived directory in content repository. Dilip

If nifi is writing to a disk using PutHDFS or PutFile, and it is killed or process goes down, after writing 4 blocks out of 10. After restoring will nifi write from 5th block onwards or rewrite the whole file again creating duplicates?

INSERT OVERWRITE DIRECTORY '/tmp' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE SELECT visit_id, ivm from abcd.xyz WHERE feed_date BETWEEN '2006-04-01' and '2006-05-01' AND ivm IS NOT NULL limit 10; Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [rvchinta] does not have [WRITE] privilege on [/tmp] (state=42000,code=40000) 0: jdbc:hive2://hostname.abc.com:8443> ranger policy /tmp has 777 to public HDP version 2.6.1

I see an option 'Concurrent Tasks' in Scheduling in Nifi version 2.0. Here are my questions regarding Concurrency. 1. I see this only at indivudual processor level. Can this be set at 'Processor Group' level. In short can I parameterize it. 2. How can I arrive at this number. What are all the fators that decide concurrency. E.g. Memory, average data load etc, Other application in the same box like spark.

What would be the recommendation for a 40gbps NiFi cluster? And How would load balancing be handled with clustering? Would 1 IP be given for the cluster, lets say 1 HandleHTTP processor. and then the Network load will be distributed to the other nodes? Thank you!

I am trying to enable kerberos with existing Active Directory KDC. Need to test ldaps url from RHEL Linux

@Rahul Soni Hi Rahul..I wanted to monitor the jvm heap usage in Nifi.So could you please suggest me a way to load flowfile attributes heavily?

Hi, My NiFi is hosted on 3 node cluster, my requirement is to append data at the end of the file, since my NiFi is working on clustered mode how we should make sure that only one node should write data to the file? There should not be any conflict in write operation. Thanks,

We are experiencing duplicate data when our cluster's primary node switches over. Two nodes, m1 and m3, both TailFile processors in them tailing a log but only set to tail the Primary Node. When the Primary Node switches (say from m1 to m3), one TailFile processor stops tailing and the other picks it up (I am assuming it is picking it up at the beginning of the file according to the settings in the TailFile prcoessor) which causes data duplication. I saw there was an option to have it start at the current time instead of the Beginning of File. Would this be a reasonable fix to ensure when the Primary Node switches we dont get duplicate data? We've always had this set at Beginning of File and didn't seem to have this problem before so I am hesitant to change it without a response as to if I undestand this functionality in this situation correctly.

How do I view the flowfile attributes names/values once it processor completes processing and move it to queue. Wanted to be able to see the flow file attribute/name values available, such that they can be used in subsequent processor expressions.

Hi I have observed a confusing pattern over the last few days to do with the amount of data queued on a connection. Every day, at the start of the day, we have about 4 million flow files in a particular queue, with the total data shown on the queue as approx 70 GB. As the day goes on the number of flow files in the queue reduces as the queue is processed, but the total size of the queue rises, to over 100GB (it eventually starts dropping again, though). This behaviour is not what I expect and I can't find anything about it in the user guide. My working theory is that the total might also include archived content (we have retention set to 12 hours) and/or content claims. However I can't find confirmation of this. Is anyone able to shed some light on this behaviour for me? Thanks Richard

I am unable to get TCP nifi to nifi to work. I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile. Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here? Attached are pics of my configs and setup. Thank you!

We have StramSets (STS) intalled using Clodera parcels on CDH 5.6.0. A pipeline is generating test and trying to save to HDFS. Kereberos is enabled. Keytab file is in the Configuration Directory.   But I get the following error:   HADOOPFS_44 - Could not verify the base directory: 'org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: sdc/<HOST>.lab.nordigy.ru@LAB.NORDIGY.RU is not allowed to impersonate devuser@LAB.NORDIGY.RU'   Any one have an idea?