Solved

We have set up the archival process in NiFI with below configuration. # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=10 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=/opt/nifi/data/content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=/nifi-content-viewer/ Please refer attached screenshot there is nothing in archived directory in content repository. Dilip

If nifi is writing to a disk using PutHDFS or PutFile, and it is killed or process goes down, after writing 4 blocks out of 10. After restoring will nifi write from 5th block onwards or rewrite the whole file again creating duplicates?

INSERT OVERWRITE DIRECTORY '/tmp' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE SELECT visit_id, ivm from abcd.xyz WHERE feed_date BETWEEN '2006-04-01' and '2006-05-01' AND ivm IS NOT NULL limit 10; Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [rvchinta] does not have [WRITE] privilege on [/tmp] (state=42000,code=40000) 0: jdbc:hive2://hostname.abc.com:8443> ranger policy /tmp has 777 to public HDP version 2.6.1

I see an option 'Concurrent Tasks' in Scheduling in Nifi version 2.0. Here are my questions regarding Concurrency. 1. I see this only at indivudual processor level. Can this be set at 'Processor Group' level. In short can I parameterize it. 2. How can I arrive at this number. What are all the fators that decide concurrency. E.g. Memory, average data load etc, Other application in the same box like spark.

What would be the recommendation for a 40gbps NiFi cluster? And How would load balancing be handled with clustering? Would 1 IP be given for the cluster, lets say 1 HandleHTTP processor. and then the Network load will be distributed to the other nodes? Thank you!

I am trying to enable kerberos with existing Active Directory KDC. Need to test ldaps url from RHEL Linux

@Rahul Soni Hi Rahul..I wanted to monitor the jvm heap usage in Nifi.So could you please suggest me a way to load flowfile attributes heavily?

Hi, My NiFi is hosted on 3 node cluster, my requirement is to append data at the end of the file, since my NiFi is working on clustered mode how we should make sure that only one node should write data to the file? There should not be any conflict in write operation. Thanks,

We are experiencing duplicate data when our cluster's primary node switches over. Two nodes, m1 and m3, both TailFile processors in them tailing a log but only set to tail the Primary Node. When the Primary Node switches (say from m1 to m3), one TailFile processor stops tailing and the other picks it up (I am assuming it is picking it up at the beginning of the file according to the settings in the TailFile prcoessor) which causes data duplication. I saw there was an option to have it start at the current time instead of the Beginning of File. Would this be a reasonable fix to ensure when the Primary Node switches we dont get duplicate data? We've always had this set at Beginning of File and didn't seem to have this problem before so I am hesitant to change it without a response as to if I undestand this functionality in this situation correctly.

How do I view the flowfile attributes names/values once it processor completes processing and move it to queue. Wanted to be able to see the flow file attribute/name values available, such that they can be used in subsequent processor expressions.

Hi I have observed a confusing pattern over the last few days to do with the amount of data queued on a connection. Every day, at the start of the day, we have about 4 million flow files in a particular queue, with the total data shown on the queue as approx 70 GB. As the day goes on the number of flow files in the queue reduces as the queue is processed, but the total size of the queue rises, to over 100GB (it eventually starts dropping again, though). This behaviour is not what I expect and I can't find anything about it in the user guide. My working theory is that the total might also include archived content (we have retention set to 12 hours) and/or content claims. However I can't find confirmation of this. Is anyone able to shed some light on this behaviour for me? Thanks Richard

I am unable to get TCP nifi to nifi to work. I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile. Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here? Attached are pics of my configs and setup. Thank you!

We have StramSets (STS) intalled using Clodera parcels on CDH 5.6.0. A pipeline is generating test and trying to save to HDFS. Kereberos is enabled. Keytab file is in the Configuration Directory.   But I get the following error:   HADOOPFS_44 - Could not verify the base directory: 'org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: sdc/<HOST>.lab.nordigy.ru@LAB.NORDIGY.RU is not allowed to impersonate devuser@LAB.NORDIGY.RU'   Any one have an idea?

Hi, Is there any limit on number of data records/files NiFi can process within seconds? I need to find a solution where , I read millions of records within 10 seconds and put them in Hadoop. Thanks,

Context I'm working on setting up a Hadoop cluster consisting of multiple VMs (for testing and development purposes) on a local desktop machine. I have a proof of concept working, where I installed Cloudera Manager manually and then setup the cluster. But now I'd like to automate it.   Problem To automate this process, I need to be able to script the installation of Cloudera Manager, so that it works without manual intervention. Using Cloudera Director doesn't seem to be an option for this use case, because we're not running the cluster on any cloud provider (AWS, Azure, etc.). Is there any way to automate the Cloudera Manager installation for a local desktop?

Hi, when i run hdfs dfsadmin -report command. I get the Access denied for user root. Superuser privilege is required error. please help? hdfs dfsadmin -reportConfigured Capacity: 778288271867904 (707.85 TB)Present Capacity: 776586815760931 (706.30 TB)DFS Remaining: 282227055766051 (256.68 TB)DFS Used: 494359759994880 (449.62 TB)DFS Used%: 63.66%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0 -------------------------------------------------report: Access denied for user root. Superuser privilege is required

I am trying to put together a hardware specification for name nodes running in HA mode. That made me have to think about disk allocation for name nodes. I pretty much get it with non-HA. Use one RAID drive and another NFS mount for redundancy. SNN incrementally applies changes in the edit log to the fsimage, etc. But I want to run HA. And I want to use Journal Nodes (JN) and the Quorum Journal Manager (QJM) approach. So, that made me think about this scenario and I was not sure I was getting it right and wanted to ask some gurus for input. Here's what I think... Can you please confirm or correct? I think a scenario type question will help me more easily ask the questions so here goes. Assume a clean install. Primary and failover NNs both have empty fsimage files. Primary starts running and writing changes to all three JN's. As I understand it, the failover NN will be reading all those changes, via the JNs, and applying them to his empty fsimage to prepare it to be 100% complete should

HI All, Thanks a lot to this awesome community. I have a lot of process groups in my nifi ui and some of them are overwhelmed with the data, the success queue is filled up. Is there a way to look what all success queue inside of process queue's are filled up from the main page? Thanks Dhieru

Hi, I am new with NiFi. I installed version 1.1.2 on a CentOS6 virtual machine for testing purposes. Installation went fine and I also secured it with certificates/keys etc via de toolkit. In the authorizers.xml I configured CN=admin as Initial Admin Identity. CN=admin was also used with the toolkit to generate the keys etc. I imported the certifacte in the browser and works fine. As testing purpose I want to assign roles to the admin. I can create a user but same as with the admin I don't know where to add which roles. In the admin guide on the apache nifi website they talk about connecting an ldap but I don't want that. I just want an admin and 1 or 2 users maybe in the application and assign roles. I see the described roles described in the admin guide but where and how do I configure that? The users.xml and authorizations.xml where nicely created. In the users.xml I have: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><tenants> <groups/> <users&gt

Hi everybody, I use Nifi 1.0.0 on AIX server. My ListFile processor gives the same file in two different dataflows. It schedules every 15 seconds. The file O27853044.1135 begins to fill at 11:35 and ends at 11:45. Is it normal that the processor creates a dataflow at 11:42 ? How avoid ListFile processor to create a dataflow before the end of file's update ? Thanks for you help

In my HDP cluster, I install 3 zookeeper-servers on 3 nodes, other nodes I just only install zookeeper-client. When I stand on Zookeeper-Server nodes, I can run zookeeper-client command OK. But when I stand on Zookeeper-Client nodes, I run zookeeper-client and can not connect to zookeeper servers 2017-03-01 23:19:57,076 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@23ab930dWelcome to ZooKeeper!2017-03-01 23:19:57,112 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)JLine support is enabled2017-03-01 23:19:57,205 - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refused at sun.

I currently have one namenode in a 'stopped' state due to a node failure.  I am unable to access any data or services on the cluster, as this was the main namenode.   However, there is a second namenode that I am hoping can be used to recover.  I have been working on the issue in this thread and currently I all hdfs instances started except for the bad namenode.  This seems to have improved the situation as far as node health status but I still can't access any data.   Here is the relevant command and error:   ubuntu@ip-10-0-0-154:~/backup/data1$ hdfs dfs -ls hdfs://10.0.0.154:8020/ls: Operation category READ is not supported in state standby   In the previous thread, I also pointed out that there was the option to enable automatic failure in CM.  I am wondering if that is the best course of action right now.  Any help is greatly appreciated.

A have a nifi cluster with 5 machines, their FQDNs aremycoapmny.nifiui.node01.netmycoapmny.nifiui.node02.net (is the primary node)mycoapmny.nifiui.node03.netmycoapmny.nifiui.node04.netmycoapmny.nifiui.node04.netI am configuring remoteprocessgroup on mycoapmny.nifiui.node02.net to list files in a directory on the same node.but I got the options such as Transport protocol will it be HTTP? or Raw? HTTP Proxy Server hostnames of each nodes in clusterHttp Proxy user --> is my ad group username? required here?Local netwrok interface --> bond0? since this is the name I got doing after ifconfig on the primary nodeHttp proxy server port --> which port ? 9091 since this is configured for my nifi uiHttp proxy password --> is my ad group password required here?

To reset ZKFC we have to stop both the ZKFC processes and run the following command as the hdfs super user : hdfs zkfc –formatZK Is this same for Kerberos enabled environment? Do we need to stop all HDFS Services and failover control? Where do we need to run this command in NN or anywhere? HDP Version - 2.4.3 Thanks in advance.

I have a cluster Nifi with 3 nodes: node-1, node-2, node-3. When I run a job on cluster, there is some errors and node-2 disconnected to cluster. Then I want to go UI admin of node-1 or node-3 to stop this job. But I can not stop it. It notices: Cluster is unable to service request to change flow: Node node-2:8092 is currently disconnected