Q&A

Hi, Is there any limit on number of data records/files NiFi can process within seconds? I need to find a solution where , I read millions of records within 10 seconds and put them in Hadoop. Thanks,

Context I'm working on setting up a Hadoop cluster consisting of multiple VMs (for testing and development purposes) on a local desktop machine. I have a proof of concept working, where I installed Cloudera Manager manually and then setup the cluster. But now I'd like to automate it.   Problem To automate this process, I need to be able to script the installation of Cloudera Manager, so that it works without manual intervention. Using Cloudera Director doesn't seem to be an option for this use case, because we're not running the cluster on any cloud provider (AWS, Azure, etc.). Is there any way to automate the Cloudera Manager installation for a local desktop?

Hi, when i run hdfs dfsadmin -report command. I get the Access denied for user root. Superuser privilege is required error. please help? hdfs dfsadmin -reportConfigured Capacity: 778288271867904 (707.85 TB)Present Capacity: 776586815760931 (706.30 TB)DFS Remaining: 282227055766051 (256.68 TB)DFS Used: 494359759994880 (449.62 TB)DFS Used%: 63.66%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0 -------------------------------------------------report: Access denied for user root. Superuser privilege is required

I am trying to put together a hardware specification for name nodes running in HA mode. That made me have to think about disk allocation for name nodes. I pretty much get it with non-HA. Use one RAID drive and another NFS mount for redundancy. SNN incrementally applies changes in the edit log to the fsimage, etc. But I want to run HA. And I want to use Journal Nodes (JN) and the Quorum Journal Manager (QJM) approach. So, that made me think about this scenario and I was not sure I was getting it right and wanted to ask some gurus for input. Here's what I think... Can you please confirm or correct? I think a scenario type question will help me more easily ask the questions so here goes. Assume a clean install. Primary and failover NNs both have empty fsimage files. Primary starts running and writing changes to all three JN's. As I understand it, the failover NN will be reading all those changes, via the JNs, and applying them to his empty fsimage to prepare it to be 100% complete should

HI All, Thanks a lot to this awesome community. I have a lot of process groups in my nifi ui and some of them are overwhelmed with the data, the success queue is filled up. Is there a way to look what all success queue inside of process queue's are filled up from the main page? Thanks Dhieru

Hi, I am new with NiFi. I installed version 1.1.2 on a CentOS6 virtual machine for testing purposes. Installation went fine and I also secured it with certificates/keys etc via de toolkit. In the authorizers.xml I configured CN=admin as Initial Admin Identity. CN=admin was also used with the toolkit to generate the keys etc. I imported the certifacte in the browser and works fine. As testing purpose I want to assign roles to the admin. I can create a user but same as with the admin I don't know where to add which roles. In the admin guide on the apache nifi website they talk about connecting an ldap but I don't want that. I just want an admin and 1 or 2 users maybe in the application and assign roles. I see the described roles described in the admin guide but where and how do I configure that? The users.xml and authorizations.xml where nicely created. In the users.xml I have: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><tenants> <groups/> <users&gt

Hi everybody, I use Nifi 1.0.0 on AIX server. My ListFile processor gives the same file in two different dataflows. It schedules every 15 seconds. The file O27853044.1135 begins to fill at 11:35 and ends at 11:45. Is it normal that the processor creates a dataflow at 11:42 ? How avoid ListFile processor to create a dataflow before the end of file's update ? Thanks for you help

In my HDP cluster, I install 3 zookeeper-servers on 3 nodes, other nodes I just only install zookeeper-client. When I stand on Zookeeper-Server nodes, I can run zookeeper-client command OK. But when I stand on Zookeeper-Client nodes, I run zookeeper-client and can not connect to zookeeper servers 2017-03-01 23:19:57,076 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@23ab930dWelcome to ZooKeeper!2017-03-01 23:19:57,112 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)JLine support is enabled2017-03-01 23:19:57,205 - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refused at sun.

I currently have one namenode in a 'stopped' state due to a node failure.  I am unable to access any data or services on the cluster, as this was the main namenode.   However, there is a second namenode that I am hoping can be used to recover.  I have been working on the issue in this thread and currently I all hdfs instances started except for the bad namenode.  This seems to have improved the situation as far as node health status but I still can't access any data.   Here is the relevant command and error:   ubuntu@ip-10-0-0-154:~/backup/data1$ hdfs dfs -ls hdfs://10.0.0.154:8020/ls: Operation category READ is not supported in state standby   In the previous thread, I also pointed out that there was the option to enable automatic failure in CM.  I am wondering if that is the best course of action right now.  Any help is greatly appreciated.

A have a nifi cluster with 5 machines, their FQDNs aremycoapmny.nifiui.node01.netmycoapmny.nifiui.node02.net (is the primary node)mycoapmny.nifiui.node03.netmycoapmny.nifiui.node04.netmycoapmny.nifiui.node04.netI am configuring remoteprocessgroup on mycoapmny.nifiui.node02.net to list files in a directory on the same node.but I got the options such as Transport protocol will it be HTTP? or Raw? HTTP Proxy Server hostnames of each nodes in clusterHttp Proxy user --> is my ad group username? required here?Local netwrok interface --> bond0? since this is the name I got doing after ifconfig on the primary nodeHttp proxy server port --> which port ? 9091 since this is configured for my nifi uiHttp proxy password --> is my ad group password required here?

To reset ZKFC we have to stop both the ZKFC processes and run the following command as the hdfs super user : hdfs zkfc –formatZK Is this same for Kerberos enabled environment? Do we need to stop all HDFS Services and failover control? Where do we need to run this command in NN or anywhere? HDP Version - 2.4.3 Thanks in advance.

I have a cluster Nifi with 3 nodes: node-1, node-2, node-3. When I run a job on cluster, there is some errors and node-2 disconnected to cluster. Then I want to go UI admin of node-1 or node-3 to stop this job. But I can not stop it. It notices: Cluster is unable to service request to change flow: Node node-2:8092 is currently disconnected

Hi , I have stream data (GetHDFS will be running continuosly ) which contains number of lines. e.g: <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. A stream of above lines of data will be in file I have to extract text from above message [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] I tried using a extract text processor and used custom property extract: ([hdfs.*log"]). I tried the above in java regex evaluator, it shows correct text extracted. but when I run the flow, output gets the complete text. expected: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] actual : <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. Please help me to corr

Hello, Just wondering if there is a way to help developers identify in which NiFi UI canvas they are on - Dev, Test, Prod; this is to help prevent developers from accidentally making changes in the wrong environments if they have multiple environments' canvases open; ideally a place like the top left corner where the NiFi logo is would be a good place to identify the environment; I checked the nifi properties file and other than the banner (which is a very small size font) I didn't find any other property where you can have a custom text/image/logo on the canvas. Also, I thought about the Label, but you've to add that inside each process group for it to be useful. Any ideas on how this can be done? Thanks.

Hi All, I want to fetch the data that is stored in HDFS using FetchHDFS processor . The folder structure to store our data is like /MajorData/Location/Year/Month/Day/file1.txt (/MajorData/Location/2017/01/01/file1.txt) As the day changes the folder structure will change to /MajorData/Location/2017/01/02/file2.txt How can I write a Nifi expression which will traverse through all the folders, fetch the data in NiFi?

I have HDP2.6, non-Kerberos cluster. I installed Ranger KMS. The service fails after startup. Can anyone direct me if I am missing anything? 2017-07-13 00:03:21,648 INFO log - ------------------ Ranger KMSWebApp--------------------- 2017-07-13 00:03:21,648 INFO log - provider string = dbks://http@localhost:9292/kms 2017-07-13 00:03:21,649 INFO log - URI = dbks://http@localhost:9292/kms scheme = dbks 2017-07-13 00:03:21,649 INFO log - kmsconf size= 225 kms classname=org.apache.hadoop.conf.Configuration 2017-07-13 00:03:21,649 INFO log - ----------------Instantiating key provider --------------- 2017-07-13 00:03:22,001 INFO RangerKeyStoreProvider - Credential keystore password not applied for KMS; clear text password shall be applicable 2017-07-13 00:03:22,002 ERROR RangerKMSDB - DB Flavor could not be determined 2017-07-13 00:03:23,133 INFO RangerKMSDB - Connected to DB : true 2017-07-13 00:03:23,136 INFO RangerMasterKey - Generating Master Key 2017-07-13 00:03:23,147 INFO AuditProvider

I have a flow in NiFi which splits a file into individual lines, inserts those lines into a database and after those have been inserted updates a control table. The control table only updates after every line has been inserted. To achieve this, the fragment.index is compared to fragment.count - if these are equal, then I know that every line has been processed and we can move on to updating the control table. However recently some of our files failed to update the control table. I have outputted the Attributes of the flow files to disk, and it shows something that confuses me: the number of flow files that comes out of the split text processor is 66430, which matches the number of lines in the file. However, the fragment.count attribute is 66443. Does anybody know why the fragment index would be incorrect, and how I can fix this?

I have a workflow that routes on attribute depending on a type, but it sets a RouteOnAttribute.Route param on a flow that has all of the filters, I don't want that, how can I prevent the processor to do this? Thanks

hi all, I am very new to nifi, i have try to adopt backpressure to control the throughput and i did a test as bleow, setting the highlight link backpressure threshold as screen cap. however, seems the 37.05MB data pass from "ConvertCSVToAvro" to "ConvertAvroToJSON" quickly , feels like backpressure doesn't work. I have read https://community.hortonworks.com/articles/9785/nifihdf-dataflow-optimization-part-2-of-2.html and flow the setting. Would anyone share me tips of using controlrate and backpressure to prevent overwhelm to the system? currently if the input data is large than 500M (CSV), it has OOM error.

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!! I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx). This happening for the MapReduce jobs only!!! When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!   An existing userid (user1) works fine! It seems to me that the issue is permissions-related.   Here are the HDFS permissions of the userids: [root@master ~]# hdfs dfs -ls /tmp/logsFound 5 itemsdrwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfsdrwxrwxrwx&nbs

I'm in the America/New_York timezone and my servers run ntp, and they show the right time. The header of NiFi running on those servers is off by one hour, eg system time says 11:00 and NiFi says 10:00. This question is very similar to https://community.hortonworks.com/questions/76724/change-timezone-in-nifi.html but I've tried that solution and no joy. This feels like a daylight savings problem, as the gui says 10:00:00 EST, and really that should be EDT since at time of writing it is summer. I'm using Oracle Java 1.8.0_71. Any ideas?

1. I am trying to get data from https://support.appsflyer.com/hc/en-us/articles/207034356-Push-APIs-Real-Time-Installation-and-Conversion-Notification-APIs The page say: NOTE: Push API only supports the use of port 80 and 443. I believe I can use NIFI to stream data from a webhook. My question is how to run the HttpListen Processor on port 80 or 443. I tried putting these ports in there and that seems to not work. Secondly, I tried posting to my small workflow in Nifi that has just two components httplistener and putHDFS. It just shows the bytes stuck in the queue and not being written to hdfs. Besides how do I run this on port 80 or 443 as these are the only two ports supported. Are there any other processors I could possibly use? Any help me is appreciated

I want to give all Read access for UI and all new crated processes and process groups by default from coding for each new user. Instead of explicitly go and give the view rights. I think we can do this by making changes in java files. But I am not able to find out the exact flow from java files. Expecting a solution for this. Thanks in advance.

I am running a ListHDFS processor pointing to a directory on hdfs on a timer driven schedule set to execute once per hour. After making sure the state is clear on the processor, I run it and see that it creates a flow file for all but 1 file in the directory. There are 5 files in the directory, and only 4 flowfiles are created. If I add more files and clear the state and attempt to run again, the pattern repeats, always one less flowfile is create, so one file is missed. It is not the same file that is missed with each run. Why is the processor missing 1 file each time? Is this by design? This is in HDF 2.1.0.1 and Apache NiFi - Version 1.1.0.2.1.0.1-1

I am working with nifi cluster. As in tutorial it is suggestion to use remote process group with cluster for load balancing. But to use Remote process group, we have to provide url of other nifi instance which is hardcoded. So what will happen if that nifi instance goes down in between of workflow execution. Is remote process call is working in that situation as load balancing concept is implemented in Remote process call.