Cloudera Community
simran_k
user rank
Expert Contributor

multiple listenHttpProcessor on same port in nifi

by Expert Contributor in Support Questions
‎06-02-2017 09:38 AM
‎06-02-2017 09:38 AM
The webhook I am getting data from allows only port 80 to listen to it. I am using Nifi to get events from this webhook and first ListenHttp Processor on port 80 works fine. How do I run more ListenHttp Processors on this same port? is that not possible using diff. base url's? If not, what is the workaround? ... View more
Labels:

Re: Nifi putHDFS processor not working although pu...

by Expert Contributor in Support Questions
‎06-01-2017 11:07 AM
‎06-01-2017 11:07 AM
Thanks. Where exactly can I find nifi-app.log?Turns out I did not add core-site xml and hdfs-site xml in the configuration . I have and seems to work. BUt have a quick question. I have a file each for a JSON object streamed. That sounds like a bad thing to me to have too many small files. Is there a work around for this? ... View more

NIFI stream using ListenHttp Processor creates too...

by Expert Contributor in Support Questions
‎06-01-2017 10:00 AM
‎06-01-2017 10:00 AM
A new file is being created for every JSON object. I believe that's too much and will create too many small files in HDFS (I am using putHDFS processor for that) . Is it alright? Isn't it a bad idea to have too many small files in HDFS? Is there another way to get around this? ... View more
Labels:

Nifi putHDFS processor not working although putFIl...

by Expert Contributor in Support Questions
‎06-01-2017 06:33 AM
‎06-01-2017 06:33 AM
I have a small workflow with a listenProcesser that listens to the webhook and that is connected to a put Hdfs processor. Now the problem is that the mesages keep getting queued rather than writing to HDFS. Where do I check the logs and how do I fix it? I have tried connecting it to putFile and that works. Not sure why putHDFS is not working. In putHDFS, I am writing it in /user/centos directory and so am through putFile, which means user and group is same for the directories used in both putFile and putHDFS. Attached the snapshot: ... View more
Labels:

using NIFI HttpListen Processor to get data from a...

by Expert Contributor in Support Questions
‎05-31-2017 01:42 PM
‎05-31-2017 01:42 PM
1. I am trying to get data from https://support.appsflyer.com/hc/en-us/articles/207034356-Push-APIs-Real-Time-Installation-and-Conversion-Notification-APIs The page say: NOTE: Push API only supports the use of port 80 and 443. I believe I can use NIFI to stream data from a webhook. My question is how to run the HttpListen Processor on port 80 or 443. I tried putting these ports in there and that seems to not work. Secondly, I tried posting to my small workflow in Nifi that has just two components httplistener and putHDFS. It just shows the bytes stuck in the queue and not being written to hdfs. Besides how do I run this on port 80 or 443 as these are the only two ports supported. Are there any other processors I could possibly use? Any help me is appreciated ... View more
Labels:

column names in a hive table

by Expert Contributor in Support Questions
‎04-20-2017 06:54 AM
‎04-20-2017 06:54 AM
Is there a way to change column names in a hive table using first row in the CSV? I know I can skip the first row through skip.header property but I would rather like to set column names using this so as to be able to have right names in the table if the structure of CSV ever changes. Is there a workaround or a proper way to do that? ... View more
Labels:

Re: namenode is not formatted

by Expert Contributor in Support Questions
‎04-03-2017 04:52 AM
‎04-03-2017 04:52 AM
I checked. Unfortunately it is not, and for some reason, there is nothing in namenode and secondarynamenode folder although hdfs has data. Does that mean I can't recover it at all? ... View more

Re: namenode is not formatted

by Expert Contributor in Support Questions
‎04-01-2017 06:53 AM
‎04-01-2017 06:53 AM
@Romil Choksi Also, Maximum size of an xattr: 16384 Apr 1, 11:45:18.670 AM INFO org.apache.hadoop.hdfs.server.common.Storage Lock on /dfs/nn/in_use.lock acquired by nodename 9392@warehouse.swtched.com Apr 1, 11:45:18.672 AM WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem Encountered exception loading fsimage java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) Apr 1, 11:45:18.684 AM INFO org.mortbay.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@warehouse.swtched.com:50070 ... View more

Re: namenode is not formatted

by Expert Contributor in Support Questions
‎04-01-2017 06:50 AM
‎04-01-2017 06:50 AM
when the system went down, namenode on first server went into standby mode I believe . I could use hadoop fs -ls on the second server then. I can not right now. Any clues? ... View more

Re: namenode is not formatted

by Expert Contributor in Support Questions
‎04-01-2017 06:49 AM
‎04-01-2017 06:49 AM
@Romil Choksi I did. I have only 2 nodes in my cluster. One with nn and snn. other once has yarn, hdfs and zookeeper. first one is master node. The entire system went down but the hard disk is safe so there was no data loss for sure. We are trying to bring back the services one by one. If I run this on ther node, it says What is the best way I can get this working? 17/04/01 12:15:29 INFO namenode.NameNode: createNameNode [-bootstrapStandby] 17/04/01 12:15:29 ERROR namenode.NameNode: Failed to start namenode. java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:426) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1504) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) Caused by: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:378) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:421) ... 2 more ... View more
Contact Me
Online
Offline
Last Visited
‎05-04-2018 06:36 AM
Community Statistics
Member Since ‎05-16-2016 08:37 AM
Last Visited ‎05-04-2018 06:36 AM
Posts 270
Kudos received 18