Member since
05-16-2016
270
Posts
18
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1722 | 07-23-2016 11:36 AM | |
3065 | 07-23-2016 11:35 AM | |
1574 | 06-05-2016 10:41 AM | |
1166 | 06-05-2016 10:37 AM |
06-02-2017
09:38 AM
The webhook I am getting data from allows only port 80 to listen to it. I am using Nifi to get events from this webhook and first ListenHttp Processor on port 80 works fine. How do I run more ListenHttp Processors on this same port? is that not possible using diff. base url's? If not, what is the workaround?
... View more
Labels:
- Labels:
-
Apache NiFi
06-01-2017
11:07 AM
Thanks. Where exactly can I find nifi-app.log?Turns out I did not add core-site xml and hdfs-site xml in the configuration . I have and seems to work. BUt have a quick question. I have a file each for a JSON object streamed. That sounds like a bad thing to me to have too many small files. Is there a work around for this?
... View more
06-01-2017
10:00 AM
A new file is being created for every JSON object. I believe that's too much and will create too many small files in HDFS (I am using putHDFS processor for that) . Is it alright? Isn't it a bad idea to have too many small files in HDFS? Is there another way to get around this?
... View more
Labels:
- Labels:
-
Apache NiFi
06-01-2017
06:33 AM
I have a small workflow with a listenProcesser that listens to the webhook and that is connected to a put Hdfs processor. Now the problem is that the mesages keep getting queued rather than writing to HDFS. Where do I check the logs and how do I fix it? I have tried connecting it to putFile and that works. Not sure why putHDFS is not working. In putHDFS, I am writing it in /user/centos directory and so am through putFile, which means user and group is same for the directories used in both putFile and putHDFS. Attached the snapshot:
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi
05-31-2017
01:42 PM
1. I am trying to get data from https://support.appsflyer.com/hc/en-us/articles/207034356-Push-APIs-Real-Time-Installation-and-Conversion-Notification-APIs The page say: NOTE: Push API only supports the use of port 80 and 443. I believe I can use NIFI to stream data from a webhook. My question is how to run the HttpListen Processor on port 80 or 443. I tried putting these ports in there and that seems to not work. Secondly, I tried posting to my small workflow in Nifi that has just two components httplistener and putHDFS. It just shows the bytes stuck in the queue and not being written to hdfs. Besides how do I run this on port 80 or 443 as these are the only two ports supported. Are there any other processors I could possibly use? Any help me is appreciated
... View more
Labels:
- Labels:
-
Apache NiFi
04-20-2017
06:54 AM
Is there a way to change column names in a hive table using first row in the CSV? I know I can skip the first row through skip.header property but I would rather like to set column names using this so as to be able to have right names in the table if the structure of CSV ever changes. Is there a workaround or a proper way to do that?
... View more
Labels:
- Labels:
-
Apache Hive
04-03-2017
04:52 AM
I checked. Unfortunately it is not, and for some reason, there is nothing in namenode and secondarynamenode folder although hdfs has data. Does that mean I can't recover it at all?
... View more
04-01-2017
06:53 AM
@Romil Choksi Also, Maximum size of an xattr: 16384
Apr 1, 11:45:18.670 AM INFO org.apache.hadoop.hdfs.server.common.Storage
Lock on /dfs/nn/in_use.lock acquired by nodename 9392@warehouse.swtched.com
Apr 1, 11:45:18.672 AM WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem
Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
Apr 1, 11:45:18.684 AM INFO org.mortbay.log
Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@warehouse.swtched.com:50070
... View more
04-01-2017
06:50 AM
when the system went down, namenode on first server went into standby mode I believe . I could use hadoop fs -ls on the second server then. I can not right now. Any clues?
... View more
04-01-2017
06:49 AM
@Romil Choksi I did. I have only 2 nodes in my cluster. One with nn and snn. other once has yarn, hdfs and zookeeper. first one is master node. The entire system went down but the hard disk is safe so there was no data loss for sure. We are trying to bring back the services one by one. If I run this on ther node, it says What is the best way I can get this working? 17/04/01 12:15:29 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
17/04/01 12:15:29 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.
at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:426)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1504)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.
at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:378)
at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:421)
... 2 more
... View more