Member since
09-29-2016
54
Posts
9
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2847 | 07-03-2017 09:12 PM | |
2199 | 05-22-2017 08:19 PM |
05-22-2017
08:15 PM
@mqureshi I was sending "inferred.avro.schema" as an attribute and the input content was set to json
... View more
05-22-2017
12:50 AM
@Deepak B S , I did not find a NiFi fix for this error, however in my case I discovered that it was due to the server getting overloaded due to a high amount of messages and because of this some messages were getting backlogged and by the time they would reach the HandleHTTPRequest processor they had experied. I raised the number of concurrent tasks in the processor on NiFi side and used a load balancer distribution rule to balance the load that was getting into NiFi and that seemed to fix the issue for me. You can try to raise the number of concurrent tasks in your processor and on your flow if the error your getting is due to a load issue that can help.
... View more
05-19-2017
09:11 PM
1 Kudo
Hello, I am trying to create a flow in NiFi that takes a valid json file and puts it directly into a hive table using the PutHiveStreaming processor. My json looks something like the following: {
"Raw_Json": {
"SystemInfo": {
"Id": "a string ID",
"TM": null,
"CountID": "a string ID",
"Topic": null,
"AccountID": "some number",
"StationID": "some number",
"STime": "some Timestamp",
"ETime": "some Timestamp"
},
"Profile": {
"ID": "ID number",
"ProductID": "Some Number",
"City": "City Name",
"State": "State Name",
"Number": "XXX-XXX-XXXX",
"ExtNumber": null,
"Unit": null,
"Name": "Person Name",
"Service": "Purchase",
"AddrID": "00000000",
"Products": {
"Product": [{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
},
{
"Code": "CODE",
"Description": "some description"
}]
}
},
"Total": {
"Amount": "some amount",
"Delivery": "some address",
"Estimate": "some amount",
"Tax": null,
"Delivery_Type": null
}
},
"partition_date":"2017-05-19"
}
I am getting the json, using the InferAvroSchema processor and from there converting the json to avro format by using the inferred avro schema and sending it into the PutHiveStreaming processor. My Flow looks something like this: The main goal is that I want all of the "Raw_Json" column to be dumped into one column in the hive table and the table will be partitioned by the "partition_date" column which will be the second column of the table. The problem is that for some reason NiFi is having problems inferring the nested json from the "Raw_Json" column and is dumping it like Null on the table as shown below: Does anyone know how could I make NiFi read the entire nested Json of the "Raw_Json" column as a string column and send it to the hive table? How could I create my own avro schema for it to do this? My main goal would be that the Raw_Json can be read as a string column. Any insight or ideas on how to fix this issue would be greatly appreciated!
... View more
Labels:
- Labels:
-
Apache NiFi
04-24-2017
08:50 PM
@Wynner, I was able to rejoin the cluster by moving the data in the repositories and restarting the node like you said. Thanks for the help!
... View more
04-24-2017
03:20 PM
@Wynner, all of the disk partitions are in less than 10% use and I just tested disconnecting and connecting the other nodes which use the exact same zookeeper connection string as the node that is giving problems and they rejoin the cluster with no issues or errors. Could something have caused a file to get corrupted on this node and make it be causing the error?
... View more
04-24-2017
02:58 PM
@Wynner, for this cluster I do not use embedded zookeeper as we have our own zookeeper cluster and that one is the one that is used to manage the NiFi cluster. However I checked zookeeper and everything seems fine on it. I will check the NiFi repositories and the space on the disk partitions.
... View more
04-24-2017
02:48 PM
Hello, I run a NiFi cluster with 3 nodes using NiFi-1.1.0 version. The cluster has been running with no issues for the last couple of months, however I checked it today and one of the nodes had suddenly disconnected and it won't join back to the cluster. I checked the logs of the node and the following error keeps appearing non stop in the logs: ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
2017-04-24 10:31:34,694 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
I had never encounter this issue before and wanted to know if someone could give me an idea of what could be causing it or how it could be fixed. I am a bit confused as no changes have been made to the configurations of the node or the cluster whatsoever and the other two nodes are working completely fine. Any insight on this issue would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
04-21-2017
04:12 PM
thanks a lot for the help! I will try this.
... View more
04-21-2017
04:07 PM
@Matt Burgess thanks for the answer and the help. I wanted to ask if it is possible to run the NiFi 1.1.0 version of HDF instead of the Apache one independently without using the full HDF?
... View more
04-21-2017
02:14 PM
@milind pandit the hive nar file my version of nifi has is nifi-hive-nar-1.1.0.nar
... View more