Member since
12-27-2018
25
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1335 | 01-16-2019 07:21 PM |
02-27-2019
05:45 PM
1 Kudo
I receive JSON data from kafka with from_json() method. It expects schema from me. My JSON structure like this;
{ "Items": { "key1": [ { "id": "", "name": "", "val": "" } ], "key2": [ { "id": "", "name": "", "val": "" } ], "key3": [ { "id": "", "name": "", "val": "" } ] }
}
Key1, Key2, Key3 are dynamic. So, they may be changed. For example, another json is;
{ "Items": { "hortoworks": [ { "id": "", "name": "", "val": "" } ], "community": [ { "id": "", "name": "", "val": "" } ], "question": [ { "id": "", "name": "", "val": "" } ] }
}
These key names are unknown. But "id-name-val" fields inside these keys are the same. I must define a json schema for read data from Kafka in Spark Structured Streaming. How can I do this?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
02-20-2019
04:44 PM
Nifi has HBase 1.1.2 client version. Can I use HBase 2.x version this HBase controller service?
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache NiFi
02-17-2019
08:08 PM
I have disabled Hyper-V. It worked. Thank you so much.
... View more
02-16-2019
11:54 AM
I have donwloaded HDP 3.0.1, and then installed in Virtualbox(its version greater than 5.1). But I am getting this error. Virtualization is enabled in BIOS. Also, Hyper -v was installed. What I will do? Note: I run Virtualbox as Adminastor.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
01-31-2019
10:07 PM
@David Miller You are right. But s2s etc. not suitable for my scenario. There are many reasons.
... View more
01-31-2019
06:30 PM
Yes, I am getting nodeAddress and its status from "rest-api/controller/cluster" for create http endpoint. I will send flowFile to other active nodes after i prepare URL. For example; Result of "rest-api/controller/cluster"; {
"cluster": {
"nodes": [
{
"nodeId": "node1Id" ,
"address": "node1Address",
"apiPort": 9999,
"status": "CONNECTED",
"heartbeat": "value",
"connectionRequested": "value",
"roles": [],
"activeThreadCount": 0,
"queued": "value",
"events": [{
"timestamp": "value",
"category": "value",
"message": "value"
}],
"nodeStartTime": "value"
},
{
"nodeId": "node2Id" ,
"address": "node2Address",
"apiPort": 9999,
"status": "CONNECTED",
"heartbeat": "value",
"connectionRequested": "value",
"roles": ["PRIMARY"],
"activeThreadCount": 0,
"queued": "value",
"events": [{
"timestamp": "value",
"category": "value",
"message": "value"
}],
"nodeStartTime": "value"
}
],
"generated": "value"
}
}<br> I get node1Address and node2Address(if they CONNECTED) values from this json. Then, I am creating URL like this; http://node1Address:9999/contentListener Then, I will send post request to listenHttp(running on each node) using postHttp with this URL. ListenHttp(running on the each node) will listen to the this path and retrieve the data.
... View more
01-30-2019
09:34 PM
@David Miller "But it sounds like you are trying to do something that would be better served by just running a completely independent flow on each node." -- Yeah, that's exactly what I want to do. So I'm reviewing all possible scenarios. Thank you.
... View more
01-30-2019
09:10 PM
@Matt Clarke I will look for answers to these questions.. Then I'll re-edit this question. Thank you so much.
... View more
01-30-2019
09:03 PM
@Matt Clarke Yes, I know that some data stuck in connection queues. But I have to use postHttp(primary node) and listenHttp for distribute data to cluster. Because, I want to send the same FlowFile to same node. For this, "Partition By Attribute" may be used. Even, its great for this scenario. But, if any node downs, some data will be waiting in the unavailable queue. But i don't want this. Because, i will lose data due to these data that waiting in the unavailable queue. Also, Roun-roind is available. But, it doesn't same data to same node. For this, I will use postHttp and listenHttp for distribute data to cluster. Thank you
... View more
01-30-2019
06:40 PM
I will send request to ListenHttp(running on each node) using PostHTTP(running on only primary node). For this, i am getting nodes address from nifi rest-api(/controller/cluster). And i prepare URL like this; http://nodeAddress:port/contentListener Can I successfully send request to ListenHttp processors in each node with PostHttp using this URL?
... View more
Labels:
- Labels:
-
Apache NiFi