Member since
12-13-2016
72
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6637 | 12-27-2017 05:06 AM |
11-01-2020
06:47 AM
Nope @Shelton . Its not kerberized one normal sasl security protocol with sasl.mechanishm plain text only. But i get this error look like this i have tried new nifi version also; 2020-10-31 23:36:16,762 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-derfsdfdsf-2, groupId=derfsdfdsf] Connection to node -1 (xxxx:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. ----- 2020-10-30 05:41:18,794 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.-
... View more
10-30-2020
02:01 AM
Hi, I'm trying to access kafka broker from nifi using consume_kafka_record_2.0 and kafka has been configured sasl_ssl plaintext manner data needs to consume; While connection on this we facing below stacktrace issue, can you please help me out from this issue; Processor configuration; 2020-10-30 08:52:38,470 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:39,427 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:40,331 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:41,235 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:42,441 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:43,597 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:44,501 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:45,605 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:46,520 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:47,526 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials. 2020-10-30 08:52:48,536 WARN [Timer-Driven Process Thread-8] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-20, groupId=devtest_grp11111] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
09-07-2020
06:37 AM
Hi, I'm referring this below article; https://community.cloudera.com/t5/Community-Articles/Create-Dynamic-Partitions-based-on-FlowFile-Content-Convert/ta-p/248367 I'm trying to create pipeline in nifi while data coming realtime streaming based say some example kafka, while data put in hdfs in partitioned location, it may ended be with many small files at the same while querying im facing performance lag issue; can you please give some apporaches to resolve small files issue in nifi itself with orc file format;
... View more
Labels:
05-03-2020
11:51 PM
I want to split and transfer the json data in NiFi, Here is my json structure look like this; I want to split json by id1,id2 array of json transfer to respective processor group say example processor_group a,b. I tried with evaluate json path $.id1,$.id2 but i didn't get exact solution. Can you please help me out from this issue; { "id1": [{ "u_name": "aa" }, { "addr": "bb" }], "id2": [{ "u_name": "aa" }, { "addr": "bb" }] }
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
03-04-2020
08:30 PM
@MattWho Thanks for the reply. Sure here i have attached my existing merge_content processor configuration and single node ec2 with 8GB of RAM. can you please clarify if flowfile size gets increase even TB's of data shall i processed with same approach multiple merge_content processor or if it is good with multi-node or increase single node maximum memory availability. JVM Configuration : # JVM memory settings java.arg.2=-Xms2048m java.arg.3=-Xmx2048m
... View more
03-04-2020
01:09 AM
Hi, I'm currently using merge_content processor as merge avro files from two sources like kafka_consumer processor and fetchHDFS file. While converting avro file into one with merge content processor yesterday around 680MB trying to convert but processor drop the file and join with new files and i can't able to recover that data also, because content_repository backup i limit. Can you please help me out for this use_case processor how much size can be good or is there any setting needs to modifiy in nifi.properties.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
-
HDFS
12-15-2019
09:26 PM
Sure thanks. @MattWho. it works!
... View more
12-12-2019
09:27 PM
Hi, We are currently using merge content processor to merge kafka messages with "minimum number of entries" = 500. when 500 messages reached it will merge as single file. That use case works fine. whenever end of the day at 11:59, is there any queue pending in merge content processor of kafka messages needs to dynamically pushed before start to the new date. Kindly help me out from this kind of use case. Workflow : kafka_consumer -> merge_content_prcessor -> puthdfs @mburgess
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
02-06-2018
11:15 AM
1 Kudo
Hi, I had been automate get request based on user query param using nifi with help of InvokeHttp.Here is the example; http://aaa.com/q="bigdata"&api_key="" http://aaa.com/q="apple"&api_key="" I did split text using line by line read and invoke http processor using fetch those response as json format. My question is before invokehttp processor, i want to extract those query param values those query values are static. I need to know which query param using i got response. Please give me some suggestion. It would much helpful. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
12-27-2017
05:06 AM
I think permission denied issue. Try to make readwrite permission to batch file. Either by using command line or gui. I think below command will able to useful; icacls "C:\Program Files (x86)\Program File" /grant Everyone:M Like that mention your nifi file location. I hope it helps!!!
... View more