Member since
12-03-2017
147
Posts
24
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1149 | 11-03-2023 12:17 AM | |
2887 | 12-12-2022 09:16 PM | |
1092 | 07-14-2022 03:25 AM | |
1747 | 07-28-2021 04:42 AM | |
2007 | 06-23-2020 10:08 PM |
11-10-2023
02:55 AM
Hello Experts, We are using Nifi for continuous data ingestion over http port (as rest api) and also we use nifi admin rest api to dynamically start/stop procesors or update variable registry or create process groups etc. We are facing a challenge with these admin rest api's (/nifi-api), i.e when nifi restarted all apis responds very quickly and works fine and when days go on (if no nifi restart in btwn) they nifi rest api's becomes very slow and at some point they stop responding or timing out. And this slowness/not responding issue will resolve only after nifi restart. Since we have real time data ingestion via http port , we cant really restart frequently. Nifi version : 1.16.3 Is there any solution for this problem ? or any suggestion to tackle this would be much appreciated Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
11-03-2023
12:17 AM
1 Kudo
Found the issue, DisrtibtedMapCacheServer was configured with default 1 mb to handle in each cache request. But I was trying to pass a attribute value to pass to wait processor from Notify which was 1.5 mb. After increasing the cache server config from 1mb to 10 mb it worked
... View more
11-02-2023
09:19 PM
Hello Experts, I am using Wait - notify processors for my flow, once in a while I am seeing this error in Notify processor. java.lang.RuntimeException: Unable to communicate with cache when processing. The screen shot attached. Any idea what is the reason and how to resolve this ? Thanks in advance. Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
10-16-2023
11:13 AM
Hello experts, I am using Apache nifi to connect to Delta table and read and write data. Nifi connected to Delta table via Spark Thrift server jdbc. (I use one custom processor and ConvertJsonToSql processor) Now the issue is - when I create a table with below create script - create table schemagetnifi.getexternal_nifi1 (userId integer, id integer, title String, completed boolean); it works fine. But if I use below table create statement create table schemagetnifi.getexternal (userId integer, id integer, title String, completed boolean) using delta; Nifi (java) is throwing error (looks like its unable read meta data) Only difference in 2 different tables creation script is "USING DELTA", with that I am facing error. Any suggestion or help would be much appreciated
... View more
Labels:
- Labels:
-
Apache NiFi
10-03-2023
11:46 AM
Hello Experts, I have a requirement where I need to re-load my custom processors nar without restarting Nifi. I tried with nifi.nar.library.autoload.directory option, If I freshly place new nar first time in autoload directory it loads automatically. But if I replace the nar in autoload directory again with some enhancements, it does not get reloaded in nifi. I tried with changing nar version, with that nifi loaded that new version nar but that resulted in 2 different versions processors in Nifi (it does not unload the pre version nar) Is there any any "auto reload" feature for custom nar or something like unload prev version and load new version automatically without restart? Any help/suggestion is much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-29-2023
11:17 AM
Hello Experts, I am connecting Spark Thrift Server from Nifi through JDBC to run sql select query against delta tables. Normal select query is working fine. But now I want to implement pagination for my sql query for large tables. I tried using OFFSET and LIMIT x,y keywords etc but both of them did not work with STS JDBC. Is there anyway to achieve pagination with thrift server jdbc sql? I am not sure if offset and/or limit keywords are not available or not working due to some version. Could you please suggest me on this? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
-
Apache Spark
07-18-2023
09:25 PM
Hello experts, We have setup our nifi to consume message from kafka broker which has SSL as security protocol. It worked well with pre-prod brokers, but facing below error in prod broker. (we have configured ssl context service with respective ssl certs) Preprod nifi <-> Preprod kafka : working Prod nifi <-> Prod kafka : Not working telnet is working from Prod nifi to Prod kafka broker. 2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Connection to node -3 (kafka3.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.150:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Bootstrap broker kafka3.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -3 rack: null) disconnected 2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Connection to node -8 (kafka8.xxx.pivotal.emea.kaas.3stripes.net/xx.xxx.5.199:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Bootstrap broker kafka8.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -8 rack: null) disconnected 2023-07-17 15:31:00,819 WARN [Timer-Driven Process Thread-127] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmModelNiFiConsumer-25, groupId=ProdSpmModelNiFiConsumer] Connection to node -6 (kafka6.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.13:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Any idea what could be the reason for this? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
07-18-2023
08:46 PM
Old version of Databricks jdbc jar has the simba driver class, with that I am able to use jdbc:spark:// connection string. But jdbc:hive2:// performance looks better with thrift for me
... View more
07-16-2023
09:48 PM
Hello Experts, I need to query delta table from apache nifi. Since Nifi cant connect delta table directly, I am trying it through Spark Thrift Server. I have used 'Hive3ConnectionPool' controller service with 'SelectHive3QL' processor and used below jdbc url and it worked for me. jdbc:hive2://localhost:10002 But I dont know whether using jdbc:hive2//<host>:<port> is right approach or should I use jdbc:spark//<host>:<port> url? Will there be any performance difference between jdbc:hive2 & jdbc:spark jdbc url's with respect to performance or any other imp thing? I tried jdbc:spark//<host>:<port> url with 'DBCPConnectionPool 1.16.3' cntroller service but could not succeed with DatabricksJDBC jar. Any suggestion would be appreciated Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
-
Apache Spark
07-16-2023
09:37 PM
@MattWho Its very clear now, thank you so much !
... View more