Hello Experts, We are using Nifi for continuous data ingestion over http port (as rest api) and also we use nifi admin rest api to dynamically start/stop procesors or update variable registry or create process groups etc. We are facing a challenge with these admin rest api's (/nifi-api), i.e   when nifi restarted all apis responds very quickly and works fine and when days go on (if no nifi restart in btwn) they nifi rest api's becomes very slow and at some point they stop responding or timing out. And this slowness/not responding issue will resolve only after nifi restart. Since we have real time data ingestion via http port , we cant really restart frequently. Nifi version : 1.16.3 Is there any solution for this problem ? or any suggestion to tackle this would be much appreciated Thanks, Mahendra ... View more

Hello experts, I am using Apache nifi to connect to Delta table and read and write data. Nifi connected to Delta table via Spark Thrift server jdbc. (I use one custom processor and ConvertJsonToSql processor)   Now the issue is - when I create a table with below create script - create table schemagetnifi.getexternal_nifi1 (userId integer, id integer, title String, completed boolean); it works fine. But if I use below table create statement create table schemagetnifi.getexternal (userId integer, id integer, title String, completed boolean) using delta; Nifi (java) is throwing error (looks like its unable read meta data)   Only difference in 2 different tables creation script is "USING DELTA", with that I am facing error. Any suggestion or help would be much appreciated ... View more

Hello Experts,   I am connecting Spark Thrift Server from Nifi through JDBC to run sql select query against delta tables. Normal select query is working fine. But now I want to implement pagination for my sql query for large tables.   I tried using OFFSET and LIMIT x,y keywords etc but both of them did not work with STS JDBC. Is there anyway to achieve pagination with thrift server jdbc sql? I am not sure if offset and/or limit keywords are not available or not working due to some version. Could you please suggest me on this?   Thanks Mahendra ... View more

Hello experts,   We have setup our nifi to consume message from kafka broker which has SSL as security protocol.   It worked well with pre-prod brokers, but facing below error in prod broker. (we have configured ssl context service with respective ssl certs)   Preprod nifi <-> Preprod kafka : working Prod nifi <-> Prod kafka : Not  working   telnet is working from Prod nifi to Prod kafka broker.   2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Connection to node -3 (kafka3.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.150:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Bootstrap broker kafka3.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -3 rack: null) disconnected 2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Connection to node -8 (kafka8.xxx.pivotal.emea.kaas.3stripes.net/xx.xxx.5.199:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Bootstrap broker kafka8.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -8 rack: null) disconnected 2023-07-17 15:31:00,819 WARN [Timer-Driven Process Thread-127] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmModelNiFiConsumer-25, groupId=ProdSpmModelNiFiConsumer] Connection to node -6 (kafka6.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.13:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.     Any idea what could be the reason for this?   Thanks Mahendra ... View more

Hello Experts,   I need to query delta table from apache nifi. Since Nifi cant connect delta table directly, I am trying it through Spark Thrift Server.   I have used 'Hive3ConnectionPool' controller service with 'SelectHive3QL' processor and used below jdbc url and it worked for me. jdbc:hive2://localhost:10002   But I dont know whether using jdbc:hive2//<host>:<port> is right approach or should I use jdbc:spark//<host>:<port> url? Will there be any performance difference between jdbc:hive2 & jdbc:spark jdbc url's with respect to performance or any other imp thing? I tried jdbc:spark//<host>:<port> url with 'DBCPConnectionPool 1.16.3' cntroller service but could not succeed with DatabricksJDBC jar. Any suggestion would be appreciated   Thanks Mahendra ... View more