Q&A

Logs from nifi-app.log file ERROR [nifi.async.multi_dest.batch_nifi.async.multi_dest.batch.solr_destWriter] o.a.s.client.solrj.impl.CloudSolrClient Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://<hostname>:8886/solr/ranger_audits_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /solr/ranger_audits_shard1_replica_n1/update. Reason:<pre>    Authentication required</pre></p></body></html>, retry? 0 Pfa the error attachment 

Hi all,I work with cloudera 7.4.4 as our solution works with hive over spark.As I load text files into hive , I may have schema changes on 3 manners:1. the source data has added a column - causes data loss on insertion till the column is updated on Hive2. the source data has omitted a column - fails the insert since that column was not dropped on hive3. the data type has escalated to different type - had Hive is not updated in the new type , for example , int to bigint , the result will be nullnevertheless, inferschema of spark may change numeric fields to alpha and vice versa. is there a certain way , to make a non external Hive table to comply with these changes.I did manage to create a program that do a filler of omitted columns to the dataframe and auto add new columns and escalates the data type, but is there a built in method?for change alphanumeric to numeric and vice versa i don't have a solution. Or, would you suggest to put the Hive as an external table over hbase/m

HI Team, I am trying to connect to Hbase via phoenix from Java SpringBoot Application.This is to fetch data from Hbase from my Microservice Application. I ma using the Phoenix connection URL : jdbc:phoenix:${zk-node}:${port}:/hbase:${principal}@${realm}:${keytab-path} I am connecting from java jdbctemplate and using the SQL Drivermanager. Am getting below error:2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.apache.phoenix.exception.PhoenixIOException: callTimeout=1200000, callDuration=1330750: Failed after attempts=16, exceptions: [ERR] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:83)2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:646)2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] at org.springf

I had set the properties in zoo.cfg below to enable TLSv1.3 on zookeper 3.8.0 , but in the logs I see that zookeeper is connected to other zookeeper node on TLSv1.2 that is the default version. I want to know how can I enable TLSv1.3 on zookeeper 3.8.0zookeeper.ssl.protocol=TLSv1.3zookeeper.ssl.quorum.protocol=TLSv1.3

unable add hive aux jars from CM

Hi Team, Job failing due to below error, can any one help me on this. could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation 

I am trying to access the NIFI using Rest API. I am using python requests library. I am able to generate the token using username and password but when I am trying to use the token to get the expiration of the token or the list of counters processors that I have created I am getting it is unauthorized.import requestsresponse = requests.post("https://127.0.0.1:8443/nifi-api/access/token?username=<username>&password=<password>",headers={"Content-Type":"application/x-www-form-urlencoded"},verify=False) # if verify is true it will throw verification failedtoken = response.content.decode() # got the token#checking the token expirationresponse = requests.get("https://127.0.0.1:8443/nifi-api/access/token/expiration",headers={"Accept":"*/*","Authorization":"Bearer {}".format(token)}) # using the generated tokenprint(response)"Unauthorized"I am able to get the response json using the same token through postman. So, I am not able to understand it. Is it because I am not verifying

 hello, I have the following problem - I have a flow and I have a name attribute in it, I need to change it to uppercase. I used UpdateAtribute processor for this, as property I have name and as vaule I have:${name:toUpper()}unfortunately it doesn't work and now instead of having name in capital letters I have Empty string set, anyone know why this is happening?         

I need to load the data from big query table using nifi.I used executesql nifi processor to load data from big query table.when i was executing processor, i got following error.java.lang.NoSuchMethodError: 'void com.google.api.gax.rpc.RequestUrlParamsEncoder.(com.google.api.gax.rpc.RequestParamsExtractor)'configure DBCPConnectionPooling Service in following class name - com.simba.googlebigquery.jdbc42.DriverI am using nifi1.21.0 version. download latest version of big query jdbc driver(version 4.2) and other all dependencies and put into nifi/lib folder I also used correct connection url when configuring pooling service.How to solve this error?

Hi,I'm new to nifi, I downloaded nifi, I'm trying to log in, but I can't enter the address:https://127.0.0.1:8443/ 2023-07-10 16:16:34,682 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.java.lang.IllegalStateException: Invalid nifi.web.https configuration in nifi.propertiesat org.apache.nifi.web.server.connector.FrameworkServerConnectorFactory.<init>(FrameworkServerConnectorFactory.java:89)at org.apache.nifi.web.server.JettyServer.configureConnectors(JettyServer.java:738)at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:207)at org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)at org.apache.nifi.NiFi.<init>(NiFi.java:164)at org.apache.nifi.NiFi.<init>(NiFi.java:83)at org.apache.nifi.NiFi.main(NiFi.java:332)Caused by: org.apache.nifi.security.util.TlsException: The truststore properties are not validat org.apache.nifi.security.util.SslContextFactory.getTrustManagers(SslContextFactory.jav

Hi guys, i'm working on a NiFi 1.21.0 and the user login is connected to AzureAd. It's working a nice way for all humans, but we want to check the nifi with our monitoring solution. My login-identity-providers.xml contains  <loginIdentityProviders> <provider> <identifier>single-user-provider</identifier> <class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class> <property name="Username">known-user</property> <property name="Password">hashed-known-password</property> </provider></loginIdentityProviders>   My authorizers.xml contains  <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property>

Hello experts, One of our client has data stored in delta table(delta lake), I am totally new to delta lake part.Do we have any option in nifi to read data from delta lake or is there any sample 3rd party implementation available for the same for nifi? Thanks,Mahendra

Hi guys, Please help me out with a strange behavior when using PutBigQuery.I am using Apache NiFi 1.19.1 So, my flow is as follows: Step a:I have a GenerateTableFetch and an ExecuteSQLRecord, which extracts some data out of Database. Step b:The data gets loaded into a GCS Bucket, using PutGCSOBject. Step c:When the data has been saved into the GCS Bucket, I have an UpdateAttribute Processor, linked to the success queue.Within this UpdateAttribute Processor, I have defined the following 3 attributes:TABLE_NAME = ${generatetablefetch.tableName:toUpper()}EXECUTION_DATE = ${now():toNumber()}MESSAGE = 1 Step d:The success queue is linked afterwards to an AttributesToJSON Processor.I have modified the properties as follows:Destination = flowfile-contentAttributes List = TABLE_NAME, EXECUTION_DATE, MESSAGE Step e:Via success, I link to an ConvertRecord, where I change from JSON to AVRO.The JSON Reader and the AVRO Writter are both defined with the following

Our CDP version is 7.1.7 and enabled kerberos, after we set up DAS service on it, we found that: queries only submitted by hive.keytab can be collected to DAS, queries submitted by other users' keytab such like hdfs.keytab/generic account keytab cannot be collected.So, does DAS only support to collect queries submitted by hive.keytab?If not, how can I config this?

Hello experts, I need to connect to my customer's kafka broker from apache nifi processors. They said security protocol is SASL_SSL (where is SASL is PLAIN) and provided username and password for SASL part.But they say they dont have any ssl certificate to share for _SSL, they informed they are using clnfluent cloud and that does not need ssl certs to connect.But when I configure nifi processor with SASL_SSL it mandatorily asks for ssl certificate. Any idea how can I get this ssl certificate to connect to confluent cloud kafka broker ?Or how Nifi can connect without ssl ?Any suggestion/help much appreciated ThanksMahendra

Hello All, I had secure Nifi and Secured Nifi-registry running on Docker.Both share same certificatesAble to access both UI's individually. Given static IP to Registry in Docker and subjectalternativenames with this IP while generating certificates. Given this static_IP:<port> in registry client.created localhost user : CN=localhost, OU=Nifi in both systems and given necessary permissions.Created few buckets and given all necessary permissions..         But when I try to do version control, it says no available buckets.   As i troubleshoot, I can see the logs that shows successful authorization to registry.. 2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.5.0.1 [CN=new_admin, OU=Nifi] GET https://localhost:8443/nifi-api/flow/registries/21d17428-0189-1000-4a0e-a91cc8cd2aa7/buckets2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.

After successfully running a 3-node NiFi cluster without any issues for the past three years, I am considering setting up an additional cluster. Upon checking the resource utilization of the 3-node ZooKeeper cluster, I noticed that the nodes have very low compute utilization. I would like to know if it is recommended to stick with a setup of 3 NiFi nodes and 3 ZooKeeper nodes, totaling 6 nodes, or if it would be more beneficial to use embedded ZooKeeper.

Having issues flattening the nested array in an array of objects back to its parent. Given this JSON :  [ { "id": "13", "uom": "EA", "name": "pOne", "display_name": "displayOne", "brand": "brandOne", "categories": [ { "level1": "cLevel1", "level2": "cLevel2" } ] }, { "id": "87", "uom": "EA", "name": "pTwo", "display_name": "displayTwo", "brand": "brandTwo", "categories": [ { "level1": "cLevel1", "level2": "cLevel2" } ] }]  I want to transform to this: [ { "pName" : "pOne", "pBrand" : "brandOne", "pVariant" : "displayOne", "pSku" : "13-EA-000", "categoryHierarchy" : "cLevel1/cLevel2"}, { "pName" : "pTwo", "pBrand" : "brandTwo", "pVariant" : "displayTwo", "pSku" : "87-EA-000", "categoryHierarchy" : "cLevel1/cLevel2"} ]  So far I have these specs: [ { "operation": "modify-default-beta", "spec": { "*": { "productSku": "=conc

driver class: com.cloudera.hive.jdbc.HS2Driverjar : HiveJDBC42-2.6.11.1014the kerberos needs a keytab file, i wonder if this can be set by 'org.apache.hadoop.security.UserGroupInformation' class thks.

I'm new to NiFi, as you'll realize in this post. I'm trying to troubleshoot this issue where after I'm logging in with the provided password in the log file, I don't get past the loading screen. Here are things I've noticed: in nifi.app log file, I get the error:Archive cleanup completed for container default; will now allow writing to this container.in cmd, I have this error: Command Failed to set permissions so that only the owner can read status file [path]; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file  when opening any .bat other than run-nifi.bat as administrator, the window immediately closes.This is the process that's looping over and over in the nifi-app.log file My feeling is that these are having to do with some sort of permission issue. I've tried taking the folder off read-only but that potential solution alone has been a rabbit hole of issue

I had setup Secure Nifi Registry using Docker compose.Everything worked and I am able to open UI at https://localhost:18443/nifi-registryBut when i try to add buckets, I don't even see the settings icon. As I checked the logs i can see below error:  nifi_registry_container_persistent | 2023-06-29 15:23:01,075 INFO [NiFi Registry Web Server-23] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[CN=admin, OU=NiFi], groups[] does not have permission to access the requested resource. Unknown user with identity 'CN=admin, OU=NiFi'. Returning Forbidden response.nifi_registry_container_persistent | 2023-06-29 15:23:01,129 INFO [NiFi Registry Web Server-20] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.nifi_registry_container_persistent | 2023-06-29 15:23:01,190 INFO [NiFi Registry Web Server-16] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.Illegal

Hi, I am running a query on Hive through my Spark application using HiveWarehouseConnector.I want to use a particular YARN queue for Tez job launched by HiveWarehouseConnector (custom queue configuration at application level).I have tried following two ways:1. By using Spark conf and setting spark.hive.tez.queue.name = <queue name>2. By setting tez.queue.name parameter in the hiveserver2 URL as suggested in the following thread https://community.cloudera.com/t5/Support-Questions/Setting-yarn-queue-for-hive-with-beeline/td-p/161499 I am able to set the queue for beeline using the URL option. However, any of the options is not working for HiveWarehouseConnector. Can anyone please help in this regard? 

I have NiFi installed on an EC2 instance NiFi version - 1.22.0 I have a requirement to pull files text files from s3 bucket and write them locally in NiFi as PoC. I have configured an input_port and FetchS3Object and PutFile processor as shown in the screenshot. The Nifi ec2 intance already has access to the s3 bucket and I have checked that by using "aws s3 cp s3_uri ./" command to copy files from s3 bucket to ec2 instance and it worked, However, for some reason, it seems NiFi FetchS3Object processor is running into some issue. The Nifi-app.log havent been very helpful even though I enabled "debug" level on the processor. It showed no error or warn messages. The only messages the processor showed is on the gui as shown in the screenshot. I have shown the values I configured in configuration. Only specified bucket name, object key, which I initially set to *.evtx but then changed to * to get any file and region option. Did not specify access key or se

I want to execute export command to take the PSQL dump of the database and transfer it using NIFI and the execute the import command at the destination db.can anyone provide a flow to do so ?

Hi guysI'm new to nifi and I need some help if possible.I am consuming from a Kafka Consumer that, when paused for some time, returns the json with several records, and I would like to divide them into separate files 1:1ConsumeKafka ResultSet: here is my json from 1 record    { "before":null, "after":{ "uid":"80faad95-3383-4093-b4c1-e1840f076eef", "nome":"yuri", "sobrenome":"feminino", "email":"yuri.feminino@garupa.co", "celular":"53999122007", "cpf":"3", "avaliacao":"AfQ=", "created_at":1687173349000, "updated_at":1687423717000, "promocode":null, "fcm_registration_id":"edzz9eK3QDOUx-jsmpix4K:APA91bFPjXEsD7h-fYzih-pbr-TMfnF21Qk6sPDQYoa52etUwuIn3jCv1nM7ts4QYQ6f2Wcg-S88KdZsAqcEk98ZUAle-xnvFreavHUjB9_JBjj5wKDfjIYx6Z0en65Po_9knVGpz1u9", "paypal_client_id":null, "paypal_refresh_token":null, "foto":"https://garupa-dev.s3.amazonaws.com/207e9b35-c4b2-4660-af4f-cc20a629b581.jpg", "stripe_cust