Find answers, ask questions, and share your expertise
Recently active
Hello Experts, I need to query delta table from apache nifi.Since Nifi cant connect delta table directly, I am trying it through Spark Thrift Server. I have used 'Hive3ConnectionPool' controller service with 'SelectHive3QL' processor and used below jdbc url and it worked for me.jdbc:hive2://localhost:10002 But I dont know whether using jdbc:hive2//<host>:<port> is right approach or should I use jdbc:spark//<host>:<port> url?Will there be any performance difference between jdbc:hive2 & jdbc:spark jdbc url's with respect to performance or any other imp thing?I tried jdbc:spark//<host>:<port> url with 'DBCPConnectionPool 1.16.3' cntroller service but could not succeed with DatabricksJDBC jar.Any suggestion would be appreciated ThanksMahendra
Hello, I've been trying to use Spark jdbc to connect to a CDP cloud virtual warehouse using the Hive jar (hive-jdbc-3.1.0-SNAPSHOT-standalone.jar) but I have the following errorCaused by: java.sql.SQLFeatureNotSupportedException: Method not supportedat org.apache.hive.jdbc.HivePreparedStatement.addBatch(HivePreparedStatement.java:78)Looking into the jar sources I see that the addBatch method throws the exception by default. Spark needs to use this method to insert the data into hive. Is there any other driver that can be used to overcome this problem.If no, does cloudera will support spark to connect to CDP virtual warehouse. Thanks in advance Best regards,Diego
Hi everyone, We are unable to get the record count into the variable after Kerberos Authentication implementation and Ranger Installation. we are using following command but it's showing count by printing. Following is we used commandI need to get the count into the variable. anyone please help me! count=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')
How to disable/remove auto TLS certificates and create self-signed certificate in Cloudera version 6.2. The Cloudera version I am using is 6.2 having self-signed certificates that expired a few days back. now the Cloudera manager is not restarting. I want to remove the existing ones and create a new self-signed certificate and apply it to the cluster. can anyone help?
Hello Community, I can assign an attribute's value using a reference to a parameter context value ( e.g. #{variablle_abc} ). But when I attempt to do this same thing within the "Advanced" window in an UpdateAttribute processor, it fails. I've verified the Process Group Parameter Context is set. Anyone know how to make this work in the Advanced window of an UpdateAttribute processor?
Hello all!Is there a way to use PGR and input/output ports in the same nifi instance?The aim of that is to reuse the same processor multiple times by only reference to one PGR.If not, are there any other suggestions to achieve the same idea؟
Hello everyone, can someone please take 15 minutes with me to solve some issues in apache nifi ? i just started learning apache nifi and there is some mistakes that i don't know how to solve. i'm reay to prepare a meet for that. if anyone is available it would be so grateful !! Thank you.
Problem Summary---------------------------------------------------Behavior of auth_user table in CDH 6.3.4Problem Description---------------------------------------------------SymptomUser's first name states only one letter in hue schema under user tableQuestion:Is this the default behavior of know issue in CDH 6.3.4?User's first name has only first letter in Hue auth_user table of Hue(configured LDAP)+-----+----------+------------------+-------------+--------------------------+| id | username | first_name | FirstLetter | last_name |+-----+----------+------------------+-------------+--------------------------+| 22 | mycp2cqj | T | T | Khoo || 90 | mycp2fep | W | W | Ng || 101 | cim26384 | M | M | Mohd Baharudin || 243 | mycp1sfs | J | J | Wong || 264 | mycp22yf | G | G | Lee || 272 | mycp2hg4 | W | W | Ab Rahman || 282 | mycp1vk8 | Hui Sue | H | Khoo | <<<<<<<<<<| 284 | phcc1yb9 | R | R | Rochelle Anne C de Matta |
Hello Experts, I am using ConsumeAzureEventHub processor to consume messages in Nifi 1.16.'Concurrent Tasks' field of this processor is disabled so I am not able to speed up the consume processor.What is the reason for this limitation and is there any work around for this ?(we usually run kafka consumer with higher concurrency matching partitions) Thanks,Mahendra
Logs from nifi-app.log file ERROR [nifi.async.multi_dest.batch_nifi.async.multi_dest.batch.solr_destWriter] o.a.s.client.solrj.impl.CloudSolrClient Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://<hostname>:8886/solr/ranger_audits_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /solr/ranger_audits_shard1_replica_n1/update. Reason:<pre> Authentication required</pre></p></body></html>, retry? 0 Pfa the error attachment
Hi all,I work with cloudera 7.4.4 as our solution works with hive over spark.As I load text files into hive , I may have schema changes on 3 manners:1. the source data has added a column - causes data loss on insertion till the column is updated on Hive2. the source data has omitted a column - fails the insert since that column was not dropped on hive3. the data type has escalated to different type - had Hive is not updated in the new type , for example , int to bigint , the result will be nullnevertheless, inferschema of spark may change numeric fields to alpha and vice versa. is there a certain way , to make a non external Hive table to comply with these changes.I did manage to create a program that do a filler of omitted columns to the dataframe and auto add new columns and escalates the data type, but is there a built in method?for change alphanumeric to numeric and vice versa i don't have a solution. Or, would you suggest to put the Hive as an external table over hbase/m
HI Team, I am trying to connect to Hbase via phoenix from Java SpringBoot Application.This is to fetch data from Hbase from my Microservice Application. I ma using the Phoenix connection URL : jdbc:phoenix:${zk-node}:${port}:/hbase:${principal}@${realm}:${keytab-path} I am connecting from java jdbctemplate and using the SQL Drivermanager. Am getting below error:2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.apache.phoenix.exception.PhoenixIOException: callTimeout=1200000, callDuration=1330750: Failed after attempts=16, exceptions: [ERR] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:83)2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:646)2023-04-18T09:43:15.335+05:30 [APP/PROC/WEB/0] [ERR] at org.springf
I had set the properties in zoo.cfg below to enable TLSv1.3 on zookeper 3.8.0 , but in the logs I see that zookeeper is connected to other zookeeper node on TLSv1.2 that is the default version. I want to know how can I enable TLSv1.3 on zookeeper 3.8.0zookeeper.ssl.protocol=TLSv1.3zookeeper.ssl.quorum.protocol=TLSv1.3
unable add hive aux jars from CM
Hi Team, Job failing due to below error, can any one help me on this. could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation
I am trying to access the NIFI using Rest API. I am using python requests library. I am able to generate the token using username and password but when I am trying to use the token to get the expiration of the token or the list of counters processors that I have created I am getting it is unauthorized.import requestsresponse = requests.post("https://127.0.0.1:8443/nifi-api/access/token?username=<username>&password=<password>",headers={"Content-Type":"application/x-www-form-urlencoded"},verify=False) # if verify is true it will throw verification failedtoken = response.content.decode() # got the token#checking the token expirationresponse = requests.get("https://127.0.0.1:8443/nifi-api/access/token/expiration",headers={"Accept":"*/*","Authorization":"Bearer {}".format(token)}) # using the generated tokenprint(response)"Unauthorized"I am able to get the response json using the same token through postman. So, I am not able to understand it. Is it because I am not verifying
hello, I have the following problem - I have a flow and I have a name attribute in it, I need to change it to uppercase. I used UpdateAtribute processor for this, as property I have name and as vaule I have:${name:toUpper()}unfortunately it doesn't work and now instead of having name in capital letters I have Empty string set, anyone know why this is happening?
I need to load the data from big query table using nifi.I used executesql nifi processor to load data from big query table.when i was executing processor, i got following error.java.lang.NoSuchMethodError: 'void com.google.api.gax.rpc.RequestUrlParamsEncoder.(com.google.api.gax.rpc.RequestParamsExtractor)'configure DBCPConnectionPooling Service in following class name - com.simba.googlebigquery.jdbc42.DriverI am using nifi1.21.0 version. download latest version of big query jdbc driver(version 4.2) and other all dependencies and put into nifi/lib folder I also used correct connection url when configuring pooling service.How to solve this error?
Hi,I'm new to nifi, I downloaded nifi, I'm trying to log in, but I can't enter the address:https://127.0.0.1:8443/ 2023-07-10 16:16:34,682 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.java.lang.IllegalStateException: Invalid nifi.web.https configuration in nifi.propertiesat org.apache.nifi.web.server.connector.FrameworkServerConnectorFactory.<init>(FrameworkServerConnectorFactory.java:89)at org.apache.nifi.web.server.JettyServer.configureConnectors(JettyServer.java:738)at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:207)at org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)at org.apache.nifi.NiFi.<init>(NiFi.java:164)at org.apache.nifi.NiFi.<init>(NiFi.java:83)at org.apache.nifi.NiFi.main(NiFi.java:332)Caused by: org.apache.nifi.security.util.TlsException: The truststore properties are not validat org.apache.nifi.security.util.SslContextFactory.getTrustManagers(SslContextFactory.jav
Hi guys, i'm working on a NiFi 1.21.0 and the user login is connected to AzureAd. It's working a nice way for all humans, but we want to check the nifi with our monitoring solution. My login-identity-providers.xml contains <loginIdentityProviders> <provider> <identifier>single-user-provider</identifier> <class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class> <property name="Username">known-user</property> <property name="Password">hashed-known-password</property> </provider></loginIdentityProviders> My authorizers.xml contains <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property>
Hello experts, One of our client has data stored in delta table(delta lake), I am totally new to delta lake part.Do we have any option in nifi to read data from delta lake or is there any sample 3rd party implementation available for the same for nifi? Thanks,Mahendra
Hi guys, Please help me out with a strange behavior when using PutBigQuery.I am using Apache NiFi 1.19.1 So, my flow is as follows: Step a:I have a GenerateTableFetch and an ExecuteSQLRecord, which extracts some data out of Database. Step b:The data gets loaded into a GCS Bucket, using PutGCSOBject. Step c:When the data has been saved into the GCS Bucket, I have an UpdateAttribute Processor, linked to the success queue.Within this UpdateAttribute Processor, I have defined the following 3 attributes:TABLE_NAME = ${generatetablefetch.tableName:toUpper()}EXECUTION_DATE = ${now():toNumber()}MESSAGE = 1 Step d:The success queue is linked afterwards to an AttributesToJSON Processor.I have modified the properties as follows:Destination = flowfile-contentAttributes List = TABLE_NAME, EXECUTION_DATE, MESSAGE Step e:Via success, I link to an ConvertRecord, where I change from JSON to AVRO.The JSON Reader and the AVRO Writter are both defined with the following
Our CDP version is 7.1.7 and enabled kerberos, after we set up DAS service on it, we found that: queries only submitted by hive.keytab can be collected to DAS, queries submitted by other users' keytab such like hdfs.keytab/generic account keytab cannot be collected.So, does DAS only support to collect queries submitted by hive.keytab?If not, how can I config this?
Hello experts, I need to connect to my customer's kafka broker from apache nifi processors. They said security protocol is SASL_SSL (where is SASL is PLAIN) and provided username and password for SASL part.But they say they dont have any ssl certificate to share for _SSL, they informed they are using clnfluent cloud and that does not need ssl certs to connect.But when I configure nifi processor with SASL_SSL it mandatorily asks for ssl certificate. Any idea how can I get this ssl certificate to connect to confluent cloud kafka broker ?Or how Nifi can connect without ssl ?Any suggestion/help much appreciated ThanksMahendra
Hello All, I had secure Nifi and Secured Nifi-registry running on Docker.Both share same certificatesAble to access both UI's individually. Given static IP to Registry in Docker and subjectalternativenames with this IP while generating certificates. Given this static_IP:<port> in registry client.created localhost user : CN=localhost, OU=Nifi in both systems and given necessary permissions.Created few buckets and given all necessary permissions.. But when I try to do version control, it says no available buckets. As i troubleshoot, I can see the logs that shows successful authorization to registry.. 2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.5.0.1 [CN=new_admin, OU=Nifi] GET https://localhost:8443/nifi-api/flow/registries/21d17428-0189-1000-4a0e-a91cc8cd2aa7/buckets2023-07-05 10:25:47,312 INFO [NiFi Web Server-468] o.a.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.