Q&A

I am facing an issue with my Nifi as I am unable to access the Nifi anymore. I think this maybe because I have modified and replaced the nifi.properties file from the original downloaded version of the file.When I try to run Nifi as a service using sudo service nifi start it does not start the service.Here are the nifi-app.log that I can see...   ERROR [main] org.apache.nifi.NiFi Failure to launch NiFijava.lang.IllegalArgumentException: There was an issue loading properties at org.apache.nifi.NiFi.initializeProperties(NiFi.java:361) at org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:343) at org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:339) at org.apache.nifi.NiFi.main(NiFi.java:331)Caused by: java.lang.IllegalStateException: Sensitive Properties Key [nifi.sensitive.props.key] not found: See Admin Guide section [Updating t at org.apache.nifi.properties.NiFiPropertiesLoader.getDefaultPr

Need to connect from apache nifi to AWS app config to get the details from app config. Able to connect to aws secret manager but not able to connect to aws app config. Can anyone guide in this process

I have a flow file that I want to enrich with the contents from a list of JSON values based on the "id".  Basically perform an inner join of the flow file with this look-up data set.  However, I need the look-up dataset to be in memory, then using the "id", query the look-up dataset and append the results to the current flow file.  Here is an example: Incoming flow file: { "id": "abc123", "fname": "The", "lname": "Rock"}  Contents of the look-up data set: [ { "id": "abc123", "dob": "03/09/1977", "phone": "987-654-0001" }, { "id": "def765", "dob": "04/08/1976", "phone": "789-654-0001" }, { "id": "hij765", "dob": "05/06/1975", "phone": "456-654-0001" }]  Enriched flow file:{ "id": "abc123", "fname": "The", "lname": "Rock", "dob": "03/09/1977", "phone": "987-654-0001"} I need to be able to look up the correct record in the look-up dataset based on the "id" then append the

Hi all, I installed Hue on my CDP 7.1.8 cluster recently but not able to access the UI. Here are the screencap when trying to access the UI:1.2. I have tried a solution found in the Cloudera Community and checked the Bind Hue Server to Wildcard Address option. However, the problem doesn't solved. Please help me out with this issue. Thanks in advanced.

So I can't seem to sum up correctly to show that there is no difference, even though I've verified in Excel that there is no difference- and that Column A + Column B = Column C in all cases. It appears to be adding or subtracting incorrectly at times- no matter if I use minus or add(even tried switching them around), or using ABS(absolute).Data from table Sales:CurrencyColAColBTotal_ColCAED2,600,0000{null}2,600,000AUD465,000(406,000)59,000CAD(1,450,000,000)35,000,000(1,415,000,000)CHF490,000(114,000)(376,000)CNH(216,000)2,200,0001,984,000This is the query that I'm trying to double-check that the sum of column ColA + ColB = column Total_ColC:SELECT currency,CASEWHEN SUM(COALESCE(ColA,0)) + SUM(COALESCE(ColB,0)) = (SUM(COALESCE(Total_ColC,0)))THEN 0ELSE SUM(COALESCE(ColA,0)) + SUM(COALESCE(ColB,0)) - (SUM(COALESCE(Total_ColC,0)))END AS Total_CheckFROM `reports`.salesWHERE DateOfBusiness = '2024-08-10'GROUP BY CurrencyHere are the incorrect results that I get for some rows- notably,

Hello,We are tring to use hadoop distcp command to transfer data from on-prem hadoop cluster to GCS by using a private endpoint.We have tested many different method, and finally we added directly the following lines into /etc/hosts. (Other GCP auth info has been added into core-site.xml.)XX.XX.XX.XX storage.googleapis.comXX.XX.XX.XX googleapis.com*XX.XX.XX.XX is the IP of our private endpointhadoop fs -ls and -cp command can list or copy object correctly to the GCS bucket.But with hadoop distcp command, its mapreduce jobs always go to public endpoint...Does anyone knows how to make the distcp working with GCS private endpoint?Or does anyone have any idea about the private endpoint of distcp trying to reach (so that I can add it into /etc/hosts)?Thank you.

I have a problem with the topic __consumer_offsets, the topic is very large, kafka metrics show that I have 4 billion messages on it, I suspect that message compaction does not work correctly, I constantly see an increase in new messages in the metrics on this topic (however, the total log dir size fluctuates periodically between 19 and 20 GB). What could be the reason that this topic is not being cleaned up?

My service is to forcibly interrupt the impala query, but the interrupt requires queryId, impala driver does not find the corresponding method, jdbc does not see queryId return, driver log level LogLevel=6, and queryId is not seen

24/08/29 10:21:15 WARN DataStreamer: Exception for BP-942923949-172.20.0.202-1722967626672:blk_1074290600_560526Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: java.io.EOFException: Unexpected EOF while trying to read response from serverAug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:521)Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1137)Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: 24/08/29 10:27:30 WARN DataStreamer: DataStreamer ExceptionAug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: java.net.SocketTimeoutException: 495000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/172.20.0.194:40396 remote=/172.20.0.194:9866]Aug 29

I want to stop a processor on the same machine  with restApi but I get an authorization error when I call the rest api.I can solve this problem with bearer token, but since the token has expired, I need to change the bearer  token manually again.Is there a way I can authorize automatically?I am using the executeScript processor when calling the API.Script engine is Groovy.My Nifi version 1.23.2 and Nifi is secure  and not clustered.I login with ldap on UI. This is the part where I call the API in the script:def post = new URL(NifiBaseUrl.value + NifiComponentId.value).openConnection();post.setSSLSocketFactory(sslContext.getSocketFactory())post.setRequestMethod("PUT")post.setDoOutput(true)post.setRequestProperty("Content-Type", "application/json")post.getOutputStream().write(content.getBytes("UTF-8"));post.getResponseCode() Error:java.io.IOException: Server returned HTTP response code: 401 for URL

Good Day!We have a requirement to read the TCP port data and publish it to Kafka. But at times the Kafka gets into a hung state and we need to restart the processor manually. So please help us with a way to configure the "Put Email" processor to trigger an email whenever this queue gets 80% full. Thanks in advance!  

Hi all, in my test cluster I am noticing a slowdown in the execution of acid queries. Analyzing in detail I noticed that the compactions remain stuck at "ready for cleaning" and there are many delta files.  I also tried to manually launch the compaction without any result. hive.metastore.housekeeping.threads.on and hive.metastore.housekeeping.threads.on is true only in 1 hive metastore host. This is a table properties:bucketing_version 2transactional truetransactional_properties defaulttransient_lastDdlTime 1720453037In the development cluster with the identical configuration I do not have this problem. Do you have any suggestions?I'm running in CDP 7.1.9Thanks, Lorenzo

Hi,I'm trying to use PowerBI to analyze my Hive data.But when I'm connecting to Hive by using ODBC, I can see all databases even if I don't have right permissions for other databases.These are the databases I have permissions in Ranger. This is a PowerBi screenshot when I connected Hive by using ODBC.As you can see above, I can see all databases. This is a ODBC setting for connecting to Hive. But if I connect to Hive by using HiveLLAP, I can  see right databases like below. So question isWhy can I see all hive databases without permissions when I connected to Hive by using ODBC. If anyone knows the reason, please let me know.I would really appreciate it. Thanks.  

Hello Experts,I have a nifi kafka consumer (ConsumeKafka_2_6) where the kafka message body (value/flow file content) and message key (kafka.key in flow file attributes) both are avro serialized as per the Confluent kafka way of  serializing.When we use "ConvertRecord + AvroReader CS + ConfluentSchemaRegistry CS"   to convert message body (value/flow file content) , it works fine as it is deserilising the magic byte and schema id to the correct value. But we are trying below to deserilise kafka.key as well the same as value.Bringing the FF attribute kafka key to content (using ReplaceText processor with ${kafka.key}) and use "onvert Record + Avro Reader + ConfluentSchemaRegistry" to deserialize, in this case nifi is resulting in a wrong schema id - 3567 instead of the correct schema id 3545.Is it happening because when Nifi reads kafka.key (originally byte array) and pushes to downstream in FF as FF attribute and which is string?Is there any other way

I have a simple file having say 1000 lines each of which contains say username and useraddress. I need to insert it into db. How to insert it in order. I am able to use PutFile and PutDatabaseRecord both for the insertion purpose. But they insert randomly in any order. I wanted it to be inserted in the same order it is present in the file.How to do that.

 have HDP 2.6.5, in which I'm able to run Ambari UI. Now, when I tried to install Pip by using PuTTy. I am getting this error.These are the steps I tried in order; now after this, I have to install MapReduce as well. *yum-config-manager --save --setopt=HDP-SOLR-2.6-100.skip_if_unavailable=true*yum install https://repo.ius.io/ius-release-e17.rpm https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm*yum install python-pip I'm in 3rd Step. This is error I'm getting.

I am using Nifi 1.14.1, Kafka 2.13-2.7.1 and the processor ConsumeKafkaRecord_2_6 to process messages from a topic where the key and the value where both serialized using avro - schemas for the key and value are stored in the confluent schema registry. But the processor fails to parse the message because there is not a way - that I can see - to specify that both key and value are avro serialized with schemas stored in the confluent schema registry. The convention for naming the schema is usually [topic name]-value and [topic name]-key. I can read the messages just fine using kcat, formerly kafkacat using: kcat -b broker1:9092,broker2:9092,broker3:9092 -t mytopic -s avro -r http://schema-registry_url.com -p 0Is there a way to read such messages or am I supposed to add my own processor to nifi? Here's an error trace:[Timer-Driven Process Thread-9] o.a.n.p.k.pubsub.ConsumeKafkaRecord_2_6 ConsumeKafkaRecord_2_6[id=f9933326-017e-1000-60be-e5077975f031] Failed to parse message fconfigur

I just register cloudera on public cloud 5 days trial. I want to assess dataflow if it's suit on my usecase. When i follow the "Hello world" project there is deploy button that i can't click which needs to have admin account. Stated on https://community.cloudera.com/t5/Support-Questions/CDP-trial-version-admin-rights/m-p/348347#M235350 i need to contact support (create case) to give me admin access, however i can't access it due to access restriction, how come???  

When I attempt to' join' tables or even use 'group by', I get the following error: Hortonworks Hive DSN -- [Hortonworks][Hardy] (35) Error from server: error code: '1' error message: 'Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask'.(HY000,35) Yet, I am able to do a simple 'select-from-where' query with no problem. I am using the IDE Rapid SQL.  Any idea of what is going on?

Hello, I have a simple use case. I have an incoming file having say user name and user address in each line. User name is from char 1 to char 5 and user address is from char 6 to char 15. How to use ExtractText processor for it. I tried using Search Value as ^(.{5})(.{10}) and in replacement value as $1,$2Issue I am having is that for address it is capturing all the chars from 6th char to last and not necessarily upto 15th char. What should I modify?Just for experimentation I tried doing ^(.{5})(.{10})(.{1}) and $1,$2,$3 and this is able to capture properly from 6th char to 15th char. Please help.

Hello everyone,  I'm running a 11-nodes NiFi 1.15.3 cluster. One of the process groups is versioned on NiFi Registry and for some reason the local flowfile does not reflect the versioned configuration, so now the process group is stuck: I cannot do anything on it, not even moving it on the canvas because it always return an error: Node XXXXXXXXX is unable to fulfill this request due to: [15, xxxxx-xxxxx-xxxxxx] is not the most up-to-date revision. This component appears to have been modified The local configuration shows no changes, and nothing I tried so far worked (deleting the flow file, restarting the cluster node, etc). so I just want to delete the process group and deploy it again from the registry, but the web interface won't let me, throwing the same error. Is there a way to force the deletion the process group? Thanks

Hello everyone,recently on Ranger, we noticed that when enabling users in a security zone, they are unable to see tables in a database through Impala. When connecting via JDBC through the command line or directly from HUE, they can view the tables and their respective data, but if all of this is done through Impala, they have no visibility at all.We have 4 clusters, and all of them have the same type of configuration and permissions, but we encounter the issue in 3 of these clusters. All configurations in Ranger have been checked and appear to be correct in all environments.At this point, I have no idea where to begin the analysis of the problem.Also, there are no access issues reported from Ranger to tables in the security zone, and there are no error logs or access denied logs.I realize that without providing an error log, it's difficult to conduct an analysis, but perhaps some of you have suggestions on where to start investigating the problem.Have any of you ever encountered a simi

Whenever I enter Ambari I get the next error in a box with a green "okay" button.500 status code received on GET method for API: /api/v1/clusters/datalakev02/requests?to=end&page_size=10&fields=Requests/end_time,Requests/id,Requests/progress_percent,Requests/request_context,Requests/request_status,Requests/start_time,Requests/cluster_name,Requests/user_name&minimal_responseThen if I want to go to any configs of the services or restart something, I get a similar but shorter message:500 status code received on GET method for API: /api/v1/clusters/datalakev02/kerberos_descriptors/COMPOSITE?evaluate_when=trueI checked the ambari-server logs and I have only this message: 2024-08-28 03:05:10,868 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:133 - Exception receivedjava.lang.RuntimeException: Failed to construct cluster topology while replaying request: org.apache.ambari.server.topology.InvalidTopologyException: NAMENODE HA requires at least 2 hosts running NAMENODE bu

Hello, I want to copy record from one table to another table that has the same structure in Oracle 12c+.One not-null column in source has white space. When trying to insert record to the target table, I got error "cannot insert NULL", so I added UpdateRecord processor to replace Null with white space, but still have the same error.It works if I replace Null with any other character like "_". Is there any way to keep white space when inserting to the target table?Below is my Flowfile and the UpdateRecord properties. Thank you 

Hi,I am trying to configure Ranger using ambari. I copied the postgress jar to ambari server. but I am getting error when I start the range admin.  I have ranger running on separate VM. Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database! Error Code: 0 at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467) at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) at org.springframework.transaction.intercepto