Find answers, ask questions, and share your expertise
Recently active
ConsumerKafka2.6 I connect to kafka using ssl I added a keystore and a triac from kafka servera I placed the same ones on the nifi server but in the nifi logs I get this error.ConsumeKafka_2_6[id=9da42b1a-0187-1000-ffff-ffffb41254ef] Exception while interacting with Kafka so will close the lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@62e55d3b due to SSL handshake failed: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed- Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address my settings StandardSSLContextServiceKeystore Filename/var/ssl/nifi/kafka_broker.keystore.jksKeystore PasswordSensitive value setKey PasswordSensitive value setKeystore TypeJKSTruststore Filename /var/ssl/nifi/kafka_broker.truststore.jksTruststore PasswordSensitive value setTruststore TypeJKSTLS ProtocolTLS My settings consumerKafka2.6Security Protocol SSLSASL Mechanism SCRAM-SHA-512Username
Below are the version detailsCDP 7.1.8CM : 7.8.1HIVE : 3.1.3We are trying to insert data into partitioned table using ORC file. It contains approx 50,000 rowsUsing below command to load dataLOAD DATA INPATH '/user/test/r_14_5.orc' INTO TABLE va_offer_16;we see that all the map and reducer task are completed withing 500s.----------------------------------------------------------------------------------------------VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED----------------------------------------------------------------------------------------------Map 1 .......... container SUCCEEDED 7 7 0 0 0 0Reducer 2 ...... container SUCCEEDED 33 33 0 0 0 0----------------------------------------------------------------------------------------------VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 2601.60 s----------------------------------------------------------------------------------------------However further there is no progress for next 2000 sec be
Hi Team,I am having two issues with cloud era Hbase connection:1. Connecting Hbase through the Phoenix driver is not working.2. Somehow I am able to connect hbase through hbase shell and inserted data. But I have a problem while reading data from Phoenix driver.The steps I followed are below:created habse Table using sqlline and inserting data through hbase shell put API from spark java application. But my consumer app reading data through phoenix driver.can you please help me any table configuration helps me to use hbase table from hbase shell as we all phoenix driver. Issue with existing table is: I am able to query data properly through hbase shell command, But when I am querying data through Phoenix driver rowkey value is getting truncated (only the first letter) and other columns are good.While creating a table, using the following configurations:Column_encoded_bytes=0, slatbucket=88, Compression=snappy, data_block_encoding= fast_DIFF
I am using QueryRecord to translate data from one format to another, but once in a while, I am getting this error here. Not sure what to do about and how to handle it. I am using Nifi 1.19.1[Timer-Driven Process Thread-15] o.a.n.c.r.StandardProcessSession Failed to asynchronously commit session StandardProcessSession[id=25336651] for QueryRecord[id=dba21f2d-ea2b-327b-3692-5eefbb53cd52] java.lang.IllegalStateException: Cannot commit session while reading from FlowFile at org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:245) at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:274) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:556) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:510) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org
Hi ,I am trying to run a "alter table drop partition" command using spark sql which prints the metastore lost connection stack trace : 23/04/19 06:47:53 WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException: SASL authentication not completeat org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:472)at org.apache.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)at org.apache.hadoop.hive.thrift.TFilterTransport.write(TFilterTransport.java:72)at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_partitions_ps_with_auth(ThriftHiveMetas
Hello,with this topic i need a clue. I have a text based data, this is an example row: CA20230405112337BOL006057 20230405V1260 2023021000409I 311003000102504CA20230405112337BOL006057 20230405V1260 2023021000509I 311003000102504 I need to convert to json but:1 - i need to remove all spaces in between2 - i need to separate fields with a separator in order to create a schema and make a conversionExample:CA;20230405;bla bla blaCA;20230405;bla bla bla How can i achieve this?Is there any other way to do that? Many Thabks
Hello,I am using jolt to change the structure of a payload to another form, and it must be dynamic, especially the top level elements "test" which can be multiple. I have tried but not getting it. I would appreciate your kind help.Here is my input json:{ "test1": "test output", "test2": "test output", "Details": { "0": { "id": "first", "name": "the first one" }, "1": { "id": "second", "name": "the second one" } }} and my output json is :{ "Details" : [ { "test1" : "test output", "test2" : "test output", "id" : "first", "name" : "the first one" }, { "test1" : "test output", "test2" : "test output", "id" : "second", "name" : "the second one" } ]} my current jolt spec is:[ { "operation": "shift", "spec": { "Details": "Details", "*": "outer.&" } }, { "operation": "shift", "spec": { "Details": { "*": { "*": "Details[&1].&", "@(2,outer)": "Details[&am
Hello,I'm running Hive 3.1.3, Hue 4.10.0, Oozie 5.2.1 Through Oozie, I'm running a periodic Hive job which takes data from an HDFS path, and adds it to an existing hive table.It works fine when I'm running it using Hive shell. However, when I run it using Oozie, the underlying data in Hive gets updated correctly, but the Oozie job fails without giving any meaningful error. This is a screenshot explaining the situation. The job ran for 1hr 11m, the data in Hive got updated, but Oozie responded with a failed status.Here are the applicationLogs : Parameters:------------------------ DATE=2023-04-12------------------------Hive command arguments : --hiveconf hive.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-log4j.properties --hiveconf hive.exec.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675
Hello, I would like to know if this CVE which impacts Apache Hadoop is already resolve into HDP or CDP products ? Apache Hadoop’s FileUtil.unTar(File, File) API does not escape the input file name before being passed to the shell. An attacker can inject arbitrary commands.Versions affected: 2.0.0 to 2.10.1, 3.0.0-alpha1 to 3.2.3, 3.3.0 to 3.3.2 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-25168 And, do we have any precautions other than upgrading?Thanks in advance for your help.
Hello, I have a 3 node NiFi Cluster up and running. The Initial Admin User is able now to successfully log into the NiFi cluster. I would now like to add new users to the NiFi cluster and SSL i.e., signed PKI certs for each user as the basis for these users to gain access to NiFi. I do not want to use LDAP, I am in an environment that will require use of PKI certs for access to NiFi. Can someone provide a prescriptive set of steps I can follow to successfully use PKI certs/SSL as a means of providing access to new NiFi cluster users and specifically, how do I add new users? I would think the process of creating new users and using SSL would be explained explicitly. Can someone help me with this? VR, Dave
Dears, How can I extract current time in Nifi and load into a column of a table in database Thankyou!
Hi, I want to convert the data from JSON to Avro using the ConvertRecord processor. This is my flow and the error I am getting: I have a docker image for the confluent schema registry. The curl for registering the schema in the schema registry is :curl --location --request GET 'http://localhost:8085/schemas/ids/42' \--header 'Content-Type: application/vnd.schemaregistry.v1+json' \--data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"username\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"}' Generate flow file configurations:convertRecord Configurations: AvroRecordSetWriter configurations:JsonTreeReader configurations:Confluent Schema Registry :
Hello NiFi Community, I want to convert decimal degree coordinates (e.g. -45.123658) to radians, and I want to use the built-in NiFi math function to do so. However, simply following the simple example in the Expression Language documentation for math, the example fails. I keep getting an "unexpected token" error whenever trying to use toDouble or the math function itself. Anyone have any success with using the math function and any of the methods associated with it, particularly if they take a double as an argument? Since the function is subjectless, I'm : ${math("toRadians", ${latitude:toDouble()})} But I've also tried variations of: ${latitude:toDouble():math("toRadians", ${latitude})}${latitude:math("toRadians", ${latitude:toDouble()})} I appreciate any insight.
When we enabled HTTPS with Nifi using certificates. Do we know if NIFI is using SSL or TLS, and what version of SSL or TLS is being used. Is there a way to dictate TLS 1.2 for NIFI to use?
Hi All, Looking for some help with JOLT transformatio, which i am completely new.my input JSON looks like this { "type": "delete", "timestamp": 1681484884000, "binlog_filename": "bin.000003", "binlog_position": 6378700, "database": "db1", "table_name": "records", "table_id": 108, "columns": [{ "id": 1, "name": "id", "column_type": 4, "value": 259 }, { "id": 2, "name": "index_id", "column_type": 4, "value": 1 }, { "id": 3, "name": "state_id", "column_type": 4, "value": 1 }]}This is from mysql binlogs. I want to transform it to have a single field.. [{ "id": 259}] from one of the tutorials here in cloudera comminity, i saw this JOLT transformation.[ { "operation": "shift", "spec": { "columns": { "*": { "@(value)": "[#1].@(1,name)" } } } }] But it gives all fileds.. but I just need one filed which is the ID. I need to drop rest all fields.. Can someone please help in this regards.
Hi, trying to set placement rules as described in https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/yarn-allocate-resources/topics/yarn-placement-rule-policies.html, but tasks still in default queue.Also tried to enable tag-based authorization with the same result.What could be wrong?
Hi All,I am trying to f=import the data from oracle database and writing the data to hdfs using pyspark.Oracle has 480 tables i am creating a loop over list of tables but while writing the data into hdfs spark taking too much time.when i check in logs only 1 executor is running while i was passing --num-executor 4.here is my code # oracle-example.pyfrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContextappName = "PySpark Example - Oracle Example"master = "yarn"spark = SparkSession.builder.master(master).appName(appName).enableHiveSupport().getOrCreate()spark.sparkContext.getConf().getAll()#to get the list of tables present in schemasql = "SELECT table_name FROM all_tables WHERE owner = '**'"user = "**"password = "**"jdbc_url = "jdbc:oracle:thin:@****/**"# Change this to your Oracle's details accordinglyserver = "**"port = **service_name = '**'jdbcDriver = "oracle.jdbc.OracleDriver"# Create a data frame by reading data from Oracle via JDBC to get the list of tables
Hi All, I am new to CDF Nifi(1.18) and it doesn't support API calls I wanted to save variable data as the link of the flow. I need the processor ID to concatenate it to the link of the Nifi. In HDF I was able to do that using API calls in bulk but in CDF I am not able to do due to the limitations of API. Can we do anything to fetch the processor ID of the current processor group? CFM: 2.2.6.0Nifi: 1.18.0Public Cloud Thanks in advance
Hello, I'm trying to add a new kms service, but the "test connection" is returning this error:HTTP Status 403 – Forbidden The server understood the request but refuses to authorize it.GSSException: No valid credentials providedthe users configured in the keytab file are : HTTP and ranger-admin for ranger admin serverHTTP and ranger-kms for ranger kms server this is my configuration: /opt/ranger-2.3.0-kms/ews/webapp/WEB-INF/classes/conf.dist/kms-site.xml: <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
I need to provide "Storage credentials" in "AzureStorageCredentialsControllerService_v12" processor. I am using "AzureStorageCredentialsControllerService_v12" to provide "Storage Account Name" and "SAS Token" by hardcoding in it. But the requirement is I need to provide this credentials using "Parameter context". I understand "Expression language Scope" is not supported in "PutAzureBlobStorage_v12" processor.How should I solve my requirement? Any suggestions will be helpful. Thankyou.
I am using ExecuteSQL Nifi processor to read from a Postgresql table defined as per below:CREATE TABLE otherdatatype( column1 int, column2 varchar(255), column3 bytea);I get the values of that table successfully, as per below:column1 : 1column2 : "hello world"column3 : \x4920616d206261636b20616761696e2e2e2e (converted to text "I am back again...") and the AVRO schema automatically generated by ExecuteSQL Nifi processor is:{ "type": "record", "name": "otherdatatype", "namespace": "any.data", "fields": [{ "name": "column1", "type": ["null", "int"] }, { "name": "column2", "type": ["null", "string"] }, { "name": "column3", "type": ["null", "bytes"] }]} Then I try to use PutDatabaseRecord Nifi processor to insert the values already read into another table, as per below:The AvroReaderInferSchema record reader property is
When setting hue ldap integration, I would like to know why the setting is not applied when unchecked like the first picture I confirmed that it had to be put in the snippet like the second picture to be applied.Aren't they the same configuration value? 1. 2.
hi community, Do you know some procedure to apply the comulative hotfix in cloudera 7.7.3? Thanks
I need to practice Cloudera CDP and I didn't have enough resources so I registered to Cloudera multi cloud free trial 60 but I didn't get access until now
I am trying to connect remote ImpalaDB in my springboot application. In pom.xml I am having following dependency for impalaDB:<dependency> <groupId>com.cloudera</groupId> <artifactId>ImpalaJDBC41</artifactId> <version>2.6.4.1005</version></dependency> and in application.properties: server.port=9090spring.application.name=ImpalaConnectspring.cloud.config.enabled=falsespring.datasource.driver-class-name=com.cloudera.impala.jdbc41.Driverspring.datasource.url=jdbc:impala://remote-impala-address:21050;AuthMech=3;UID=user_id;PWD=@PassWord;SSL=1;AllowSelfSignedCerts=1; when I run the application , its giving me following error: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500592) Authentication failed..at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) ~[ImpalaJDBC41-2.6.4
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.