Q&A

Hello, I would like to know if this CVE which impacts Apache Hadoop is already resolve into HDP or CDP products ? Apache Hadoop’s FileUtil.unTar(File, File) API does not escape the input file name before being passed to the shell. An attacker can inject arbitrary commands.Versions affected: 2.0.0 to 2.10.1, 3.0.0-alpha1 to 3.2.3, 3.3.0 to 3.3.2 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-25168 And, do we have any precautions other than upgrading?Thanks in advance for your help.

Hello, I have a 3 node NiFi Cluster up and running.  The Initial Admin User is able now to successfully log into the NiFi cluster.   I would now like to add new users to the NiFi cluster and SSL i.e., signed PKI certs for each user as the basis for these users to gain access to NiFi. I do not want to use LDAP, I am in an environment that will require use of PKI certs for access to NiFi. Can someone provide a prescriptive set of steps I can follow to successfully use PKI certs/SSL as a means of providing access to new NiFi cluster users and specifically, how do I add new users? I would think the process of creating new users and using SSL would be explained explicitly.   Can someone help me with this? VR, Dave

Dears,    How can I extract current time in Nifi and load into a column of a table in database Thankyou!

Hi, I want to convert the data from JSON to Avro using the ConvertRecord processor. This is my flow and the error I am getting:  I have a docker image for the confluent schema registry. The curl for registering the schema in the schema registry is :curl --location --request GET 'http://localhost:8085/schemas/ids/42' \--header 'Content-Type: application/vnd.schemaregistry.v1+json' \--data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"username\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"}' Generate flow file configurations:convertRecord Configurations: AvroRecordSetWriter configurations:JsonTreeReader configurations:Confluent Schema Registry : 

Hello NiFi Community, I want to convert decimal degree coordinates (e.g. -45.123658) to radians, and I want to use the built-in NiFi math function to do so.  However, simply following the simple example in the Expression Language documentation for math, the example fails.  I keep getting an "unexpected token" error whenever trying to use toDouble or the math function itself.  Anyone have any success with using the math function and any of the methods associated with it, particularly if they take a double as an argument? Since the function is subjectless, I'm : ${math("toRadians", ${latitude:toDouble()})} But I've also tried variations of: ${latitude:toDouble():math("toRadians", ${latitude})}${latitude:math("toRadians", ${latitude:toDouble()})} I appreciate any insight.  

When we enabled HTTPS with Nifi  using certificates.  Do we know if NIFI is using SSL or TLS, and what version of SSL or TLS is being  used.   Is there a way to dictate TLS 1.2 for NIFI to use?

Hi All, Looking for some help with JOLT transformatio, which i am completely new.my input JSON looks like this  { "type": "delete", "timestamp": 1681484884000, "binlog_filename": "bin.000003", "binlog_position": 6378700, "database": "db1", "table_name": "records", "table_id": 108, "columns": [{ "id": 1, "name": "id", "column_type": 4, "value": 259 }, { "id": 2, "name": "index_id", "column_type": 4, "value": 1 }, { "id": 3, "name": "state_id", "column_type": 4, "value": 1 }]}This is from mysql binlogs. I want to transform it to have a single field.. [{ "id": 259}] from one of the tutorials here in cloudera comminity, i saw this JOLT transformation.[ { "operation": "shift", "spec": { "columns": { "*": { "@(value)": "[#1].@(1,name)" } } } }] But it gives all fileds.. but I just need one filed which is the ID. I need to drop rest all fields.. Can someone please help in this regards.

Hi, trying to set placement rules as described in https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/yarn-allocate-resources/topics/yarn-placement-rule-policies.html, but tasks still in default queue.Also tried to enable tag-based authorization with the same result.What could be wrong?

Hi All,I am trying to f=import the data from oracle database and writing the data to hdfs using pyspark.Oracle has 480 tables i am creating a loop over list of tables but while writing the data into hdfs spark taking too much time.when i check in logs only 1 executor is running while i was passing --num-executor 4.here is my code  # oracle-example.pyfrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContextappName = "PySpark Example - Oracle Example"master = "yarn"spark = SparkSession.builder.master(master).appName(appName).enableHiveSupport().getOrCreate()spark.sparkContext.getConf().getAll()#to get the list of tables present in schemasql = "SELECT table_name FROM all_tables WHERE owner = '**'"user = "**"password = "**"jdbc_url = "jdbc:oracle:thin:@****/**"# Change this to your Oracle's details accordinglyserver = "**"port = **service_name = '**'jdbcDriver = "oracle.jdbc.OracleDriver"# Create a data frame by reading data from Oracle via JDBC to get the list of tables

Hi All, I am new to CDF Nifi(1.18) and it doesn't support API calls I wanted to save variable data as the link of the flow. I need the processor ID to concatenate it to the link of the Nifi. In HDF I was able to do that using API calls in bulk but in CDF I am not able to do due to the limitations of API. Can we do anything to fetch the processor ID of the current processor group?  CFM: 2.2.6.0Nifi: 1.18.0Public Cloud Thanks in advance

Hello, I'm trying to add a new kms service, but the "test connection" is returning this error:HTTP Status 403 – Forbidden The server understood the request but refuses to authorize it.GSSException: No valid credentials providedthe users configured in the keytab file are : HTTP and ranger-admin for ranger admin serverHTTP and ranger-kms for ranger kms server this is my configuration: /opt/ranger-2.3.0-kms/ews/webapp/WEB-INF/classes/conf.dist/kms-site.xml:  <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

I need to provide "Storage credentials" in "AzureStorageCredentialsControllerService_v12" processor. I am using "AzureStorageCredentialsControllerService_v12" to provide "Storage Account Name" and "SAS Token" by hardcoding in it. But the requirement is I need to provide this credentials using "Parameter context". I understand "Expression language Scope" is not supported in "PutAzureBlobStorage_v12" processor.How should I solve my requirement? Any suggestions will be helpful. Thankyou.

I am using ExecuteSQL Nifi processor to read from a Postgresql table defined as per below:CREATE TABLE otherdatatype(    column1 int,    column2 varchar(255),    column3 bytea);I get the values of that table successfully, as per below:column1 : 1column2 : "hello world"column3 : \x4920616d206261636b20616761696e2e2e2e (converted to text "I am back again...") and the AVRO schema automatically generated by ExecuteSQL Nifi processor is:{  "type": "record",  "name": "otherdatatype",  "namespace": "any.data",  "fields": [{    "name": "column1",    "type": ["null", "int"]  }, {    "name": "column2",    "type": ["null", "string"]  }, {    "name": "column3",    "type": ["null", "bytes"]  }]} Then I try to use PutDatabaseRecord Nifi processor to insert the values already read into another table, as per below:The AvroReaderInferSchema record reader property is

When setting hue ldap integration, I would like to know why the setting is not applied when unchecked like the first picture I confirmed that it had to be put in the snippet like the second picture to be applied.Aren't they the same configuration value? 1.    2. 

hi community,   Do you know some procedure to apply the comulative hotfix in cloudera 7.7.3?    Thanks           

I need to practice Cloudera CDP and I didn't have enough resources so I registered to Cloudera multi cloud free trial 60  but I didn't get access until now 

I am trying to connect remote ImpalaDB in my springboot application. In pom.xml I am having following dependency for impalaDB:<dependency> <groupId>com.cloudera</groupId> <artifactId>ImpalaJDBC41</artifactId> <version>2.6.4.1005</version></dependency>  and in application.properties: server.port=9090spring.application.name=ImpalaConnectspring.cloud.config.enabled=falsespring.datasource.driver-class-name=com.cloudera.impala.jdbc41.Driverspring.datasource.url=jdbc:impala://remote-impala-address:21050;AuthMech=3;UID=user_id;PWD=@PassWord;SSL=1;AllowSelfSignedCerts=1; when I run the application , its giving me following error: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500592) Authentication failed..at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) ~[ImpalaJDBC41-2.6.4

Hi Team, I have configured the ldap connection for nifi login, when I try to access the nifi in web UI I am getting error like "Insufficient Permission Unknow user with identity 'xyz123' Contact the system administrator". Authentication is happing successfully, but authorization is not happening could you please advice.  

Hi there,So, I need your experience with a flow I have been struggling with.My Flow is as follows: ConsumeKafka --> ConvertRecord --> ConvertJSONToSQL --> PutSQL.I am extracting some data out of some kafka brokers, which come in the following format:  {"Column1": "1e39c17d25cb420e8e720aa4b1ae005e","TimeStamp": "2023-04-10T10:43:15.794241429+03:00","Column3": "some_string","Column4": false,"Column5": "some_string","Column6": "some_string","Column7": false,"Column8": "some_string","Column9": "","Column10": 0,"Column11": 0,"Column12": "","Column13": 0,"Column14": 137,"Column15": "","Column16": 0,"Column17": 0,"Column18": "","Column19": 138,"Column20": 1550,"Column21": "some_string","Column22": ""}   I am using afterwards an ConverRecord, with an RecordReader = JsonTreeReader (with default configuration: Infer Schema and Root Node) and an RecordWriter = JsonRecordSetWriter (with default configuration: Inherit Record Schema and Pretty print json = false). T

I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic.  I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck. Here are my configurations for the ExecuteStreamCommand:Command Arguments Strategy: Command Arguments PropertyCommand Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topicsCommand Path: curlIgnore STDIN: falseArgument Delimiter: " " (Sp

Hello, I'm using the InvokeHTTP processor to make a GET request from an API, however, before getting access to the API, there's a keycloak sign-in page that I need to bypass and the InvokeHTTP process returns that page as a csv response. Is the a way to sign-in or to use a token when making the GET request so I can make the right API call.

 Caused by: java.lang.Exception: The specified login identity provider class 'org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider' is not known to this nifi. I am trying to set up a nifi instance in a Ubuntu Server 22.04 machine.It's been hours I've been tinkering with the configurations and xml files and still I'm getting the error above.These are the configurations I am passing in the nifi.properties, authorizers.xml, login-identity-providers.xml and authorizations.xml files are in the screenshot below. I have tried specifying a user and password in the identity providers file but still nothing... 

Im trying to make kafka ranger plugin communicate with the policy manager url over ssl, i have the truststore and keystore jks file but im getting PKIX path building failed, i know this error usually is visible when the files are not in the right place. I am using original ranger repository, please let me know where are we supposed to keep the files?

Hi,I am using distcp to copy data from hadoop hdfs to s3. below is the shorthand command of what i use hadoop distcp -pu -update -delete hdfs_path s3a://bucket recently got into an issue with the below case i have a file in hdfs -> temp_file with data 1234567890 with size 27kbfor the first time when i use distcp. it pushes the file to s3 bucket without any issue. second time i update the same file temp_file with different content abcdefghij but with same size 27kbnow when i run distcp. instead of checking the checksum of source and target distcp skips the file directly and doesnt copy the updated file from hdfs to s3 Am i missing any options in distcp command to make this scenario work? 

Hi, We are looking to migrate our data from SQL server to CDP environment and would like some insights on getting this done. Would appreciate any help/guidance in the same. Thanks Amn