Solved

Dears,    How can I extract current time in Nifi and load into a column of a table in database Thankyou!

Hi, I want to convert the data from JSON to Avro using the ConvertRecord processor. This is my flow and the error I am getting:  I have a docker image for the confluent schema registry. The curl for registering the schema in the schema registry is :curl --location --request GET 'http://localhost:8085/schemas/ids/42' \--header 'Content-Type: application/vnd.schemaregistry.v1+json' \--data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"username\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"}' Generate flow file configurations:convertRecord Configurations: AvroRecordSetWriter configurations:JsonTreeReader configurations:Confluent Schema Registry : 

Hello NiFi Community, I want to convert decimal degree coordinates (e.g. -45.123658) to radians, and I want to use the built-in NiFi math function to do so.  However, simply following the simple example in the Expression Language documentation for math, the example fails.  I keep getting an "unexpected token" error whenever trying to use toDouble or the math function itself.  Anyone have any success with using the math function and any of the methods associated with it, particularly if they take a double as an argument? Since the function is subjectless, I'm : ${math("toRadians", ${latitude:toDouble()})} But I've also tried variations of: ${latitude:toDouble():math("toRadians", ${latitude})}${latitude:math("toRadians", ${latitude:toDouble()})} I appreciate any insight.  

When we enabled HTTPS with Nifi  using certificates.  Do we know if NIFI is using SSL or TLS, and what version of SSL or TLS is being  used.   Is there a way to dictate TLS 1.2 for NIFI to use?

Hi All, Looking for some help with JOLT transformatio, which i am completely new.my input JSON looks like this  { "type": "delete", "timestamp": 1681484884000, "binlog_filename": "bin.000003", "binlog_position": 6378700, "database": "db1", "table_name": "records", "table_id": 108, "columns": [{ "id": 1, "name": "id", "column_type": 4, "value": 259 }, { "id": 2, "name": "index_id", "column_type": 4, "value": 1 }, { "id": 3, "name": "state_id", "column_type": 4, "value": 1 }]}This is from mysql binlogs. I want to transform it to have a single field.. [{ "id": 259}] from one of the tutorials here in cloudera comminity, i saw this JOLT transformation.[ { "operation": "shift", "spec": { "columns": { "*": { "@(value)": "[#1].@(1,name)" } } } }] But it gives all fileds.. but I just need one filed which is the ID. I need to drop rest all fields.. Can someone please help in this regards.

Hi All,I am trying to f=import the data from oracle database and writing the data to hdfs using pyspark.Oracle has 480 tables i am creating a loop over list of tables but while writing the data into hdfs spark taking too much time.when i check in logs only 1 executor is running while i was passing --num-executor 4.here is my code  # oracle-example.pyfrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContextappName = "PySpark Example - Oracle Example"master = "yarn"spark = SparkSession.builder.master(master).appName(appName).enableHiveSupport().getOrCreate()spark.sparkContext.getConf().getAll()#to get the list of tables present in schemasql = "SELECT table_name FROM all_tables WHERE owner = '**'"user = "**"password = "**"jdbc_url = "jdbc:oracle:thin:@****/**"# Change this to your Oracle's details accordinglyserver = "**"port = **service_name = '**'jdbcDriver = "oracle.jdbc.OracleDriver"# Create a data frame by reading data from Oracle via JDBC to get the list of tables

I need to provide "Storage credentials" in "AzureStorageCredentialsControllerService_v12" processor. I am using "AzureStorageCredentialsControllerService_v12" to provide "Storage Account Name" and "SAS Token" by hardcoding in it. But the requirement is I need to provide this credentials using "Parameter context". I understand "Expression language Scope" is not supported in "PutAzureBlobStorage_v12" processor.How should I solve my requirement? Any suggestions will be helpful. Thankyou.

I am using ExecuteSQL Nifi processor to read from a Postgresql table defined as per below:CREATE TABLE otherdatatype(    column1 int,    column2 varchar(255),    column3 bytea);I get the values of that table successfully, as per below:column1 : 1column2 : "hello world"column3 : \x4920616d206261636b20616761696e2e2e2e (converted to text "I am back again...") and the AVRO schema automatically generated by ExecuteSQL Nifi processor is:{  "type": "record",  "name": "otherdatatype",  "namespace": "any.data",  "fields": [{    "name": "column1",    "type": ["null", "int"]  }, {    "name": "column2",    "type": ["null", "string"]  }, {    "name": "column3",    "type": ["null", "bytes"]  }]} Then I try to use PutDatabaseRecord Nifi processor to insert the values already read into another table, as per below:The AvroReaderInferSchema record reader property is

When setting hue ldap integration, I would like to know why the setting is not applied when unchecked like the first picture I confirmed that it had to be put in the snippet like the second picture to be applied.Aren't they the same configuration value? 1.    2. 

hi community,   Do you know some procedure to apply the comulative hotfix in cloudera 7.7.3?    Thanks           

I need to practice Cloudera CDP and I didn't have enough resources so I registered to Cloudera multi cloud free trial 60  but I didn't get access until now 

Hi there,So, I need your experience with a flow I have been struggling with.My Flow is as follows: ConsumeKafka --> ConvertRecord --> ConvertJSONToSQL --> PutSQL.I am extracting some data out of some kafka brokers, which come in the following format:  {"Column1": "1e39c17d25cb420e8e720aa4b1ae005e","TimeStamp": "2023-04-10T10:43:15.794241429+03:00","Column3": "some_string","Column4": false,"Column5": "some_string","Column6": "some_string","Column7": false,"Column8": "some_string","Column9": "","Column10": 0,"Column11": 0,"Column12": "","Column13": 0,"Column14": 137,"Column15": "","Column16": 0,"Column17": 0,"Column18": "","Column19": 138,"Column20": 1550,"Column21": "some_string","Column22": ""}   I am using afterwards an ConverRecord, with an RecordReader = JsonTreeReader (with default configuration: Infer Schema and Root Node) and an RecordWriter = JsonRecordSetWriter (with default configuration: Inherit Record Schema and Pretty print json = false). T

I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic.  I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck. Here are my configurations for the ExecuteStreamCommand:Command Arguments Strategy: Command Arguments PropertyCommand Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topicsCommand Path: curlIgnore STDIN: falseArgument Delimiter: " " (Sp

 Caused by: java.lang.Exception: The specified login identity provider class 'org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider' is not known to this nifi. I am trying to set up a nifi instance in a Ubuntu Server 22.04 machine.It's been hours I've been tinkering with the configurations and xml files and still I'm getting the error above.These are the configurations I am passing in the nifi.properties, authorizers.xml, login-identity-providers.xml and authorizations.xml files are in the screenshot below. I have tried specifying a user and password in the identity providers file but still nothing... 

Im trying to make kafka ranger plugin communicate with the policy manager url over ssl, i have the truststore and keystore jks file but im getting PKIX path building failed, i know this error usually is visible when the files are not in the right place. I am using original ranger repository, please let me know where are we supposed to keep the files?

Hi,I am using distcp to copy data from hadoop hdfs to s3. below is the shorthand command of what i use hadoop distcp -pu -update -delete hdfs_path s3a://bucket recently got into an issue with the below case i have a file in hdfs -> temp_file with data 1234567890 with size 27kbfor the first time when i use distcp. it pushes the file to s3 bucket without any issue. second time i update the same file temp_file with different content abcdefghij but with same size 27kbnow when i run distcp. instead of checking the checksum of source and target distcp skips the file directly and doesnt copy the updated file from hdfs to s3 Am i missing any options in distcp command to make this scenario work? 

Hi, We are looking to migrate our data from SQL server to CDP environment and would like some insights on getting this done. Would appreciate any help/guidance in the same. Thanks Amn

I am having a HDP hadoop cluster. I took snapshot of phoenix table from this cluster & I cloned that table to a CDP cluster. Hbase table was created from the snapshot & the hfiles are imported to the table's data directory . While selecting the table from phoenix shell, empty data it is showing. If I do a scan operation from hbase shell data is displaying (encoded). Is there any workaround for this issue. Thanks

I had a 3 node NiFi cluster and created a NiFi registry client with this a URL that goes through Knox and looks something like this https://xyz.com/gateway/registry/nifi-registry-app/nifi-registry-api. However, when I restart NiFi the registry client in NiFi strips the URL path so it comes back as https://xyz.com which isn't correct and can't connect to the registry. Has anyone experienced this before and/or solved this issue?

Hi there, any support would be much appreciated. First things first, here is our setup. We are running nifi on aws ec2 in a docker container. It's a two node cluster setup. We are planning to do an in place upgrade by upgrading the docker container and changing the nifi.properties parameters needed to get 1.19 running. changing the port number to 8443using https instead of httpsetting the sensitive properties, was empty in 1.13we are using the web.proxy.host with 3 entries, the hostname of the 2 nodes and the clusternamewe are running anonymously, without ldap. We'll set this up later.We made a backup of the flow.xml.gz file and deleted the original. We are planning to use the nifi-registry to import our flow on the clean canvas. So this works great! We have succesfully upgraded to 1.19. Now what we're trying to figure out is our fallback plan, just in case upgrading doesn't work in the production environment.When reverting all the changes above and deleting the flow.xml.gz,

I have Installed CDP Private Cloud 7.1.7getting 500 error code while accessing 'http://hostname:50111/templeton/v1/status'  

Hello, i'm using the cetic helm chart to deploy nifi to GCP gke cluster. I have configured SSL and OIDC with azure AD, i was able to see the AD page and log in, after that i'm seeing this error  Values:   replicaCount: 1image: repository: apache/nifi tag: "1.16.3" pullPolicy: "IfNotPresent"securityContext: runAsUser: 1000 fsGroup: 1000sts: podManagementPolicy: Parallel AntiAffinity: soft useHostNetwork: null hostPort: null pod: annotations: security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000 #prometheus.io/scrape: "true" serviceAccount: create: false #name: nifi annotations: {} hostAliases: [] startupProbe: enabled: false failureThreshold: 60 periodSeconds: 10properties: sensitiveKey: changeMechangeMe externalSecure: true isNode: false httpsPort: 8443 httpPort: 8080 webProxyHost: nifi-test.qa.cloud.paycomplete.com clusterPort: 6007 provenanceStorage: "8 GB" siteToSite: port: 100

Hello TeamI have a small development POC HDP Cluster installed on 4 machinesI was using the Public Repositories of Hortonworks/Cloudera that are no longer validhttp://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/repodata/repomd.xmlQuote"Starting with the HDP 3.1.5 release and Ambari 2.7.5 access to HDP, Ambari repositories requires authentication. Authentication credentials for new customers are sent sent from Cloudera to registered support contacts"To be a registered support contact I suppose one needs a subscription. How do I obtain the subscription for access to the repositories and how much does it cost annually and monthlyapproximately. Are they still available for legacy HDP and Ambari Platforms

Hello friends, I wrote a custom processor and now wanted to add a custom controller. The problem is, that the properties I wrote for the controller do not appear when I create the service. I followed the structure of this controller: https://github.com/Automna/NiFi/blob/master/nifi-rel-nifi-1.1.2/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java  Here is what I did so far:public class MyNewControllerService extends AbstractControllerService implements MyControllerService { public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder() .name("Username") .expressionLanguageSupported(ExpressionLanguageScope.NONE) .displayName("Set username") .required(true) .defaultValue("User") .description("Username to login to graph service") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)

Hello All, I have configured nifi flow with three node with external zookeepernode1.domain.comnode2.domain.comnode3.domain.com with three zookeeper nodeszk1.domain.comzk2.domain.comzk3.domain.com Everything is working fine but the requirement here is my flow has trigger points from external client systems to whom I have give a common URL likenode.domain.com to avoid any changes on the client teams if any of nodes goes down. Any idea how to configure this.