Q&A

Hi All,I hope you are doing well. Please help me with below request.Currenty our client is using MS Dynamics and they gave basic details required to fetch token.In Postman - 1. I'm choosing OAuth 2.0 in Authorization.                     2. Giving the basic details(shared below) it asked for.                     3. Then clicking on "Get New Access Token"                     4. New window pops up to enter my Username and Password.                     5. Post validating Creds, it gives the Token in another window.Problem : All the steps which I'm doing in postman, I want to implement the same in Apache Nifi(to fetch the Token). Basically I need an API along with a request payload to fetch the token.(I'm worried if it is possible to do so in one call, as we need to gi

Hi Everyone, While kerberizing my cluster using MIT-KDC and Ambari Kerberos Wizard. I am facing the following window popup at the time of Testing client after client installation saying :I'm using ambari 2.7.8HDFS 3.3.6UBUNTU 22config of krb5.conf[logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log[libdefaults]default_realm = EXAMPLE.COMdns_lookup_realm = falsedns_lookup_kdc = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = true[realms]EXAMPLE.COM = {kdc = admin.comadmin_server = admin.com}[domain_realm].example.com = EXAMPLE.COMexample.com = EXAMPLE.COMconfig of /etc/hostnamelist principal isroot@admin:/# kadmin.local -q "listprincs"Authenticating as principal admin/admin@EXAMPLE.COM with password.K/M@EXAMPLE.COMadmin/admin@EXAMPLE.COMkadmin/admin@EXAMPLE.COMkadmin/changepw@EXAMPLE.COMkrbtgt/EXAMPLE.COM@EXAMPLE.COM ambari-server.log:KERBEROS_SERVICE_CHECK, inputs :{HAS_RESOURCE_FILTERS=true}, resource

Is there any way which we can pass dynamic CRON expression as a run schedule input to invoke HTTP processor (using NiFi 1.19.1).Because I have CRON expression which is stored in DB. Where I am fetching it through ( executesql -> convertAvroToJson -> SplitJson -> Evaluate JsonPath (schedule: $.schedule)).If I add ${schedule} in invoke HTTP processor getting Unexpected character error.Looking for expert's help @MattWho @SAMSAL @cotopaul  

Is there a way to import backups in nifiregistry which will store by default .snapshot format?@MattWho @TimothySpann @Nifi 

Hi experts,I was wondering what is the best way to troubleshoot an application or job that is taking longer than usual. Maybe a 5minute job that is taking 1 hour or longer to complete etc.What are some things I should start looking at first? Or can someone bring me through the process? Thanks,

Hi, I'm facing some data loss while doing the insert overwrite through multiple hive views and in case of some column qualifiers data is getting rolled back . Can we do data upserts through multiple hive views on single hbase table. Can anyone help me to understand this issue.

Hello there!We create a hive external table that refers to a HBase table:CREATE EXTERNAL TABLE  default.c_COUNTRIES_AP(               key         BINARY,               CF1_COUNTRY_ID_1       BINARY,               CF1_COUNTRY_NAME_2              BINARY,               CF1_COUNTRY_REGION_3           BINARY,               CF1_COUNTRY_SUBREGION_4    BINARY )STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.c

Hi ,I'm having a hard time writing my custom processor using python extension. it seems this feature is not mature as I thought or maybe I'm miss understanding something. To summarize I have encountered two issues but they are both related to the same error, as follows:1- I was experimenting with developing a custom processor using the RecordTransform class to partition a json array, my  processor is simple, it takes json array of records where each record is a json object that has a "name" and "age" fields. The processor then try to return RecordTransformResult from the transform method as follows: def transform(self, context, record, schema, attributemap): return RecordTransformResult(schema=None,record=record,relationship="success",partition={'name':record['name']}) I'm trying to use partition to split the record and passing the partition key just as described in the developer guide:https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.htm

Hi,Im trying to write my own python extension that takes an excel bytes and produce an output flowfile for each sheet represented in html. Im using pandas dataframe for the conversion. I have been looking to the python extension developer guide but I cant find anything that will point me in the right direction:https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/python-developer-guide.html#record-transformThe TransformRecord talks about partitioning flowfile input but it seems like it has to be nifi readable format (json, csv , avro...etc).In executescript processor you could have easily generated multiple flow files by passing array of flowfiles to the session.transfer method:https://community.cloudera.com/t5/Support-Questions/Split-one-Nifi-flow-file-into-Multiple-flow-file-based-on/td-p/203387However neither RecordTransformResult  nor FlowFileTransformResult can do that.Can someone provide me with the code if applicable.Thanks 

Hi Cloudera,I am developing a custom processor which uses nifi inbuilt processors as dependencies in pom.xml. This is creating a duplicate processor of existing nifi standard processors with my custom nar version in the NiFi(Please refer to the attachment picture). Is there a way to include these dependencies without bundling them with my custom processor nar. I have tried added the dependencies as `provided` but that is causing the build to fail. I want these dependencies to be referenced from the lib folder where I have placed my nar file.Here are my pom.xml files1. Main root pom.xml  <?xml version="1.0" encoding="UTF-8"?><project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.apache.nifi</groupId>

This is the query which is running on metastore_dbSELECT`A0` . `PART_NAME`FROM`PARTITIONS` `A0`LEFT OUTER JOIN`TBLS` `B0`ON`A0` . `TBL_ID` = `B0` . `TBL_ID`LEFT OUTER JOIN`DBS` `C0`ON`B0` . `DB_ID` = `C0` . `DB_ID`WHERE`C0` . `NAME` = ?AND `C0` . `CTLG_NAME` = ?AND `B0` . `TBL_NAME` = ?AND `A0` . `PART_NAME` LIKE ? ESCAPE ?how we can optimize it ? Is it system generated?or from where it is coming at metastore_db database that is inside mysql.SELECT"AO""PART NAME' FROM 'PARTITIONSAO' LEFT OUTER JOIN 'TBLS"RO' ON 'AO'• 'TBL ID'** Note ** I am using Trino hive architecture. Thanks in advance for your help....

Hi everyone, I'm trying to use rest api in a cloudera cluster with ssl and kerberos. I am testing the use of the same by authenticating with a bearer token to gain access to the resource. Below is what is used and working: curl 'https://nifi-node:8443/nifi-api/access/token' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username=myuserad&password=mypasswordad' --compressed --cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pemcurl -H 'Authorization: Bearer token generated 'Content-Type: application/json' -XPUT -d '{"id":"****","state":"RUNNING"}' https://nifi-node/nifi-api/flow/process-groups/****--cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pemTo avoid entering the password in clear text as in curl N.1 I am testing the token generation via Kerberos: curl -X POST --negotiate -u : https://nifi-node:8443/nifi-api/access/kerberos --cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts .pem Using t

Hello Community,I have an old CDH6 and realized the following suspicious traces in cloudera.flood.log in /var/log/cloudera-scm-serverdht_pkt_alert ==> [82.43.248.101:6881]dht_pkt_alert <== [82.43.248.101:6881]and the same for another almost 100 different IPs outside the network by using the p2p libraries (and not sure that they are Cloudera Repository Ps)Did you see anything similar? this activity is quite suspicious.Thanks for your help     

Hi Team, One of our client is using Grafana as a visualization tool in HDP and we need to migrate to CDP.  Just want to understand, Does Grafana will be supported as a visualization tool in CDP?  Thanks and RegardsSunil Kumar KS

I'm trying to run  miniFi  in ubuntu. When I try to start miniFi, I got following error in bootstrap and app log. App.log2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected.2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow...2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down.java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key

Hi,Using pythong extension Im trying to write my custom processor , I noticed when I return failure rel as follows: return FlowFileTransformResult("failure", contents=None,attributes={"Feeling.Good.About.PY.Ext":"No"}) I cant find any attribute "Feeling.Good.About.PY.Ext" in the failure rel flowfile with value "No". I can guarantee that its not because the attribute name or the value 🙂 . Not sure if this is done by design or what.Any thoughts on this?I have also posted another question related to splitting byte input into multiple flowfiles here but no answer:https://community.cloudera.com/t5/Support-Questions/python-extension-generate-multiple-flowfiles-from-bytes/m-p/383095 @MattWho, @steven-matison , @cotopaul , @joseomjr Thanks

Hi fellas, I have a folder structure where the parent folder is the date which contains multiple locations folder which contains the actual txt files.So the structure is such Logfolder/20240515/Location/logs.txtMy nifi setup is to retrieve the date folders inside the parent directory of Logfolder and pass it to executestreamcommand processor which contains my python script which requires the date folders to be sent in as argument along with all the contents inside the datefolder with original directory structure.But currently the flowfile is sending logs.txt to my executestreamcommand instead of the 20240515 as the flowfile. How do I structure such that the date folders as a whole is passed to the python script and not file by file?I have tried chatgpt but it isn't quite helpful. I appreciate any help on this as it is quite urgent.

Hello, We have a strange issue here. We are on CDP7.1.9 and have ranger in place for authorizations. A user is trying to execute queries from Hue on Impala Editor and fails with Authorization errors. However, same query works fine via Hive and also via Impala shell. It fails only from Hue editor. We have:1. Refreshed metadata2. Invalidated metadata3. Verified permissions are in place in Ranger4. There are no logs generated (I checked in Hue, Sentry Server and Catalog server). Please suggest if any other place should also be checked. Example query:select * from DB.table limit 5; Kindly help to diagnose and resolve this issue. 

I am getting the below error while connecting to hive using jdbc urlCaused by: com.cloudera.hiveserver2.support.exceptions.GeneralException: [Cloudera][HiveJDBCDriver](500171) Error occurred Connecting to ZooKeeper: At: <my_host>:2181/hiveserver2With Error Message: KeeperErrorCode = AuthFailed for /hiveserver2.JDBC URL:jdbc:hive2://zk=<my_host>:2181/hiveserver2,<my_host>:2181/hiveserver2;SSL=1;SSLTrustStore=/trustore_path/cm-auto-global_truststore.jks;SSLTrustStorePwd=My_pwd;AuthMech=1;KrbRealm=MYREAL.ORG.COM;KrbHostFQDN=_HOST;KrbServiceName=hive;LogPath=/tmp/log

I am trying to access the hadoop filesystem but I am facing below error.How to provide the access right to the user.even getfacl command also giving same error.I have tried the kinit command user has valid TGT.My main objective is to use the HadoopDelegation token.but code is failing even when tried to just list the hdfs files.FileSystem fs = FileSystem.get(conf);Path root = new Path("/");System.out.println((root));fs.listStatus(root); ----- code failing on this lineERROR message :java -jar HadoopDelegation-1.0-SNAPSHOT.jarlog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.java.io.IOException: DestHost:destPort master.localdomain.com:8020 , LocalHost:localPort master.localdomain.com/10.49.0.149:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Cl

HDFS cluster in HA enabled with one active namenode and one standby namenode. During checkpointing in how editlogs and fsimage get merged ?I tired with the source code check but got confused by the flow, Can someone help on this?, how editlogs and fsimage file get merged ?

Hi All,   We are trying to install CDP 7.10.1 in RHEL 8.9 . As per the Cloudera Support Matrix i dont see it is listing . During the installation we are encountering with Mutiple issues .Kindly let me know if any one have tried with the similar installation combination . As we are mainly using the RHEL 8.9 as standard image we would like to keep the same standard for Cloudera also . Any suggestion is welcome. RegardsJiji

I have a cloudera cluster up and running. I have Knox which forwards requests to webhbase and hbase uses ranger for authorization. Ranger is connected to FreeIPA LDAP and we use kerberos internally for authentication.In Ranger, I have a policy that gives access to read a table. In that same ranger policy a group in my freeIPA instance has the ability to access that table. I am having an issue. When I remove a member from that group in freeIPA and rerun ranger user sync, I try a curl call to get data from the table it still works. However when I run it again it gives the expected result as a denial. This has been a consistent pattern for all changes to user membership in ldap groups. The Ranger HBase Policy Sync is working as expected and the Usersync is working as expected as well. After I run Usersync, I confirm that the user has been removed from that group in Ranger but it's still allowing me access.  Does anyone know why this is? There are similar Kafka and HDFS poli

Hi i have problem with NIFI-I download json records from Kafka, I split to have single records-here i have round robin strategy, I add attributes, then I give the id attribute and divide the flow into two paths, on one I do some actions with invokes http to reach some data and transform it to attributes, I add attributes, clear the context and I would like to merge it back to keep these jsons from one paths and attributes on the other. I use the merge content processor for this, merging using the id attribute that I assigned at the beginning. And now I have a problem because it works, but in the case of a larger number of records, e.g. 200, it doesn't because I have an error BIN_MANAGER_FULL. Minimum Number of Entries is 2 and Maximum Number of Entries is 2, while Maximum number of Bins is 10 and I would not like to increase it. Is there any way to bypass this and make it work for a larger number of records? For example, apply a limit so that it merges only when there is space or dived

We save some flows into a registry with postgresql and now we want to transfer to a fresh nifi instance but we are getting this error:The Flow Registry with ID ? reports that no Flow exists with Bucket ?, FlowAny suggestion what is it happening?