Solved

Input Json,==============[{"CustomFields": [{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD2","FieldName": "Business Registration No","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "BusinessRegistrationNo","Type": "inputfield","Id": 99},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD3","FieldName": "Type of Business ","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "TypeOfBusiness","Type": "dropdown","Id": 100},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD4","FieldName": "Birth Certicate No / Other","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "BirthCerticateNoOther","Type": "inputfield","Id": 101},{"PrimaryId": 2191300,"ColumnName": "TRX_CUSTOMER_CUSTOMFIELD5","FieldName": "Passport No","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "PassportNo","T

Hi Team, Zepplin is Installed in CDP which does not contain Kerberos.I tried with default username and password as admin/admin, zeplin is not working in GUI.Let me know the default password of zeplin to login 

Hello, Is there a straight documentation available that would help to side car migrate Oozie jobs (50+) from HDP to CDP PB? I know of the properties file that might need some modification to point to CDP RM and relevant servers, however, facing it hard to understand / map properties file as we have 50+ workflows to be migrated. Thankssnm1523

Hello,Can you assist me in retrieving the values stored in the "customfield_10161" field from JSON input? My expectation is to consistently receive the values and name it "DEFECT_ROOT_CAUSE" , whether they are stored as a list or as a single value."How to write JOLT which will handle the both the case. let say If I will get list then extract last element of it, if not list then record as it is.?Extracting List: (works fine when input is list)when input is not list: then DEFECT_ROOT_CAUSE column is missing My expectation to get the records in DEFECT_ROOT_CAUSE columns whether its list or not.     

HelloI am trying to set up a cluster with NiFi 2.0-M3 but have an error. My docker-compose file is here - https://hastebin.com/share/woqezesefi.yamlError:2024-06-02 08:10:54,535 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 248301603f5c:8082; will use this address for sending heartbeat messages2024-06-02 08:10:54,536 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at 248301603f5c/<unresolved>:8082. Will send Cluster Connection Request to this address2024-06-02 08:10:54,571 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchorsThe same docker-compose is working fine with

2024-05-31 19:23:17,927 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [file:/user/history/done]at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:701)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:637)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:598)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:97)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:150)at org.apache.hadoop.service.AbstractService.init(AbstractService

curl -H   "Content-Type: application/JSON ; Authorization: Bearer {token here}" -d "{"revision":{"clientId":"ac153723-44ff-17fd-e42e-f975cb01817b"},"processors":{"id":"ac15a47e-44ff-17fd-ffff-fffffc691b51","running":"true"}}" -X PUT "https://aaa.com:443/nifi-api/processors/ac15a47e-44ff-17fd-ffff-fffffc691b51/run-status" we are trying to run NIFI processor using NIFI API as above, but its giving UNAUTHORIZED error always. Can any one help in getting this fixed. Are we using correct code? We are able to get the token from NIFI.

Hello Experts,I see this red color high lighted number "2(1)"on Apache nifi processor.Is this something related to background process (processor thread) failing or something?I face the issue of this custom processor getting stuck once in a while, trying to understand the issue.This processor just invokes an http post endpoint to upload a file.Any help/suggestion is appreciated. Thanks,Mahendra 

Is there any way which we can pass dynamic CRON expression as a run schedule input to invoke HTTP processor (using NiFi 1.19.1).Because I have CRON expression which is stored in DB. Where I am fetching it through ( executesql -> convertAvroToJson -> SplitJson -> Evaluate JsonPath (schedule: $.schedule)).If I add ${schedule} in invoke HTTP processor getting Unexpected character error.Looking for expert's help @MattWho @SAMSAL @cotopaul  

Hello there!We create a hive external table that refers to a HBase table:CREATE EXTERNAL TABLE  default.c_COUNTRIES_AP(               key         BINARY,               CF1_COUNTRY_ID_1       BINARY,               CF1_COUNTRY_NAME_2              BINARY,               CF1_COUNTRY_REGION_3           BINARY,               CF1_COUNTRY_SUBREGION_4    BINARY )STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.c

Hi ,I'm having a hard time writing my custom processor using python extension. it seems this feature is not mature as I thought or maybe I'm miss understanding something. To summarize I have encountered two issues but they are both related to the same error, as follows:1- I was experimenting with developing a custom processor using the RecordTransform class to partition a json array, my  processor is simple, it takes json array of records where each record is a json object that has a "name" and "age" fields. The processor then try to return RecordTransformResult from the transform method as follows: def transform(self, context, record, schema, attributemap): return RecordTransformResult(schema=None,record=record,relationship="success",partition={'name':record['name']}) I'm trying to use partition to split the record and passing the partition key just as described in the developer guide:https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.htm

Hi everyone, I'm trying to use rest api in a cloudera cluster with ssl and kerberos. I am testing the use of the same by authenticating with a bearer token to gain access to the resource. Below is what is used and working: curl 'https://nifi-node:8443/nifi-api/access/token' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username=myuserad&password=mypasswordad' --compressed --cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pemcurl -H 'Authorization: Bearer token generated 'Content-Type: application/json' -XPUT -d '{"id":"****","state":"RUNNING"}' https://nifi-node/nifi-api/flow/process-groups/****--cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pemTo avoid entering the password in clear text as in curl N.1 I am testing the token generation via Kerberos: curl -X POST --negotiate -u : https://nifi-node:8443/nifi-api/access/kerberos --cacert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts .pem Using t

Hello Community,I have an old CDH6 and realized the following suspicious traces in cloudera.flood.log in /var/log/cloudera-scm-serverdht_pkt_alert ==> [82.43.248.101:6881]dht_pkt_alert <== [82.43.248.101:6881]and the same for another almost 100 different IPs outside the network by using the p2p libraries (and not sure that they are Cloudera Repository Ps)Did you see anything similar? this activity is quite suspicious.Thanks for your help     

Hi Team, One of our client is using Grafana as a visualization tool in HDP and we need to migrate to CDP.  Just want to understand, Does Grafana will be supported as a visualization tool in CDP?  Thanks and RegardsSunil Kumar KS

I'm trying to run  miniFi  in ubuntu. When I try to start miniFi, I got following error in bootstrap and app log. App.log2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected.2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow...2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down.java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key

Hi fellas, I have a folder structure where the parent folder is the date which contains multiple locations folder which contains the actual txt files.So the structure is such Logfolder/20240515/Location/logs.txtMy nifi setup is to retrieve the date folders inside the parent directory of Logfolder and pass it to executestreamcommand processor which contains my python script which requires the date folders to be sent in as argument along with all the contents inside the datefolder with original directory structure.But currently the flowfile is sending logs.txt to my executestreamcommand instead of the 20240515 as the flowfile. How do I structure such that the date folders as a whole is passed to the python script and not file by file?I have tried chatgpt but it isn't quite helpful. I appreciate any help on this as it is quite urgent.

Hello, We have a strange issue here. We are on CDP7.1.9 and have ranger in place for authorizations. A user is trying to execute queries from Hue on Impala Editor and fails with Authorization errors. However, same query works fine via Hive and also via Impala shell. It fails only from Hue editor. We have:1. Refreshed metadata2. Invalidated metadata3. Verified permissions are in place in Ranger4. There are no logs generated (I checked in Hue, Sentry Server and Catalog server). Please suggest if any other place should also be checked. Example query:select * from DB.table limit 5; Kindly help to diagnose and resolve this issue. 

I am getting the below error while connecting to hive using jdbc urlCaused by: com.cloudera.hiveserver2.support.exceptions.GeneralException: [Cloudera][HiveJDBCDriver](500171) Error occurred Connecting to ZooKeeper: At: <my_host>:2181/hiveserver2With Error Message: KeeperErrorCode = AuthFailed for /hiveserver2.JDBC URL:jdbc:hive2://zk=<my_host>:2181/hiveserver2,<my_host>:2181/hiveserver2;SSL=1;SSLTrustStore=/trustore_path/cm-auto-global_truststore.jks;SSLTrustStorePwd=My_pwd;AuthMech=1;KrbRealm=MYREAL.ORG.COM;KrbHostFQDN=_HOST;KrbServiceName=hive;LogPath=/tmp/log

I am trying to access the hadoop filesystem but I am facing below error.How to provide the access right to the user.even getfacl command also giving same error.I have tried the kinit command user has valid TGT.My main objective is to use the HadoopDelegation token.but code is failing even when tried to just list the hdfs files.FileSystem fs = FileSystem.get(conf);Path root = new Path("/");System.out.println((root));fs.listStatus(root); ----- code failing on this lineERROR message :java -jar HadoopDelegation-1.0-SNAPSHOT.jarlog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.java.io.IOException: DestHost:destPort master.localdomain.com:8020 , LocalHost:localPort master.localdomain.com/10.49.0.149:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Cl

HDFS cluster in HA enabled with one active namenode and one standby namenode. During checkpointing in how editlogs and fsimage get merged ?I tired with the source code check but got confused by the flow, Can someone help on this?, how editlogs and fsimage file get merged ?

Hi All,   We are trying to install CDP 7.10.1 in RHEL 8.9 . As per the Cloudera Support Matrix i dont see it is listing . During the installation we are encountering with Mutiple issues .Kindly let me know if any one have tried with the similar installation combination . As we are mainly using the RHEL 8.9 as standard image we would like to keep the same standard for Cloudera also . Any suggestion is welcome. RegardsJiji

I have a cloudera cluster up and running. I have Knox which forwards requests to webhbase and hbase uses ranger for authorization. Ranger is connected to FreeIPA LDAP and we use kerberos internally for authentication.In Ranger, I have a policy that gives access to read a table. In that same ranger policy a group in my freeIPA instance has the ability to access that table. I am having an issue. When I remove a member from that group in freeIPA and rerun ranger user sync, I try a curl call to get data from the table it still works. However when I run it again it gives the expected result as a denial. This has been a consistent pattern for all changes to user membership in ldap groups. The Ranger HBase Policy Sync is working as expected and the Usersync is working as expected as well. After I run Usersync, I confirm that the user has been removed from that group in Ranger but it's still allowing me access.  Does anyone know why this is? There are similar Kafka and HDFS poli

We save some flows into a registry with postgresql and now we want to transfer to a fresh nifi instance but we are getting this error:The Flow Registry with ID ? reports that no Flow exists with Bucket ?, FlowAny suggestion what is it happening?

I am trying to create grafana dashboard for the CM charts . So is it possible to connect grafana directly or if we need to store the data in database . Any samples.

Hello,Is there a way to list all HDFS directories a (human, not Impala, ...) user has access to, using the Ranger API?Many thanks.