Solved

I have created a data pipeline with Apache Ni-Fi (version 1.15.2) to transfer data from a MySQL database in a AWS RDS instance into Timescale (postgreSQL). Ni-Fi runs in Linux Ubuntu 18.04 in Virtual Box on top of Windows 10. The pipeline has been tested successfully by using as a target the Timescale database located in my local Linux instance, without using SSL. Then I have modified the pipeline to use Timescale Cloud as a target (https://portal.timescale.cloud/login). The database name is: periodic-measurements In Timescale Cloud service page I can see these credentials: Host: <host-string>Port: 10250User: tsdbadminPassword: <timescale-cloud-service-password>Service URI: postgres://tsdbadmin:<timescale-cloud-service-password>@<host-string>:10250/defaultdb?sslmode=requireSSL mode: requireAllowed IP addresses: 0.0.0.0/0I use PutDatabaseRecord processor to write data in the postgreSQL database which uses the DBCPConnectionPool controller service.&

We're trying for HDP upgrdation from 3.1.5.0-152 to 3.1.5.6189-1.As we're having four nodes cluster, in two nodes the upgradation part completed, but in other it's failing. Please provide some solutions. Background operations log:Error getting repository data for HDP-3.1-GPL-repo-51, repository not found2022-03-09 11:09:07,436 - Installing package sqoop_3_1_5_6189_1 ('/usr/bin/yum -y install sqoop_3_1_5_6189_1')2022-03-09 11:09:11,891 - Installing package hadoop_3_1_5_6189_1 ('/usr/bin/yum -y install hadoop_3_1_5_6189_1')2022-03-09 11:09:16,296 - Installing package hadoop_3_1_5_6189_1-client ('/usr/bin/yum -y install hadoop_3_1_5_6189_1-client')2022-03-09 11:09:21,000 - Installing package snappy ('/usr/bin/yum -y install snappy')2022-03-09 11:09:25,451 - Installing package snappy-devel ('/usr/bin/yum -y install snappy-devel')2022-03-09 11:09:30,161 - Installing package hadoop_3_1_5_6189_1-libhdfs ('/usr/bin/yum -y install hadoop_3_1_5_6189_1-libhdfs')2022-03-09 11:09:34,700 - Inst

Hi All, Refer to CDSW install doc, create subdomain is part of setup pre-requisite and the subdomain has to such as  "cdsw.<your_domain>.com" and "*.cdsw.<your_domain>.com". My question is, does the CDSW setup must use "cdsw" as a subdomain name? does it support other names?    Thanks in advance RegardsMatt

Hello Team,We are facing some issue related to Hue KT renewal server not starting up.the KT renewal log saying k.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/notebook/src/notebook/urls.py'> (notebook:notebook) ^notebook/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Dynamic pattern: <RegexURLResolver <module 'dashboard.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/dashboard/src/dashboard/urls.py'> (dashboard:dashboard) ^dashboard/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Dynamic pattern: <RegexURLResolver <module 'kafka.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/kafka/src/kafka/urls.py'> (kafka:kafka) ^kafka/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Static pattern: <RegexURLPattern None ^static\/(?P<path>.*)$>[16/Mar/2022 01:30:32 +0100] kt_renewer INFO Keytab renewer not starting, no keytab configured  I have do

Hello,I have a problem with converting json to csv. I use ConvertRecord processor with attached JsonTreeReader and CSVRecordSetWriter.The schema for both might look like this. Attributes in the json are optional.     {"type": "record","name": "some_event","fields": [{ "name": "date_time", "type": ["string","null"] },{ "name": "id", "type": ["string","null"] },{ "name": "is_success", "type": ["string","null"] },{ "name": "username", "type": ["string","null"] },{ "name": "username_type", "type": ["string","null"] }]}   for the input json like   {"date_time": "2017-01-29T13:42:03.965Z","id": "20049015990001584400001""is_success": "TRUE",}   where username and username_type is not set, the output does not correspond to my expectation I get2022-01-29T13:42:03.965Z|20049015990001584400001|trueinstead of2022-01-29T13:42:03.965Z|20049015990001584400001|true||which should be the right format. Note the separator for null values at the end of the second record.   Do

Hi Experts, One of our oozie workflow got failed while sqoop import job. Please find below logs: org.apache.velocity.exception.VelocityException: Error initializing log: Failed to initialize an instance of org.apache.velocity.runtime.log.Log4JLogChute with the current runtime configuration. in other logs as : org.springframework.web.client.HttpServerErrorException: 500 null Please tell what could be the possible cause of this failure, we retried this job but getting the same error.

Hi Experts, One of the Impala daemon suddenly went down and we are unable to bring it up. Some queries were executed on the impala daemon. I have captured the logs when it got shutdown. Could not open log file: /var/log/impalad/lineage/impala_lineage_log_1.0-1647629222243@ 0x95b479 impala::Status::Status()@ 0xd00b95 impala::SimpleLogger::FlushInternal()@ 0xd01fcf impala::SimpleLogger::Init()@ 0xbb9c2a impala::ImpalaServer::InitLineageLogging()@ 0xbcb25e impala::ImpalaServer::ImpalaServer()@ 0xbb676d ImpaladMain()@ 0x8e2593 main@ 0x7f5cf7658495 __libc_start_main@ 0x9299e1 (unknown) ERROR cc:312 Aborting Impala Server startup due to failure initializing lineage logging. Impalad exiting. Can some one please tell what steps should be taken, Just to note Enable Audit Collection and Enable Impala Audit Event Generation is enabled/checked. 

Version 1.15.3Windows Server 2019 Standard After GUI login receive the following error: Invalid StateThe Flow Controller is initializing the Data Flow.

Version 1.15.3 Windows Server 2019 Standard   # zookeeper properties, used for cluster management #nifi.zookeeper.connect.string=nifi-inf-02:2181,nifi-inf-03:2181nifi.zookeeper.connect.timeout=10 secsnifi.zookeeper.session.timeout=10 secsnifi.zookeeper.root.node=/nifinifi.zookeeper.client.secure=falsenifi.zookeeper.security.keystore=nifi.zookeeper.security.keystoreType=nifi.zookeeper.security.keystorePasswd=nifi.zookeeper.security.truststore=nifi.zookeeper.security.truststoreType=nifi.zookeeper.security.truststorePasswd=nifi.zookeeper.jute.maxbuffer=   After configuration of cluster the service spits the following error on startup.   Please check the configuration of the Cluster State Provider with ID [zk-provider] in the file D:\nifi-1.15.3\.\conf\state-management.xmlat org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)at org.springframework.beans.factory.support.FactoryBeanRegistrySupport

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode? Thank you

Hello, I have a question regarding hive bucketed tables (bucketed only no partitions) optimization. Now, I know that if we have table A and table B and we want to join them using A.COL1 and B.COL1 ( A.COL1 = B.COL1), we should bucket both table A and B on col1 into same no of buckets or its multiple. But what if we have more than 2 tables ? for example, I have table A and I want to join it on table B and table C. table A is joined with table B using COL1 (A.COL1 = B.COL1) and table A is joined with table C using COL2 (A.COL2 = C.COL2 ) what columns should I cluster by for table A ??? is it bucket by COL1 and COL2 (clustered by col1,col2)?  in summary, how can I optimize one table if it is joined with more than one table using buckets only. Thanks in advance.   

Is it possible to connect to on-prem kafka cluster using Kerberos authentication from cloud deployed service?When we are trying to connect we are getting below error:Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm "ABC.COM"This is my jaas config:com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/pathtokeytab" principal="principal_name@ABC.COM" ; Please help me if anyone has faced such error.  Full stack trace: Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:162)        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) 

I have the following json file :“a”:{    “after”:{        “a123”:{                 “ip”:”1.0.0.0”,                 “p_id”:”4500”                   }           },“before”:{        “a123”:{            “ip”:”1.0.0.0”,            “p_id”:”4500                  }            }} I'm trying to create a lookuprecord Processor in NiFi to match with the p_id. If matched then route to something and if unmatched then route to something else. The lookuprecord processor I have is a MongoDBLookUpService.The user expression I've written for the lookuprecord processor is :key : /a/after/*['p_id']Note: [*] because that bit in json keeps changing with e

If I have 3 environments eg :SIT,UAT,PROD each is having 5 nodes(total 15 nodes). I need 3 independent  CDP clusters  which will be having 5 nodes each. If I take a CDP license of 15 nodes can I create  3 independent clusters  using the same license  or I  need to take separate license for each. Please assist on thisThanks & RegardsRahul P

I am trying to setup "Deploying Hortonworks Sandbox on Docker" on MAC M1 for practice , unfortunately i am not able to find a solution in spite of google search /stock overflow Official Cloudera Installation directory https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html Reference installation https://javamana.com/2021/02/20210206191356735v.html   below is my configuration Ram - 16gb - i have allocated 12 gb Rosetta2 Installed Docker installed Mac M1   Docker hub image link https://hub.docker.com/r/hortonworks/sandbox-hdp/tags docker pull hortonworks/sandbox-hdp:3.0.1 --platform linux/amd64   Script       #!/usr/bin/env sh#This script downloads HDP sandbox along with their proxy docker containerset -x# CAN EDIT THESE VALUESregistry="hortonworks"name="sandbox-hdp"version="3.0.1"proxyName="sandbox-proxy"proxyVersion="1.0"flavor="hdp"# NO EDITS BEYOND THIS LINE# housekeepingecho $flavor > sandbox-flavor# create necessary folders fo

Hello Team,Our impala services are not starting : its failing with below log file:Although we have put all configuration file in Impala configuration.4:18:23.781 PM INFO cc:370Webserver: set_ssl_option: cannot open /opt/cloudera/security/x509/key/host.pem: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt4:18:23.782 PM INFO cc:124Webserver: Could not start on address 0.0.0.0:8007@ 0xb5e579@ 0x11e5fec@ 0x110e053@ 0xac6b1c@ 0x7ffa35134554@ 0xb237e84:18:23.782 PM FATAL cc:59Webserver: Could not start on address 0.0.0.0:8007  

I was looking for jar files like below but not sure which drivers I need and where to obtain them. hive-exec.jarhive-jdbc.jarhive-metastore.jarhive-service.jar Any support is greatly appreciated! Thank YOU!

Hello I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output: I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong?FetchS3Object Config: Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error. CheersAndy

Hi Team , I trying to get information using curl command and seems CM Manager sending me login page. I  already using user name and password . curl -X GET --user 'xxxx':'xxxx' https://CM URL:7183/cmf/serviceRedirect/schemaregistry/instances<!-- __CLOUDERA__PRE__LOGIN__FORM__ --><!DOCTYPE html><html><head><meta charset="utf-8"></head><body></body></html><script type="text/javascript">var returnUrl = window.location.pathname + window.location.search + window.location.hash;var loginPageUrl = "/cmf/login";if (returnUrl !== "/cmf" && returnUrl !== "/cmf/" && returnUrl !== "/cmf/home" && returnUrl !== "/cmf/home/") {loginPageUrl = loginPageUrl + "?returnUrl=" + encodeURIComponent(returnUrl);}window.location = loginPageUrl;</script>  pls some one help. Thanks

Hi, We are trying to fetch the data from one of our kafka cluster (200000 messages per seconds) where we have 19 partition on the topic and using nifi (cluster of 4 hosts) inbuilt processor (ConsumeKafka_2_6) however we are observing delay in processing messages in the processor (linage time of 7 sec per message). We tried using increase concurrent task to match up with the partition as well but it did not work. however if we use demarcation strategy (to club multiple messages into single flowfile) it is working fine but then we need individual messages for data transformation.   The other processor which does the JSON evaluation also have similar problem as it can not handle large incoming data from the cosumekafka processor.   Notes : Backpressure threshold is 100000 file and size threshold is 1 GB PFB reference SS of the processor configured.   Evaluate Json path config:  (Run duration 0.05 ms,  concurrent task-4 per node )      

Hi,I am facing issue accessing one of the child json attribute while forming SQL in QureyRecord processor. PFB details here is my Avro schema added for JSON reader and writter.{"name": "MyClass","type": "record","namespace": "com.acme.avro","fields": [{"name": "labels","type": {"name": "labels","type": "record","fields": [{"name": "__name__","type": "string"},{"name": "cucsMemoryUnitInstanceId","type": "string"},{"name": "instance","type": "string"},{"name": "job","type": "string"},{"name": "monitor","type": "string"},{"name": "site_identifier","type": "string"}]}},{"name": "name","type": "string"},{"name": "timestamp","type": "string"},{"name": "value","type": "string"},{"name": "producedAt","type": "string"}]} now the SQL quary for first level attribute works fine.  like SELECT * from FLOWFILE where name = 'xyz'   or SELECT * from FLOWFILE where producedAt = 'dd-mm-yyyy'however If I have to access attribute of nested JSON it wont work &nbs

Hi, I want to grant a set of users access to add tag based policies in ranger. I have adde the users to data steward role which grants ranger/atlas admin access but still they get a access denied when create a tag based access policy. Creating resource based policies is working for them. Ideally we want to grant them permissions to add tag based policies. We use CDP public cloud 7.2.12 in AWS. Any pointers are welcome. Thanks.

Planning to migrate to Hive3 on CDH and CDP private base. What are the feature wise changes happened from hive2 to hive3 and also point out the risk areas for hive3

I am using HDP hive-3.1.My configuration as follows:hive.support.concurrency=true;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; hive.exec.dynamic.partition.mode=nonstrict;hive.compactor.initiator.on=true;hive.compactor.worker.threads=2; By default, transactional table is created in hive. however, i want to create non-transactional table , by default. I tried to change hive.txn.manager to Dummymanager. Though after changing the configuration, hive table is created as transactional. any suggestion to create non-transactional table, be default? 

Hi All, When I list yum packages on the Redhat/Centos server which is CDH runs, I see some samba packages.Are they necessary or related to some Cloudera services? I have also checked whether there are dependencies on samba but there are no on Linux level. We need to remove or update this package due to CVE-2021-44142.  Note: First, We wanted to open a ticket through Cloudera support but it is not possible to open a ticket other than any CDH/CDP component. How can we know more detail about that? Thanks.