Q&A

If I have 3 environments eg :SIT,UAT,PROD each is having 5 nodes(total 15 nodes). I need 3 independent  CDP clusters  which will be having 5 nodes each. If I take a CDP license of 15 nodes can I create  3 independent clusters  using the same license  or I  need to take separate license for each. Please assist on thisThanks & RegardsRahul P

I am trying to setup "Deploying Hortonworks Sandbox on Docker" on MAC M1 for practice , unfortunately i am not able to find a solution in spite of google search /stock overflow Official Cloudera Installation directory https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html Reference installation https://javamana.com/2021/02/20210206191356735v.html   below is my configuration Ram - 16gb - i have allocated 12 gb Rosetta2 Installed Docker installed Mac M1   Docker hub image link https://hub.docker.com/r/hortonworks/sandbox-hdp/tags docker pull hortonworks/sandbox-hdp:3.0.1 --platform linux/amd64   Script       #!/usr/bin/env sh#This script downloads HDP sandbox along with their proxy docker containerset -x# CAN EDIT THESE VALUESregistry="hortonworks"name="sandbox-hdp"version="3.0.1"proxyName="sandbox-proxy"proxyVersion="1.0"flavor="hdp"# NO EDITS BEYOND THIS LINE# housekeepingecho $flavor > sandbox-flavor# create necessary folders fo

I'm having issues with ExecuteStreamCommand, I have set up the Argument Delimiter as ~ like screenshot below:  However when ERROR_DETAIL attribute contains newline characters (\n or %0A) apparently Nifi treats the newline as a delimiter and hence the shell script receives more than 1 arguments in the place of ERROR_DETAIL. I was expecting Nifi to use strictly just the ~ character as the argument delimiter. Am I missing a different config in Nifi? Is there a way to make Nifi ignores the newline and only use the ~ as the delimiter?

Dear: Recently i met a KUDU issue which was randomly occurred. Below is the details.CDH Version: 5.14.2KUDU Version: 1.6.0-cdh5.14.2ERROR Information:21/11/28 06:56:03 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.kudu.client.NoLeaderFoundException: Master config (10.186.93.6:7051,10.186.93.24:7051,10.186.93.8:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection disconnected,org.apache.kudu.client.RecoverableException: connection disconnectedorg.apache.kudu.client.NoLeaderFoundException: Master config (10.186.93.6:7051,10.186.93.24:7051,10.186.93.8:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection disconnected,org.apache.kudu.client.RecoverableException: connection disconnectedat org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:273)at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)at org.a

Hello Team,Our impala services are not starting : its failing with below log file:Although we have put all configuration file in Impala configuration.4:18:23.781 PM INFO cc:370Webserver: set_ssl_option: cannot open /opt/cloudera/security/x509/key/host.pem: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt4:18:23.782 PM INFO cc:124Webserver: Could not start on address 0.0.0.0:8007@ 0xb5e579@ 0x11e5fec@ 0x110e053@ 0xac6b1c@ 0x7ffa35134554@ 0xb237e84:18:23.782 PM FATAL cc:59Webserver: Could not start on address 0.0.0.0:8007  

I was looking for jar files like below but not sure which drivers I need and where to obtain them. hive-exec.jarhive-jdbc.jarhive-metastore.jarhive-service.jar Any support is greatly appreciated! Thank YOU!

Hello I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output: I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong?FetchS3Object Config: Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error. CheersAndy

Hi Team , I trying to get information using curl command and seems CM Manager sending me login page. I  already using user name and password . curl -X GET --user 'xxxx':'xxxx' https://CM URL:7183/cmf/serviceRedirect/schemaregistry/instances<!-- __CLOUDERA__PRE__LOGIN__FORM__ --><!DOCTYPE html><html><head><meta charset="utf-8"></head><body></body></html><script type="text/javascript">var returnUrl = window.location.pathname + window.location.search + window.location.hash;var loginPageUrl = "/cmf/login";if (returnUrl !== "/cmf" && returnUrl !== "/cmf/" && returnUrl !== "/cmf/home" && returnUrl !== "/cmf/home/") {loginPageUrl = loginPageUrl + "?returnUrl=" + encodeURIComponent(returnUrl);}window.location = loginPageUrl;</script>  pls some one help. Thanks

Hi, We are trying to fetch the data from one of our kafka cluster (200000 messages per seconds) where we have 19 partition on the topic and using nifi (cluster of 4 hosts) inbuilt processor (ConsumeKafka_2_6) however we are observing delay in processing messages in the processor (linage time of 7 sec per message). We tried using increase concurrent task to match up with the partition as well but it did not work. however if we use demarcation strategy (to club multiple messages into single flowfile) it is working fine but then we need individual messages for data transformation.   The other processor which does the JSON evaluation also have similar problem as it can not handle large incoming data from the cosumekafka processor.   Notes : Backpressure threshold is 100000 file and size threshold is 1 GB PFB reference SS of the processor configured.   Evaluate Json path config:  (Run duration 0.05 ms,  concurrent task-4 per node )      

Hi,I am facing issue accessing one of the child json attribute while forming SQL in QureyRecord processor. PFB details here is my Avro schema added for JSON reader and writter.{"name": "MyClass","type": "record","namespace": "com.acme.avro","fields": [{"name": "labels","type": {"name": "labels","type": "record","fields": [{"name": "__name__","type": "string"},{"name": "cucsMemoryUnitInstanceId","type": "string"},{"name": "instance","type": "string"},{"name": "job","type": "string"},{"name": "monitor","type": "string"},{"name": "site_identifier","type": "string"}]}},{"name": "name","type": "string"},{"name": "timestamp","type": "string"},{"name": "value","type": "string"},{"name": "producedAt","type": "string"}]} now the SQL quary for first level attribute works fine.  like SELECT * from FLOWFILE where name = 'xyz'   or SELECT * from FLOWFILE where producedAt = 'dd-mm-yyyy'however If I have to access attribute of nested JSON it wont work &nbs

2022-03-14 04:03:03,899 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []2022-03-14 04:03:03,902 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED2022-03-14 04:03:03,902 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED2022-03-14 04:03:03,903 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED2022-03-14 04:03:03,931 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_3122022-03-14 04:03:03,941 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...2022-03-14 04:03:03,942 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /root/nifi-1.15.22022-03-14 04:03:03,942 INFO [main] org.apache.nifi.bootstrap.Command Command: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -classpath /root/nifi-1.15.2/./conf:/root/

Hi, I want to grant a set of users access to add tag based policies in ranger. I have adde the users to data steward role which grants ranger/atlas admin access but still they get a access denied when create a tag based access policy. Creating resource based policies is working for them. Ideally we want to grant them permissions to add tag based policies. We use CDP public cloud 7.2.12 in AWS. Any pointers are welcome. Thanks.

Planning to migrate to Hive3 on CDH and CDP private base. What are the feature wise changes happened from hive2 to hive3 and also point out the risk areas for hive3

I am using HDP hive-3.1.My configuration as follows:hive.support.concurrency=true;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; hive.exec.dynamic.partition.mode=nonstrict;hive.compactor.initiator.on=true;hive.compactor.worker.threads=2; By default, transactional table is created in hive. however, i want to create non-transactional table , by default. I tried to change hive.txn.manager to Dummymanager. Though after changing the configuration, hive table is created as transactional. any suggestion to create non-transactional table, be default? 

I am using the below NIFI EXPRESSION in the update attribute This works fine, but after division getting scientific notation, say for an example if the GROSSDEPTH = 0.010000, then the expression output is coming as : GROSSDEPTH1.0E-4how to remove this scientific notation. Please help

Hi All, When I list yum packages on the Redhat/Centos server which is CDH runs, I see some samba packages.Are they necessary or related to some Cloudera services? I have also checked whether there are dependencies on samba but there are no on Linux level. We need to remove or update this package due to CVE-2021-44142.  Note: First, We wanted to open a ticket through Cloudera support but it is not possible to open a ticket other than any CDH/CDP component. How can we know more detail about that? Thanks.

Hi Team, I bought CDP Public platform. And I have user and my purpose is to create Data-Hub Cluster and with my custom definition and template I have created a data hub cluster. And I see list of URL for access like Cloudera Manager, Hue and some. If I click cloudera manager URL it directly login to Cloudera Console page without any authentication. But I want to create multiple user with Cloudera access, Hue UI access. I have created machine user and I have created workload password also but how to access the UI with this User as there is only one UI access which I created initially but for the rest I'm not able to create or access console page for any services.Please help me with this. Regards,V

Creating an external table in hive to query hbase table in CDP 7 CREATE EXTERNAL TABLE emp(id INT, empno INT,ename STRING,designation STRING, manager INT, salary INT,deptno INT) STORED BY "org.apache.hadoop.hive.hbase.HBaseStorageHandler" WITH SERDEPROPERTIES ( "hbase.columns.mapping" =':key,cf:empno,cf:enmae,cf:designation,cf:manager,cf:salary,cf:deptno') TBLPROPERTIES("hbase.table.name" = "emp_data");     ERROR :  FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. org/apache/hadoop/hbase/mapreduce/TableInputFormatBase22/03/09 12:16:40 [HiveServer2-Background-Pool: Thread-25]: ERROR ql.Driver: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. org/apache/hadoop/hbase/mapreduce/TableInputFormatBaseOK

未处理的表格：{    "日期":"2022-09-03 19:23:53",    “设备名称”：“032-013”，    "device_id":"1144",    “传感器”：[        {            "sensor_id":"5836",            “价值”：“1”，            “传感器名称”：“设备状态”        },        {            "sensor_id":"5842",            “价值”：””，            “传感器名称”：“12345”        }…         ],    “状态”：“1”}最后结果： {    “日期” ：“2022-09-03 19:23:53” ，    “设备名称” ：“032-0

Hi I am very new to nifi and try to install and configure the system. I started nifi.sh start and it shows  Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/HomeNiFi home: /Users/daniel/Downloads/nifi-1.15.3 Bootstrap Config File: /Users/daniel/Downloads/nifi-1.15.3/conf/bootstrap.conf And when i enter nifi.sh  status, terminal shows: 2022-02-26 09:31:33,238 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 52929, PID=74313 I changed nifi.web.http.port in nifi.properties to 8080. But still I do not see any on localhost:8080. In the log file there is no message that server has started on 8080 port.  I have no idea what problem caused the issue. Please advise me. Thanks

Hello Guys,How to Extract recent past Saturday based on today date and is there any expression language to get by update attribute 

I am a newbie to Nifi Jolt transform.  I've been looking through the document, but still cannot figure out how to do a simple Jolt Transform.Input:{"id": 123456,"manager_id": 98765}desired output:{"reportTo": [{"userId": 98765}]} I've got the part to change the manager_id to userId by the following Spec.  But I was stuck how to make it another layer.  Any insights/help to solve this? [{"operation": "shift","spec": {"manager_id": "userId"}}}]

Hello,I'm currently facing an issue concerning apache hive specifically in relation with surrogate key function.In ordre to recreate the probleme i tried to use the exemple on cloudera that uses surrogate key:<https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive_surrogate_keys.html>I have created the table students_v2 as shown on the website, and tried to insert the following line:`insert into students_v2(row_id, name, dorm) values (1, "ami", "ne")`However i receive the following error:```SQL Error [2] [08S01]: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1644246846031_0051_1_00, diagnostics=[Task failed, taskId=task_1644246846031_0051_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1644246846031_0051_1_00_000000_0:java.lang.RuntimeException: java.lang.RuntimeException:

Hi , We're having three node mangers , out of which two not working due to the below error:resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su yarn -l -s /bin/bash -c 'ulimit -c unlimited; /usr/hdp/3.1.5.0-152/hadoop-yarn/bin/yarn --config /usr/hdp/3.1.5.0-152/hadoop/conf --daemon start nodemanager'' returned 1. ERROR: Cannot set priority of nodemanager process 30232 Please suggest some solutions.

https://github.com/GitJason1027/cm_api_programs/blob/main/bdr_build_plan.pyHere is my program to create schedule in BDR function.（CDH professional version）But some options I cannot find in the cm_api interface,s such as "Permanently Delete", "Copy HDFS File " options. Is someone can tell me how to deal with it ? https://github.com/cloudera/cm_api/blob/master/python/src/cm_api/endpoints/types.pyline 686 class ApiHdfsReplicationArguments