Solved

I'm trying to change an existing variable. If the variable was defined in a process group and this process group is then copied, I get the following message when I try to change the variable.   "Unable to complete variable update request: Failed to update Variable Registry because failed while performing step: Applying updates to Variable Registry"   NiFi 1.10

Me and My colleagues are having a discussion regarding the pros and cons on running a Cloudera cluster on Physical Servers versus running the cluster on several Virtual Machines on a Hyper Converge servers.

I am using CDH 6.1.1 Cluster.Cluster is configured to use Spark as the execution engine for Hive.  Is there anything wrong with using SparkSQL on this Cluster?Is it ok to create Hive Tables and change data using SparkSQL? Since SparkSQL uses the Hive Metastore, I suspect that there may be a conflict between SparkSQL and Hive on Spark. In addition, please refer to documentation on how to intergrate Cloudera CDH Hive with Apache Zeppelin's Spark interpreter.  Thank you. 

I have a kudu table with schema:  create table test_table( `time` timestamp not null, -- `id` string not null, -- ..... primary key(`time`,`id`))partition by hash(id) partitions 6stored as kudu; and I try to use spark to copy the data to a parquet table in hdfs:  val df = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table" -> KuduTable)).format("kudu").load .where("time> '2019-10-29 08:05:10' AND time < '2019-10-29 08:05:30'") df.write .mode("append") .parquet("hdfs://parquet") But the performance is low and the job seems to be doing a full table scan against the kudu table (from spark UI, I can see the "Scan Kudu impala::table" is the number of entire table).For comparison I did a copy using impala's "insert into from" which is much faster and the "where" predicate seems to be working. Is this full table scan behavior expected or am I missing something here? The kudu version is 1.10.0

Dear Community,   I am trying to install an ambari cluster using Use Public Repository. I have already configured the Password Less on all the servers and I could connect from ambari node to the others using ssh root@node1,2... I copied the right ssh private key however the confirmation of hosts failed with:   Error while bootstrapping:/usr/bin/env: ambari-python-wrap: No such file or directory  I could not find any ambari-agent folder , should I have? I did not install agent manually ..   Please help Thanks and Regards

I import data from Postgresql to hdfs and hdfs to S3 in my daily operation. (sqoop import [postgres to hdfs] & distcp [from hdfs to s3])I wanted to remove intermediate step (hdfs) and directly import data to S3 bucket by using sqoop.However, same sqoop string fails in the end of the import operation.  sqoop import -Dmapreduce.map.memory.mb="8192" -Dmapreduce.map.java.opts="-Xmx7200m" -Dmapreduce.task.timeout=0 -Dmapreduce.task.io.sort.mb="2400" --connect $conn_string$ --fetch-size=20000 --username $user_name$ --p $password$ --num-mappers 20 --query "SELECT * FROM table1 WHERE table1.id > 10000000 and table1.id < 20000000 and \$CONDITIONS" --hcatalog-database $schema_name$ --hcatalog-table $table_name$ --hcatalog-storage-stanza "STORED AS PARQUET LOCATION s3a://path/to/destination"--split-by table1.id    I also tried   --target-dir s3a://path/to/destination   instead of   ....... LOCATION s3a://path/to/desti

All, I have a HDFS/Spark/YARN gateway node that has been assigned the roles via cloudera manager. However, Kerberos is preventing me from accessing HDFS from the gateway node. I could generate TGT via kinit command. I could access HDFS from other nodes in the hadoop cluster using the same user id and same steps. All kerberos configuration files in the gateway node seems to be in place. Any suggestions what i could have missed?    

Hi,   I have a requirement where-in I have to take a count(*) using selecthiveQL and if count is 0 then I have to insert a row.  My flow is selectHive QL → RouteOnAttribute where in I am trying to capture the 0  Json is as below: [ {"_c0" : 0} ] can some one please tell me how to capture this zero, I tried multiple ways but not able to.  Also if anyone has better solution pls let me know. This is quite urgent. 

I have Nifi 1.9.2 and I'm trying to run it into 3 different ports on the same machine (Localhost). I can't seem to find Anything on the internet that can help me with that.Can we do this?I need your help guys, Thanks in advance

The version of CDH5.16.1 I used recently has the following error, which makes it impossible to use the cm management platform to execute some commands, such as distributing the Gateway configuration. I hope to provide some solutions. 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: (1 skipped) Unable to generate configuration for GATEWAY ba se group 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: Unable to generate configuration for HIVESERVER2: hive-HIVE SERVER2-0f0c09243092e98c6d762d9b05f72d25 2019-11-07 14:00:02,163 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (GlobalPoolsRefresh) for schedu le (1) java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:968) at

Hi, everyone, my CDH cluster can't execute commands on the cm management platform, such as pushing the client configuration, killing the jobs on the yarn, etc. The error log of the server is as follows. Thank you for your suggestions. Thank you. com.cloudera.cmf.command.CmdExecException: com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException A server error has occurred. The full stack trace is not shown here due to security reasons. See Cloudera Manager Server log for details. WARN 986168735@scm-web-253:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:119) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885) at com.cloude

Hi, We are planing for CDP cluster deployment for Ms Azure cloud , Any link to get the repo file 

This problem has been bothering me for a long time The oozie role on Cloudera Manager starts to report an error. The information is as follows. How do I troubleshoot it, and where is oozie.keytab? , which logs can be queried to help solve? SERVER[xxxx.com] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], failure to login: for principal: oozie/xxxx.com@XXXX.COM from keytab oozie.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) - PREAUTH_FAILED

I run a PC with Ubuntu 16.04 LTS. I have installed CDH 5.16.1 (installation by parcels) and had accidentally allowed it to install Oracle Java 7 into my default Java directory (/usr/lib/jvm). Prior to my CDH installation I had installed OpenJDK 8 on my PC in the same location.    I then downloaded the JDBC for Postgresql for Java 8 and higher postgresql-42.2.5.jar and placed it in the recommended directory (/var/lib/sqoop).   In attempting to connect to my database and list the tables present, I got the following error:   Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/Driver : Unsupported major.minor version 52.0at java.lang.ClassLoader.defineClass1(Native Method)at java.lang.ClassLoader.defineClass(ClassLoader.java:800)at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)at java.net.URLClassLoader.access$100(URLClassLoader.java:71)at java.net.

I install Kafka in Ambari and configure two listeners for every broker, one for inner traffic and one for outer traffic. E.g., broker A has two ip addresses: 192.168.1.1 and 88.88.88.88, and its configuration is like follows:   SASL_PLAINTEXT://192.168.1.1:9092,EXTERNAL://88.88.88.88:19092listener.security.protocol.map=SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXTsecurity.inter.broker.protocol=SASL_PLAINTEXT   With this configurations, Kafka works fine but keeps alerting, the message is:   Connection failed: [Errno 111] Connection refused to 192.168.1.1:19092    This is not valid because ip and port do not match. And I suspect that it could just split by colon and use the last element. So I exchange the order of the two listeners:   EXTERNAL://88.88.88.88:19092,SASL_PLAINTEXT://192.168.1.1:9092   To my surprise, it still alerts and the message changes to    Connection failed: [Errno 111] Connection refused to 88.88.88.88:9092  

Hi, We are getting the error while executing Hive query on spark as execution engine. Hive version: 1.2.1, Spark version : 1.6   set hive.execution.engine=spark;set spark.home=/usr/hdp/current/spark-client;set hive.execution.engine=spark;set spark.master=yarn-client;set spark.eventLog.enabled=true;set spark.executor.memory=512m;set spark.executor.cores=2;set spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-exec.jar;   Query ID = svchdpir2d_20191106105445_a9ebc8a2-9c28-4a3d-ac5e-0a8609e56fd5Total jobs = 1Launching Job 1 out of 1In order to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>In order to limit the maximum number of reducers:set hive.exec.reducers.max=<number>In order to set a constant number of reducers:set mapreduce.job.reduces=<number>Starting Spark Job = c6cc1641-20ad-4073-ab62-4f621ae595c8Status: SENTFailed to execute spark task, with exception 'java.lang.IllegalStateExcept

hive metastore is consuming complete heap(tried max heap : 24G) after sometime  the metastore is not responding , the metastore is ok if we disable ranger hive authentication  this issue is reported in (https://jira.apache.org/jira/browse/HIVE-20568) ,It seems ranger plugin is causing memory leaks during audit when it encounters hive db names containing '_' just wondering to know if there is a work around for it in HDP-3.1.0 or the upgrade from HDP-3.1 to HDP-3.1.4 will fix it,  Thanks in advance. Current Version  and config Information :========================HDP-3.1hive-3.1Hive Authentication - RangerJDK-1.8mysql -7.5mysql-connector-8.0.13.jar    

I'm running a from-scratch cluster on AWS EC2. I have an external table (partitioned) defined with data on S3. I'm able to query this table and receive results to the console with a simple select * statement:hive> set hive.execution.engine=tez;hive> select * from external_table where partition_1='1' and partition_2='2';[correct results returned]Running a query that requires Tez doesn't return the results to the console:hive> set hive.execution.engine=tez;hive> select count(*) from external_table where partition_1='1' and partition_2='2';Status: Running (Executing on YARN cluster with App id application_1572972524483_0012)OK+------+| _c0 |+------++------+No rows selected (8.902 seconds)However, if I dig in the logs and on the filesystem, I can find the results from that query:(yarn.resourcemanager.log) org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1572972524483_0022

/usr/local/Hbase/bin# ./hbase shellJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.jruby.java.invokers.RubyToJavaInvoker (file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar) to method java.lang.Object.registerNatives()WARNING: Please consider reporting this to the maintainers of org.jruby.java.invokers.RubyToJavaInvokerWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future releaseArgumentError: wrong number of arguments (0 for 1)method_added at file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10method_added at file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129Pattern at file:/usr/local/Hbase/lib

After upgrading the Cloudera Manager from 6.2.0 to 6.3.1, the CDSW won't come online anymore. In the health logs, this line stands out:   MaxRetryError: HTTPSConnectionPool(host='xxx.xxx.xxx.xxx', port=6443): Max retries exceeded with url: /api/v1/secrets (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f364478d210>: Failed to establish a new connection: [Errno 111] Connection refused',))  The port is closed on the CDSW master node, and I couldn't find any service in the docs that depends on it. Is it Kubernetes? Further, from cdsw validate it seems that some chains are missing from iptables: The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM]  However, I cannot remember any step in the installation process that required to set such rules, so I a

I'd like to use API to set CM monitoring thresholds so I can deploy the modified ones on other CMs.  For example, to set 'Cloudera Manager Agent Log Directory Free Space Monitoring Absolute Thresholds' Warning at 5GB and Critical to 2GB instead of using the default values.   I am not sure with  API to use by looking at  this webpage: https://archive.cloudera.com/cm6/6.0.0/generic/jar/cm_api/apidocs/index.html   Would anyone provide some help?   Thanks, Kevin

How do I add the queue name to the Apache Zeppelin JDBC interpreter settings?

Hi,   We are currently using HDP 2.6.4 image for our development environment and plan to use the same for client deliveries in production environment. This HDP image supports Java8 whereas the company standard says we need support for Java11 for any client deliveries. How can we tackle this problem if we plan to use HDP in our production environment?

Hello ,I installed Ambari server and Ambari agent but I have  ten day that I try to connect ,I receive this error in ambari-agent.log:  INFO 2019-11-06 09:46:39,682 NetUtil.py:62 - Connecting to https://localhost:8440/caWARNING 2019-11-06 09:46:39,683 NetUtil.py:93 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refusedWARNING 2019-11-06 09:46:39,683 NetUtil.py:116 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds...INFO 2019-11-06 09:46:49,683 NetUtil.py:62 - Connecting to https://localhost:8440/caWARNING 2019-11-06 09:46:49,684 NetUtil.py:93 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refusedWARNING 2019-11-06 09:46:49,685 NetUtil.py:116 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds...  And in Ambari-server I have:  05 Nov 2019 11:28:49,329 INFO [ambari-hearbeat-monitor] TopologyManager:487 - Hearbeat for host localhost.localdomain l

Hello! This is a very rudimentary request perhaps, so fair warning! However, recently I installed CFM and had NiFi working fine without NiFi Toolkit CA and NiFi Registry services.I wanted to add LDAP authentication, so I removed the NiFi service, and then added the 3 services in the order that is recommended by the CFM 1.0.1 Documentation, and proceeded to configure the services for LDAP. Now, the NiFi Node fails to stay online. Everything else seems to be in order, but NiFi Node shuts off very quickly after it reports good health in SCM.My first guess is that it could be an issue with Registry, because I am using only 1 CFM host, yet I have all 3 services running on the same, and additionally I am using the same host as the NiFi Gateway. I did not read anything about this in the documentation, but perhaps I've overlooked something simple due to my lack of experience. Did I mess something up by setting the Gateway as the same host as the NiFi node?Any info you can offer to help me trou