Q&A

hi there, Below spark-sql comand failed to run on an edgenode with only spark-client installed spark-sql --executor-cores 2 --num-executors 2 -e 'select profitcenter, count(1) from default.tableX group by profitcenter' --master yarn --name spark_sql_test1 --verbose --queue defaultThe error log is:19/01/23 19:18:24 ERROR ApplicationMaster: Failed to connect to driver at emr-header-1.cluster-92503:35638, retrying ...19/01/23 19:18:24 ERROR ApplicationMaster: Uncaught exception:org.apache.spark.SparkException: Failed to connect to driver! at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:672) at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:532) at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$runImpl(ApplicationMaster.scala:347) at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$run$2.apply$mcV$sp(ApplicationMaster.sca

If I were planning on decommisioning a node, is it possible to drain the flume channels? If I disable the sink, Flume doesn't proceed to do anything and if I change the batch size on the Kafka source to 0, it still fetches data.

I just upgraded ambari to 2.7.3 on my cluster. The cluster had previously been configured with kerberos, and running correctly. Now whenever I try to modify components I get an "admin session expired" dialog. I enter my admin credentials (kerberos) and get an invalid KDC admin error returned. I verified that the principal I entered has admin rights by logging into kadmin on the command line. The principal I am entering is the one I used when I initially configured the cluster to use kerberos. Why would this not work anymore, and how can I fix it?

hello all,I need to pull files from linux server, so at first i used listFile and fetchFile processors.The files are being writing and when they are done, nifi need to pull them.Unfotunatly, listFile timestamp already list the file but fetchFile did not pull them because they are still being writing (i added last modified limitation). so the files left in the server. To solve this problem, i started using getSftp which is really good, but it does not include file size attribute.I can see the file size at DETAILS tab in the flowFile but cant access it.Do you have any suggestion?

hello all, Trying to update a  table where date_num is  a  date(timestamp) and "day" is the field i want to update but i am getting errorsAnyhting wrong with the query below? update anp_catdidai1_working.dim_bsp_dateset day=day(date_num)

Hi,   I noticed that the time it takes for load data to run in order to load a single file into a partitioned, internal, parquet table is proportional to the partition and table size.   Then I did an experiment in which I run load time for the same parquet file (renaming it uniquely), which has around 1MB in size, until one partition was filled up with 10k files. Then, I continue filling up the next partition, until 5 partitions were full with 10k files each (full partition has around 12GB in size).   The results are the following:    It is noticeable that load data time increases slightly inside a partition and the next load data time continues from where the previous partition left.   Obs: we are using Ibis framework to do the load data.   Could you please explain a little bit about the Impala load data internals in order to justify increase in time as the table grows?   Thanks, Paulo.

Hello,I'm looking for ways of enforcing specific cloud configurations for CB template to limit and simplify user experience. We'd like to pre-configure regions, VPCs, subnets, encryption configuration and so on. We are using rc-2.9 branch of Cloudbreak.Thanks!

By default with ambari installation, Zeppelin is set to have yarn client mode for Spark Interpreter which means the driver runs in the same host of Zeppelin Server. This incur high memory pressure on the Zeppelin Server host especially when Spark Interpreter is ran in isolated mode.I'm trying to switch to yarn-cluster mode which would let yarn decide on where spark driver should be executed depending of the available resources in the cluster. This mode is supported by Zeppelin since the version 0.8.0 but I'm facing the following issue https://issues.apache.org/jira/browse/ZEPPELIN-3633. Basically, the node where yarn decided to run spark driver doesn't have zeppelin installed so is unable to start. There is a fix on Zeppelin's github https://github.com/apache/zeppelin/pull/3181 but I can't find the files that I need to change. Any chance that this can be fixed easily or should I just install zeppelin on every nodes?

Hi,there are many log files in /var/log/hadoop/hdfs, and file name like hdfs-audit.log.2018-06-01,and some file name like hadoop-hdfs-namenode-hostname are very large.may i delete it all? (this host running namenode, ranger, datanode flume and node manager.)and if any configure can setting those log as auto cycling or auto delete? Thanks

Hello, every one.Currently, I try to use Presto in docker on YARN.I have 3 node manager hosts and 2 resource maneger hosts (this is because resource manager is HA) in my HDP cluster.I completed creating docker containers on all node manager hosts and was able to run Presto server (one of three is presto-coordinator and remain are presto-workers) by using overlay network.Also, I know that YARN supports docker containers and can manage docker containers resource (for example: CPU Usages).Now, in my above use case, I want to know how to configure docker containers resource management by YARN.Thank you.

HI all, A simple question:  It's safe to delete replication logs under /user/hdfs/.cm/ ? Thanks in advance.Ivan

Hi,  I have a good old GSS initiate failed Keberos error message when my application (JAVA) tries to connect to Impala via JDBC. I tried to eliminate all the usuall root causes but this time I think I missed something, because it still not connects:      Cannot connect: connection refused: Java::JavaSql::SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed. The settings are quite standard, MIT Kerberos, kerberized Impala with SSL, here are the params for the JDBC:   -> KDC host: ip-10-85-150-11.eu-west-1.compute.internal-> ImpaladD host: ip-10-85-150-6.eu-west-1.compute.internalJDBC params:-> Host: ip-10-85-150-6.eu-west-1.compute.internal -> Port: 21050-> Additional params in URL: KrbHostFQDN=ip-10-85-150-11.eu-west-1.compute.internal;KrbRealm=HADOOP.DEV.REALM.LOCALL;KrbServiceName=impala;SSL=1;CAIssuedCe

hi allwe want to verify the name nodes status from ambari by API ( namenode adtive/stand by under hdfs )capture.pngis it possible to create API for that?

i want change the following property in cluster-env.xml: <property> <name>recovery_enabled</name> <value>true</value> <description>Auto start enabled or not for this cluster.</description> <on-ambari-upgrade add="true"/> </property> <property> <name>recovery_type</name> <value>FULL</value> <description>Auto start type.</description> <on-ambari-upgrade add="true"/> </property> <property> <name>recovery_lifetime_max_count</name> <value>1024</value> <description>Auto start lifetime maximum count of recovery attempt allowed per host component. This is reset when agent is restarted.</de$ <on-ambari-upgrade add="true"/> </property> <property> <name>recovery_max_count</name> <value>1</value> <description>Auto start maximum count of recovery attempt al

 无法安装cloudera Manager代理程序包 我试图从本地存储库安装cm6.1.0和cdh6.1.0。 系统是Centos7.2  下载包：cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm | 25 MB 00:00:01正在运行事务检查运行事务测试事务检查错误：来自安装了Cloudera管理器 - 代理 - 6.1.0-769885.el7.x86_64的文件/usr/lib/systemd/system/supervisord.service与来自程序包监事，3.1.3-3.el7的文件冲突错误摘要-------------

Hi, i am sorry if  i am postiing this where it should not be posted.New to impala/hive , new to this community site but i need to populate an empty table of one field (date) with dates (from 1990-01-01 till 2099-12-31) and  the  "while" loop does not seem to be working here. Any suggestions,or links  you guys can provide please?Thanks,Tom

I am getting below error when loading Json data from Hive table of type TEXTFILE to ORC type Hive table. Please share your thoughts to overcome from this issue. Thanks.Input sample file TestFile1.json: {"jsondata":[{ "id":1, "m":"Edward the Elder", "cty":"Uited Kigdom", "hse":"House of Wessex", "yrs":"899925" },{ "id":2, "m":"Edward the Elder", "cty":"Uited Kigdom", "hse":"House of Wessex", "yrs":"899925" }]}Cluster version: HDP 2.6.5.0-292TEXFILE Hive Table schema:CREATE EXTERNAL TABLE IF NOT EXISTS TestJson1 (jsondata array<struct<id:int,nm:varchar(30),cty:varchar(30),hse:varchar(30),yrs:varchar(20)>>) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' Location '/data/3rdPartyData/Hive/TestJson1';Load json into above table: load data local inpath '/data/Hive/TestFile1.json' overwrite into table TestJson1;ORC Hive table schema:CREATE EXTERNAL TABLE IF NOT EXISTS TestJson1ORC(jsondata array<struct<id:int,nm:varchar(30),cty:varchar(30),hse:varchar(30),yrs:varc

I am getting an error when querying Hive table over JSON Data. I think the root cause would be JSON SerDe. Should I have to change Hive native JSON SerDe to any other JSON SerDe? Please adviseError: Failed with exception java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: java.io.IOException: Start token not found where expected

what problem?

Hello The documentation for upgrading/migrating the JDK shows installing Java OpenJDK 1.8 using the package manager and therefore from the OS repositories, which for Redhat/CentOS 7 is: su -c yum install java-1.8.0-openjdk-devel Does this mean that the whichever version (1.8.0.191 at time of writing) and all future versions of OpenJDK 1.8 are supported for Cloudera Manager and CDH 6.x? The above command is taken from the documentation here: https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_jdk8.html Thanks,JG

hello everyone,I have a query regarding EvaulateJSONPath processor configuration.whats the significance of Execution in Scheduling of EvaulateJSONPath.I have observed when I select Primary node instead of All nodes, Queue builds up as shown in image and stays the same, dosen't decrease , but if I select All nodes Queue gradually becomes 0.Please suggest.screen-shot-2019-01-17-at-60906-pm.png

Hello Team,   I am considering HA configuration of Kerberized CDH cluster.   Is it possible to avoid SPoF of Load Balancer when constructing Load Balancer such as HAProxy and configuring HA of HiveServer2?For example, is it possible to deploy two Load Balancer with Active-Active and specify two host and port for setting(HiveServer2 Load Balancer property)?Or, should deploy Active and Stanby of Load Balancer be constructed, and when Active fails, should I change the setting(HiveServer2 Load Balancer property) to refer to Stanby? Impalad thinks the same way.   I refer to the following procedure.  * https://www.cloudera.com/documentation/enterprise/6/6.0/topics/admin_ha_hiveserver2.html#concept_u4b_c5d_wv * https://www.cloudera.com/documentation/enterprise/6/6.0/topics/impala_proxy.html#proxy   thank you for reading.  

Hi,   I have upgraded my CM/CDH to 6.1 and every time I login into Hue I receive this error:   Solr server could not be contacted properly: HTTPConnectionPool(host='localhost', port=8983): Max retries exceeded with url: /solr/admin/info/system?user.name=hue&doAs=mpanahi&wt=json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))   The problem is I have never had Solr installed nor configured inside Hue. The Solr is None in Hue inside CM and there is no such thing that refers to [search] or [dashboard] inside my hue_safety_valve.ini so I don't know why it tries to connect to something without me setting it up.   Does anyone have any idea why in the new Hue it tries to connect to Solr without being it installed or selected?   Many thanks,  

Hi all, I have NiFi behind a nginx server proxy, and when I try upload template I recieve HTTP 403. If I access directly I can upload template. Below are my config file nginx.upstream nifi-app { ip_hash; server node-01:8443; server node-02:8443;}server { listen 443 ssl; server_name nifi-server; ssl_certificate /etc/nginx/conf.d/cert.pem; ssl_certificate_key /etc/nginx/conf.d/key.pem; location /nifi { proxy_pass https://nifi-app; proxy_set_header X-ProxyScheme https; proxy_set_header X-ProxyHost myhost.com; proxy_set_header X-ProxyPort 443; proxy_set_header X-ProxyContextPath /; }}Any help will be appreciated

Hi,I ´m not able to create 2 nodes cluster. I´m getting the next error at start:Infrastructure creation failed. Reason: Following tag values are not well formatted: [aws.xxxxxx_xx@xxxxxxxxx.com]('x' obviously are not the real content, it is a valid mail with simple characters.)