Q&A

I use IntelliJ IDE installed on my Windows 10 laptop and try to run spark job in yarn mode on my 5 node HDP 3.1.1 cluster.My codes:package p1import org.apache.spark.sql.{SparkSession, functions => F}import org.apache.log4j.{Logger, Level}object SparkDeneme extends App { Logger.getLogger("org").setLevel(Level.INFO) val spark = SparkSession.builder() .appName("SparkDeneme") .master("yarn") .config("spark.hadoop.fs.defaultFS","hdfs://node1.impektra.com:8020") .config("spark.hadoop.yarn.resoursemanager.address","node1.impektra.com:8030") .getOrCreate()} import spark.implicits._ val sc = spark.sparkContext val dfFromList = sc.parallelize(List(1,2,3,4,5,6)).toDF("rakamlar") // dfFromList.printSchema() dfFromList.show()When I run get following error:19/07/26 20:00:32 ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Could not parse Master URL: 'yarn'    at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$c

Hi all, We have currently a Hadoop cluster in test in a private cloud with private ip and hostname:- <private_ip_1> mh1.example.com (namenode 1 in private cloud)- <private_ip_2> mh2.example.com (namenode 2 in private cloud)- <private_ip_3> w1.example.com (worker 1 in private cloud)...- <private_ip_n> wn.example.com (worker n private cloud)- <private_ip_m> eg1.example.com (edge node 1 + cloudera manager service in public cloud)- <private_ip_o> eg2.example.com (edge node 2 + hue etc. in public cloud) We can access on the private cloud only through the edge nodes and the cluster is kerberized with sentry.For the moment we use only self-certificate, but now we want to put the cluster in production. My question is that: Could we use official authority certificate in our case ? Because if I undestand well, authority certificate is only for public domain and public IP. Or what I was thinking for production is:- Self-Certificate for Cloud

Hi everyone, I'm using command in subject to fetch all yarn applications and use grep on the output to filter out a specific application . I wanted to find out what is the limit of this command, as in how far in the history does it go to get these states. Is this limited by the job history server limit or does it return something below that like last 1000 jobs etc? Regards

Confused about what Ambari users and groups are.When looking at the docs and the Ambari UI (Admin/Users and Admin/Groups), I get the impression that users / groups created in this interface should appear across all nodes in the cluster, but this does not seem to be the case, eg...[root@HW01 ~]# id <some user created in Ambari UI>id: <some user created in Ambari UI>: no such userSame situation for groups created in Ambari UI admin section.Not sure I understand to use of the Ambari users and groups if they do not somehow have a link back to user and groups locally on the hosts. Can someone please explain what is going on here?

We have a slow query like: select max(partition_col_1) from some_table where partition_col_2 = 'x' and it's super slow, scanning all records (hundreds of billions) in the filtered partitions, even though it's actually not getting anything out of them... select only includes partitioning column.  Absolutely no need to read any files I don't think. Any way or hint to get around this?

I'am struggling with Expression Language concerning nested and-/or-conditions.In my UpdateAttribute the variable KONTO_IST_AN should be getting the result in boolean which is provided by this expression. Sample content of RESPONSE (always exactly one element at one time):[{"antauschbis":"20191231","antauschvon":"20190101","id":"(422)"}]  => expected result: true[{"antauschbis":null,"antauschvon":"20190101","id":"(422)"}]  => expected result: true[{"antauschbis":"20190531","antauschvon":"20190101","id":"(422)"}]  => expected result: false[{"antauschbis":null,"antauschvon":"20200101","id":"(422)"}]  => expected result: falseExpression:${RESPONSE:jsonPath('$.[0].id'):equals(${KONTO_BV}):   and(${RESPONSE:jsonPath('$.[0].antauschvon'):le(${now():format("yyyyMMdd")}):        and(${RESPONSE:jsonPath('$.[0].antauschbis'):ge(${now():format("yyyyMMdd")}):              or(   

Not sure if this is the right subsection of this community forums but I am curious if there are fundamental changes that would prevent a custom CSD that was written for (and works for) a 5.x CM from working on a CM 6.x build. I will more than likely end up testing this myself soon but I guess I am wondering if there any caveats that I should be aware of or watch out for. Cheers!

hi allis it necessary to restart the ambari server after we delete ambari service by REST API ?I as asking because after we delete some service from the ambari by APIwe saw that service really Disappeared from ambari dartboard so I not sure if we need also to restart the ambari server to take affect?

Attempting to use / startup HDFS NFS following the docs (ignoring the instructions to stop the rpcbind service and did not start the hadoop portmap service given that the OS is not SLES 11 and RHEL 6.2), but running into error when trying to set up the NFS service starting the hdfs nfs3service:[root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" [root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# service nfs status Redirecting to /bin/systemctl status nfs.service Unit nfs.service could not be found. [root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# service nfs stop Redirecting to /bin/systemctl stop nf

Trying to upload a simple csv file into HDFS via the Ambari files view, getting the error below java.net.NoRouteToHostException: hw04.ucera.local:50075: No route to host (Host unreachable)     at java.net.PlainSocketImpl.socketConnect(Native Method)     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)     at java.net.Socket.connect(Socket....For reference, here the hosts on the clusterand the services on the HW04 nodeAccumulo ClientDataNodeHBase ClientHDFS ClientHive ClientHST AgentInfra Solr ClientLog FeederMapReduce2 ClientMetrics MonitorOozie ClientPig ClientSpark2 ClientSqoop ClientTez ClientYARN ClientZooKeeper ClientNot sure what the error m

When I run: hive --service metastore, it raised errors as below /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 24: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 25: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 26: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-common.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 27: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 28: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.

Are the ODBC Drivers for Hive the same for Cloudera and Hortonworks? Flexera indicates they are the same product due to the merger. However, there are downloads for both products, and they are not the same version: Cloudera 2.6.4: https://www.cloudera.com/downloads/connectors/hive/odbc/2-6-4.html Hortonworks 2.6.1.1: https://www.cloudera.com/downloads/hdp.html

Hello,We have upgraded our cluster and are getting error " The Oozie Server build version(4.1.0-cdh5.16.2) and the Oozie Server shared library version(4.1.0-cdh5.14.2) do not match". We have done the following:Upgrade the Oozie SharedLibGo to the Oozie serviceIf the Oozie service is stopped, start it:Select Actions > Start and click Start to confirm.Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm. The service became Green from some time and then again its Red, kindly suggest / guide path to fix this  

Indent to call sqoop on a client node and need to be able to connect to an oracle DB. Currently seeing error...19/07/25 10:48:55 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriverWhen looking into this, appears need to install the proper jdbc driver for sqoop to use. However, the only docs I can find on installing sqoop drivers is for Cloudera CDP and following these docs runs into problems where certain expected directories do not exist on my cluster (particularly /var/lib/sqoop) (and I am assuming that they should already be there and not need to be manually created). Are there any docs on doing this for HDP? Docs for doing this just through the Ambari UI?

hi allwe have production cluster with HDP - 2.6.4 versionwe have 186 data-node machines ( DELL MACHINES WITH 10 disks )we try to re balance the data on the disks so disks will be with the same used size but without successwe feel that 2.6.4 version not have the tools that support re balance!!!as I mentioned on each data-node machine we have 10 disks while each disk is 1.8Tand some of the disks are 55% usedand some of them are only 1% usedso we have non balanced disks ( its like some disk are not useful ) , but why HDFS not balanced the data on all disks??my question - from which HDP version , we can re balance the data-node disks ?dose 2.6.5 version support re balance ?or from 3.X ?please advice , what we can do ?as I mentioned this is very huge cluster andwe get the bad feeling that the current HDP version ( 2.6.4 ) not support any re balance - is it true?example /dev/sdc                 3842878616 357409860 3485452372  10% /data_hdfs/sdc/d

Dear colleges and friendsI just reviewing on the HDP upgrade hortronworks documentation but not see the section about backup the folder - /var/hadoop/hdfs/namenode/currentis it necessary to backup the folder - current before HDP upgrade ?astar -zcvf current_bck.tar.gz /var/hadoop/hdfs/namenode/current-rw-r--r-- 1 root root 9062640 Jul 22 09:28 current_bck.tar.gz

Dear all , nice to be here , with all professional folkswe have ambari server , and all services shown from "Add Service Wizard"as click on actions button , and click on "+add Service"there we can found all available servicesso nowhow to get all these services list by API

Hi, I'm trying to deploy a few COORDINATOR_ONLY impalad on edge nodes in my cluster (CDH 5.16.1). HDFS short-circuit read are enabled on the cluster and works fine.When starting the COORDINATOR_ONLY nodes, I'm getting following error: Invalid short-circuit reads configuration: - Impala cannot read or execute the parent directory of dfs.domain.socket.pathAborting Impala Server startup due to improper configuration. Impalad exiting.Is there a way to run COORDINATOR_ONLY impalad with hdfs short circuit reads enabled. Thanks in advance!

hi allwe have small cluster ( ambari version - 2.6.1 )3 masters machines3 kafka machines 3 workers machinesis it possible to removed completely the HDP , for new fresh HDP version ?Now we have HDP - 2.6.5And we want to install new HDP - 2.6.4

We have a security requirement to be on TLS 1.2 by the end of this year, so we need to know is 1.2 compatible with the current version of CDH we are on 5.13. If not what is not supported ?And other part of my question is whether the TLS version 1.2 will run on CDH version 5.13 supporting all the applicable components/services ?Also we would like to confirm the Postgress of KTS supports TLS 1.2. ?

Ran into an interesting scenario in Director that I have not solved yet but will (hopefully) today.   Created a cluster via the python sdk and when attempting to repair a node in any instance group I get "A template must be specified". There is nothing in the logs and this instance group only contains gateway roles for HDFS and YARN.   This is Director v6.1 and there are no "templates" in the templates tab since everything was programmatically generated.   Screenshot for reference:

Hello all, I'm a little new in Haddop world, so, please forgive me any kind of questions that I'm doing.In first place, I installed a 7 node cluster in my environment, all of them has DataNode (I know now that's the worst scenario, but, it's already installed and no time to fix it).In this morning, a simple process that need to drop a partitioned table in Hive isn't working anymore, there was no modification on cluster.The simple command is "drop table ods.tims_declaracao;".Well, I think that I got the problem worse, I removed the directory "/apps/hive/warehouse/ods.db/tims_declaracao". I thought that I would drop the table, well, I realize that it was not.The table still hanging in drop the metadata. Can some good soul help me to find out what's the problem?Regards.

 Hi,   cloudera manager url is not opening. This is what I have checked so far. 1. All the services are running: Mysql, Cloudera-scm-agent, Cloudera-scm-server 2.  /etc/cloudera-scm-server/db.properties # Auto-generated by scm_prepare_database.sh on Wed Jul 10 05:13:24 EDT 2019 # # For information describing how to configure the Cloudera Manager Server # to connect to databases, see the "Cloudera Manager Installation Guide." # com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=servername com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=password 3.  /etc/my.cnf configuration:- [mysqld] datadir=/var/lib/mysql socket=/var/run/mysql/mysql.sock transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0   key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_

Trying to access Ambari Files View, getting error when the View tries to load...As the error message is so terse, not sure what to do here for further debugging (new to HDP, but HDFS appears to be running normally and not sure which if any of the HDFS namenode logs would be relevant here). Any suggestions or fixes?

Often when provisioning clusters nodes will be cancelled due to Cloudera Director not being able to open more file handles:   [2019-07-23 17:31:18.585 +0000] ERROR [p-ebcce1c842e9-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging - com.cloudera.launchpad.pipeline.util.PipelineRunner: Attempt to execute job failedjava.net.SocketException: Too many open files at java.net.Socket.createImpl(Socket.java:460) at java.net.Socket.connect(Socket.java:587) at net.schmizz.sshj.SocketClient.connect(SocketClient.java:126) at com.cloudera.launchpad.sshj.SshJClient.attemptConnection(SshJClient.java:343) at com.cloudera.launchpad.sshj.SshJClient.attemptConnection(SshJClient.java:318) at com.cloudera.launchpad.sshj.SshJClient.access$000(SshJClient.java:68) How to increase the file handles as ulimit and limits.conf do not seem to work?