Solved

Hi, If I set the following property in Impala config for admission control and pool "default", in llama-site.xml: <property>   <name>llama.am.throttling.maximum.placed.reservations.root.default</name>   <value>4</value></property> Suppose I have 3 Impala nodes and all of them have the same configuration above.Does that mean I can run up to 12 concurrent queries (4 in each impalad) in default pool or just 4 overall? Thanks,Paulo. 

Using docker, I've succesfully installed HDP sandbox on my cloud-base CentOS 7 instance. Most services starts up and I see a lot of green in Ambari. Now, i am curious about why the deployed sandbox named my namenode: sandbox-hdp.hortonworks.com?i see that the same namenode is used in the screenshot attached to this post - as disclosed by the URL in the web-browser https://community.hortonworks.com/questions/155768/docker-sandbox-hdphortonworkscom50070-connection-r.htmlIs it expected that a user should tinker with the hosts file to translate that give name to the IP address of the node?That is strange since most of us do not own the hortonworks.com domain. Is this a business-as-usual way? is there a way to change the name of a namenode?

Hi,I have 3 versions of HDP in oracle virtual box. The versions are 2.5, 2.6.5 and 3.0. When i start VM for HDP 3.0 the Ambari UI works fine. But after that if i start VM for HDP lower versions 2.5 or 2.6.5 i can not open Ambari UI. It shows Server error. Based on the requirement i need to use different versions of HDP.How can i resolve this issue ?Thanks,Heta

hello,We have installed a secured hdp 3.1 cluster on Centos 7.5.Then we installed mpack in order to add a nifi single node.The unsecure version worked correctly (at least it displayed the ui correctly ) but upon activating ssl ( with auto generated certificate ) and activating kerberos for authentification, when connecting, we got the following error :Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not connectedThis is strange because we use the secure version and connect to nifi via https://my_nifi_FDQN_node:9091/nifi/ and it should not try to connect to 9090.In nifi-user.log, we can see :2019-02-22 11:19:10,767 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for my_ldap_user2019-02-22 11:19:10,772 INFO [NiFi Web Server-21] o.a.n.w.a.c.IllegalClusterStateExceptionMapper org.apache.nifi.cluster.manager.exception.IllegalClusterStateException: Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not co

Hello,Can anyone explain how to rise speed of "rest api"?Last month, the CPU load began to grow on ambari server and appeared alert "Ambari server performance rest api Critical 25,000ms"I have 4 CPU core and 32GB ram on ambari hostThe cluster is contain 25 nodes

I am managing CDH 5.6.0 consisting of 16 Linux servers. (1 eco, 2 namenodes, 13 datanodes) The Cloudera Hadoop has no trouble issue, but nowadays, the rate of analyzing process was increased. Anyone associated with analyzing can access CDH hive by using SQL communication program(ex, dbeaver).  What concerns me is termination of the CDH cluster, because, they send so many queries to the hive. Is there the way monitoring client IP accessing to hive?Thank you.

Hi. I am using Embedded PostgreSQL for CM. Backed up as described in the document below. https://www.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ag_backup_dbs.html#cmig_topic_5_6_3pg_dump -h hostname -p 7432 -U scm > /tmp/scm_server_db_backup.$(date +%Y%m%d) But trying to restore fails. [root@localhost ~]$ psql -w -h localhost -p 7432 -U scm -f /tmp/scm_server_db_backup.$(date +%Y%m%d)↓psql:scm_server_db_backup:54: ERROR: relation "audits" already existsALTER TABLEpsql:scm_server_db_backup:77: ERROR: relation "client_configs" already existsALTER TABLEpsql:scm_server_db_backup:89: ERROR: relation "client_configs_to_hosts" already existsALTER TABLE...psql:scm_server_db_backup:1036: ERROR: duplicate key value violates unique constraint "audits_pkey"DETAIL: Key (audit_id)=(1) already exists.CONTEXT: COPY audits, line 1psql:scm_server_db_backup:1056: ERROR: duplicate key value violates unique constraint "client_configs_pkey"DETAIL: Key (cl

I created a similar job as in : https://community.hortonworks.com/content/supportkb/48985/how-to-setup-oozie-shell-action.htmlOn execution the yarn log shows as success.But I can't see the /tmp/output file being created anywhere. I checked on the local linux host as well as hdfs.One question I have is in job.properties -oozie.coord.application.path=${nameNode}/user/root/apps/shellShould the configuration files (job.properties, coordinator.xml etc) reside in the above directory? I have them there.Not sure what is happening.Appreciate the insights.

All-   I have a cluster of 3 VM's within a lab. Memory/RAM is not an issue. I recently went to upgrade to Spark 2.3 which I have done before without issues. Originally I could not get Spark2 to start- it would fail on the 7th/8 configuration updates (Kafka). Unfortunately, I seem to be having numerous issues this time around. In researching my issue with Spark 2 not starting, I came across the following post: https://community.cloudera.com/t5/Cloudera-Manager-Installation/deployClientConfig-on-service-Spark-fails-in-CDH5/td-p/17640/page/2One of the employee posts said the following:We've seen this issue occassionally; you can work around the problem by deleting any empty alternatives under /var/lib/alternatives and restart cloudera-scm-agent ("service cloudera-scm-agent restart"). The agent will recreate any missing alternatives then.This was the "beginning of the end" for me. I did exactly what it stated, but upon restarting the agent I had numerous issues within the Cl

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder?Exampleexport hdfs_folder=/app/lib export local_folder=/home/work_appwhile under /app/lib we have the following sub folder with the jar filesas/app/lib/folder_jar1/app/lib/folder_jar2Under each of above folder we have jar filesThe following command , will copy only the jar files under /app/lib but not the Jar files under the sub folders as /app/lib/folder_jar1 , /app/lib/folder_jar2 , Etc hadoop fs -copyToLocal $hdfs_folder/*.jar $local_folder

Hi, There is a program that uses Impyla to retrieve data from the local Impala daemon. cursor.execute("select * from table;")rows = cursor.fetchall() The table has 5 million rows, the number of columns is 9, the file size at the time of CSV conversion is about 200 MB.There are four data nodes.Memory is 32 GB. Despite just that much data, fetchall () takes over 200 seconds. Query execution ends in 0.2 seconds Why is it so slow? Do you have any ideas to speed up something?  Thanks! 

I am trying to build Impala on ARM/aarch64, But I did not find any useful guide/document to read. I noticed that there are some assembly code in Impala and third-party code, Has anyone tried building it on ARM?Best RegardsYork

Hello, I am planning for CCA 175 exam. Can i know what is the minimum hardware configuration is required? I mean what should be minimum RAM you require in order to give the exam?

Hi all, Currently using CDH 5.14.4 and looking to enable user authention on HiveServer2 using OpenLDAP. The two connection options I'm seeing are LDAP and LDAPS, but we currently don't have LDAPS configured with our OpenLDAP server. Hue supports LDAP with StartTLS so I figured Hive would too. I'm wondering if StartTLS is an option that I'm not finding documentation for or if its not supported.  Thanks for your help!  

Am trying to add a new host to a cloudera managed cluster using wizard ... but it keeps failing with message "/tmp/scm_prepare_node.fj3OyPGRConnection reset " and server logs shows ... 2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=sw, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=true, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE>, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=20, skipPackageInstall=false, skipCloudConfig=false, hosts=[hadoop.localdomain.38], existingHosts=[]}.2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.fl

can we deploy the HttpFS role on more than one node? is it best practice?   we had a problem with one of the namenodes (on node02) in our cluster and it was down for almost 20 hours.once node02  was up, the active namenode ,on node01, started to alert that its not syncronizedafter restart of the namenode on node02 the alert stopped now active namenode is on node01standby namenode is on node02we can see that new logs are created on opt/hadoop/dfs/nn/current  on the actine namenode on node01but no new files . on the standby namenode no node02 - is it OK ??

Hi, I recently started learning Hadoop/Cloudera. I have successfully installed single node Cloudera Manager on my test Linux workstation I am looking for the best or suggested IDEs for creating projects to run jobs on my Hadoop fs using Scala, Java, Python etc. Any how-to docs will be greately appreciated Regards All

All three namenode directories are pointing to same metadata./hadoop/hdfs/namenode/var/hadoop/hdfs/namenode/var/log/hadoop/hdfs/namenodeWhat is the use of having 3 directories pointing to same data?

I am loading a phoenix table but in hbase this table row key is shown as hex values. how can I correct this to see the ASCII values ? that is the integer valuesqlline version 1.1.80: jdbc:phoenix:localhost:2181:/hbase-unsecur> SELECT * FROM EXAMPLE;+--------+-------------+------------+| MY_PK  | FIRST_NAME  | LAST_NAME  |+--------+-------------+------------+| 12345  | John        | Doe        || 67890  | Mary        | Jane       |+--------+-------------+------------+2 rows selected (0.053 seconds)Version 1.1.2.2.6.5.0-292, r897822d4dd5956ca186974c10382e9094683fa29, Fri May 11 08:01:42 UTC 2018hbase(main):001:0> scan 'EXAMPLE'ROW                                              COLUMN+CELL \x80\x00\x00\x00\x00\x0009               &nbs

Hi Team,We have enabled Atlas in our environment. After that we started getting connection hanging to Hive/HS2/Knox Gateway.Hiveserver doesn't create new thread connections and stops responding. when we do restart of HS2 then the new jobs start but after sometime again connection goes in hung state.This issue was a result of recently configured atlas service on the cluster. After that followed below link and changed the configuration.https://community.hortonworks.com/content/supportkb/148579/failed-hive-internal-error-javautilconcurrentrejec.htmlatlas.hook.hive.maxThreads=50 from 5.atlas.hook.hive.minThreads=5.Even i checked my user limit for hive it has been set:root@hostname:~# su - hive$ ulimit -atime(seconds) unlimitedfile(blocks) unlimiteddata(kbytes) unlimitedstack(kbytes) 8192coredump(blocks) 0memory(kbytes) unlimitedlocked memory(kbytes) 64process 16000nofiles 32000vmemory(kbytes) unlimitedlocks

Hi, We have CDH 5.15 on our production cluster with JDK 8,What is your recommendation about moving to G1GC or stay with the CMS,didn't find any clear instruction about it.Does it depend on the CDH version? or on the components I'm using? Thanks

Hi, We have CDH 5.15 on our production cluster with JDK 8,What is your recommendation about moving to G1GC or stay with the CMS,didn't find any clear instruction about it.Does it depend on the CDH version? or on the components I'm using? Thanks

Hi All,   We have sqooped data from MS-SQL 2008 into HDFS and created external tables . The record count shows same between the MS-SQL table and Hive tables. However, if i fire same query on MS-SQL table and Hive table, the output is different. The query is being fired through Hue(tested through Impala as well). Could you please help, if i missed something?   Regards. Sandeep Suman

How can I change the Apache kylin port ?The default listen port is 7070, I have salt-bootstrap running on the Azure VMs in that port.I went through kylin.properties and didn't find a relevant property for listen port.

Hello. I am trying to run a TPC-C benchmark on Impala with Kudu and therefore completed the necessary system installation and JDBC driver download. However, the driver documentation states that the AutoCommit feature is always set to true, which means explicit commits and rollbacks are not possible.  I am therefore wondering, if the TPC-C benchmark can be valid when using the Impala JDBC driver, as the TPC-C specification states that rollbacks must happen at a fixed rate? Is there a workaround that can be used instead? Thanks for your time.