Solved

hi allunder /var/log/zookeeper , we have the zookeeper logs , as the following-rw-r--r-- 1 zookeeper hadoop 104857712 Jul  5 15:54 zookeeper-zookeeper-server-master.sys87.com.log.1-rw-r--r-- 1 zookeeper hadoop       177 Jul 16 11:57 zookeeper-zookeeper-server-master.sys87.com.out-rw-r--r-- 1 zookeeper hadoop  66358998 Jul 16 23:15 zookeeper-zookeeper-server-master.sys87.com.logthe problem is that we get only the INFO severity from the logs!and when we set the TRCE severity , and restart the zookeeper service , then its not change the zookeeper log content to trace mode,can we get example of log4j for zookeeper ?we suspect that our log4j isn't configured properly

Hello everyone.I would really appreciate some help/advice with this.I run HDP Sandbox 3.1 on in a VM with 32GB memory, and plenty of disk space, for educational purposes. However, as soon as there more than two or three logged in users running Hive queries, the jobs get stuck in the queue, with one job listed as "Running", and everything else as "Accepted". And those are not even big jobs -- simply some exercises from HDP Sandbox tutorials.Can anyone suggest, is it possible to run HDP Sandbox with 25 to 50 concurrent users, working on some basic exercises from HDP Sandbox tutorials? What would I need to change in the configs for that? Many thanks!

Everytime I try to restart clean restart fails but nifi but nifi will be running. Also tried restarting from cli same issue. Because of this I am unable to apply any conf changes from Ambari. Attached screenshot of the exception. It reads asstderr: Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart self.status(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 156, in status check_process_status(status_params.nifi_node_pid_file) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning The above exception was the cause of the following exception: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in Master().execute() File "/usr/lib/amb

I have enable stats using `COMPUTE STATS` on a table. After this when I re-run the query, it's giving me an error `A RIGHT OUTER JOIN type with no equi-join predicates can only be executed with a single node plan`. Dropping the stats is execuring the query. I'm trying to improve the query performance.

On a perfectly running HDP3.1 cluster, one time the ATSv2 Timeline Reader stopped responding. Following some guide I've tried refreshing the service by cleaning the configs and restarting it byyarn app -destroy ats-hbaseAfter this, restarting the service doesn't help, and starting the service by yarn app -start ats-hbasedoesn't work, since the config json is missing:ERROR client.ApiServiceClient: File does not exist:  <>/user/yarn-ats/.yarn/services/ats-hbase/ats-hbase.jsonWhat is this .json and how do I get it running again? Where are some example templates of a working .json that I could use? I remember this file existing somewhere on a fresh install, but sadly reinstall of the cluster is not an option. Another thing, trying a fresh install on my laptop VM does not create such json file too -- I guess the reason is that the laptop is too small for enforcing hbase use?Didn't find anything in the official documentation, so would be thankful for any advice!@Geoffrey Shelton O

I'm using Altus Director 6.2.    Is the following declaration in your latest Aluts Director documentation still accurate?   Changing the Instance Type is Not Supported in Azure Changing the instance type of an already-deployed VM is not supported in Azure. Changing the instance type of a VM through a tool external to Altus Director is not supported in Azure. You cannot, for example, use the Azure Portal to change the instance type. Altus Director is not updated when instance type changes are made with external tools, and your cluster will show errors in Altus Director. I have performed both of the actions described in the bullet points above after we decided our existing workers were not sized properly for the amount of data being ingested / processed.   Cloudera Director still shows the cluster in a "green" state.  No complaints / errors / warnings thus far.   Is this really not supported?   Reference: https://www.cloudera.com/documen

Hi Team, How to perform TPC-DS test/workloads on hive,impala,spark? how to analyze the results?  Thanks,Harish

Hello team,   Cloudera Navigator Metadata Server failed to start, no log file getting updated in /var/log/cloudera-scm-navigator for metaddata server.   Only Cloudera server shows below errors.   CDH version: 5.16.1   Kindly suggest?   Error:   2019-06-20 13:49:21,099 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com}2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.Cmd

Hello,   I need to improve speed for my select query currently I am testing how best I can get from Impala with given hardware.   I have created a parquet table with partitions and block size = 512mb. Here is some more details -   [quickstart.cloudera:21000] > show table stats tbl_parq_512m;Query: show table stats tbl_parq_512m+-----------+----------------+------+------------+-----------+--------+----------+--------------+-------------------+---------+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| source_ip | destination_ip | year | event_date | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |+-----------+----------------+------+------------+-----------+-------

Hi, I have a table with a lot of data,I want to create a new table based on some column values from this based which method is most efficient and cluster resources friendly Pseudo-Code 1. single job        insert into myNewTable         select * from myOldTable         where a=xxx etc. 2. two jobs:    job1. create datafame from select statement               select * from myOldTable         where a=xxx etc. as dataframe     job2 write dataframe as new table         insert into myNewTable select from dataframe

We have failed to start ambari-metrics-collector.The following error appeared in hbase-ams-master.log.I can not find another ERROR, what should I check?----------------/var/log/ambari-metrics-collector/hbase-ams-master-host.log2019-07-11 15: 34: 41,040 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: Master not initialized after 200000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent (JVMClusterUtil.java: 229) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup (JVMClusterUtil.java: 197) at org.apache.hadoop.hbase.LocalHBaseCluster.startup (LocalHBaseCluster.java:413) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster (HMasterCommandLine.java: 232) at org.apache.hadoop.hbase.master.HMasterCommandLine.run (HMasterCommandLine.java: 140) at org.apache.hadoop.util.ToolRunner.run (ToolRunner.java: 76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain (ServerCommandLine.java

Hi,In the kerberized cluster, after enabling Ranger Hive plug-in in HDP 2.6.5, not able to run dfs commands in beeline. jdbc:hive2://test.xyz.com:10000> dfs -ls /;Error: Error while processing statement: Permission denied: user [user1] does not have privilege for [DFS] command (state=,code=1)Please help on this.Thank you.

2019-07-10 05:48:38,266 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.1175-1 -> 2.6.5.1175-1 2019-07-10 05:48:38,291 - Using hadoop conf dir: /usr/hdp/2.6.5.1175-1/hadoop/conf 2019-07-10 05:48:38,648 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.1175-1 -> 2.6.5.1175-1 2019-07-10 05:48:38,660 - Using hadoop conf dir: /usr/hdp/2.6.5.1175-1/hadoop/conf 2019-07-10 05:48:38,662 - Group['hdfs'] {} 2019-07-10 05:48:38,665 - Group['hadoop'] {} 2019-07-10 05:48:38,666 - Group['users'] {} 2019-07-10 05:48:38,667 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2019-07-10 05:48:38,669 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2019-07-10 05:48:38,671 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2019-07-10 05:48:38,673 - User['hdfs'] {

Hi All, Could someone please highlight the issues that may occur if we have CDH5.7.5 and the RHEL OS is upgraded to 7.5 and above? Thanks

Hi dear Community, When I want to connect hive with beeline I get this error. Could you help me please. Thanks in advance.   [hive@bdhost008 bin]$ beeline -u "jdbc:hive2://bdhost008:10000/default;principal=hive/bdhost008@TEST.COM"   WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Connecting to jdbc:hive2://bdhost008:10000/default;principal=hive/bdhost008@TEST.COM19/06/28 20:16:45 [main]: WARN jdbc.HiveConnection: Failed to connect to bdhost008:10000Error:

Hi,   For the last day or so, every hour, our cloudera-scm-agent.log on the host for the impala statestore has been presenting the error message:   33923 MonitorDaemon-Reporter throttling_logger ERROR    (59 skipped) Role_extractor for STATESTORE is not initialized correctly.   Has anyone seen this before; is it something to be concerned about?

Using Altus Director 6.2.1 to deploy CDH cluster to Azure via command line using bootstrap-remote.   Per the Cloudera documentation,  I added the following in my cluster bootstrap file in the base instance template:   image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55 useCustomManagedImage: Yes customImagePlan: "" managedDisks: Yes   There is no purchase plan associated with the image, so I left blank per your documentation.   Deployment fails with the following error:   ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is refe

Hi, I want to delete the /var/lib/ cloudera-service/host-monitor files. But when I search everyone is saying its not best to delete these files. But 90% of space is used by them. I know I will loose the chart data history, that wont be a problem. If I delete them, will my production cluster can work normally like before?If not what errors will I get? We are using CDH 5.10.0 parcles. I need your insight on this issue. 

Hi All Is Cloudera suggesting to use S3Guard as a solution for the consistency problem in multi-step ETL? Cause in the reference architecture, the suggested option is to load data from S3 into HDFS and then write back to S3? ThanksCK 

I'm working on a POC to take NN metadata backup. For this, i'm compressing the NN's metadata using a custom script that i will call through ambari API. (https://community.hortonworks.com/articles/139788/running-custom-scripts-through-ambari.html) . If i simply hardcode the datadirectory name it will work. However, i'm looking to pass them as an argument while submitting the request. Anyone aware of the method ?

Hello,I see the path /atsv2 in HDFS have large size, and it growing, this path contain embedded hbase data of Yarn ATS.Have Anyone can explain about this path, and how to purge old data?Thanks.

hi allby the following API we can delete the thrift server component on master01 and master03 nodes # curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/hdp/hosts/master01/host_components/SPARK2_THRIFTSERVER # curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/hdp/hosts/master03/host_components/SPARK2_THRIFTSERVERbut how to know by REST API that thrift server component , not defined on specific node ( after deletion )

hi allwe have two spark thrift servers component in our ambari clusterone thrift server is on master01 machineand the second is on master03 machinehow to stop the thrift server component on master01 and master03 machine by REST API

hi allhow to remove the spark thrift servers component from ambari by REST APIexample before removing the spark thrift serversexample after we delete the thrift servers from ambariexample of manual deletion:

hi allwe upgrade today the ambari version and HDP versionrpm -qa | grep -i ambariambari-metrics-monitor-2.6.2.2-1.x86_64ambari-metrics-collector-2.5.0.3-7.x86_64ambari-agent-2.6.2.2-1.x86_64ambari-server-2.6.2.2-1.x86_64ambari-metrics-grafana-2.6.2.2-1.x86_64ambari-metrics-hadoop-sink-2.6.2.2-1.x86_64when we login to ambari and to stack and version we not seen any info about the ambari new version - 2.6.2.2why? this info not in stak and version ?