Q&A

Hi,In the kerberized cluster, after enabling Ranger Hive plug-in in HDP 2.6.5, not able to run dfs commands in beeline. jdbc:hive2://test.xyz.com:10000> dfs -ls /;Error: Error while processing statement: Permission denied: user [user1] does not have privilege for [DFS] command (state=,code=1)Please help on this.Thank you.

2019-07-10 05:48:38,266 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.1175-1 -> 2.6.5.1175-1 2019-07-10 05:48:38,291 - Using hadoop conf dir: /usr/hdp/2.6.5.1175-1/hadoop/conf 2019-07-10 05:48:38,648 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.1175-1 -> 2.6.5.1175-1 2019-07-10 05:48:38,660 - Using hadoop conf dir: /usr/hdp/2.6.5.1175-1/hadoop/conf 2019-07-10 05:48:38,662 - Group['hdfs'] {} 2019-07-10 05:48:38,665 - Group['hadoop'] {} 2019-07-10 05:48:38,666 - Group['users'] {} 2019-07-10 05:48:38,667 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2019-07-10 05:48:38,669 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2019-07-10 05:48:38,671 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2019-07-10 05:48:38,673 - User['hdfs'] {

Hi All, Could someone please highlight the issues that may occur if we have CDH5.7.5 and the RHEL OS is upgraded to 7.5 and above? Thanks

Hi dear Community, When I want to connect hive with beeline I get this error. Could you help me please. Thanks in advance.   [hive@bdhost008 bin]$ beeline -u "jdbc:hive2://bdhost008:10000/default;principal=hive/bdhost008@TEST.COM"   WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Connecting to jdbc:hive2://bdhost008:10000/default;principal=hive/bdhost008@TEST.COM19/06/28 20:16:45 [main]: WARN jdbc.HiveConnection: Failed to connect to bdhost008:10000Error:

Hi,   For the last day or so, every hour, our cloudera-scm-agent.log on the host for the impala statestore has been presenting the error message:   33923 MonitorDaemon-Reporter throttling_logger ERROR    (59 skipped) Role_extractor for STATESTORE is not initialized correctly.   Has anyone seen this before; is it something to be concerned about?

Using Altus Director 6.2.1 to deploy CDH cluster to Azure via command line using bootstrap-remote.   Per the Cloudera documentation,  I added the following in my cluster bootstrap file in the base instance template:   image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55 useCustomManagedImage: Yes customImagePlan: "" managedDisks: Yes   There is no purchase plan associated with the image, so I left blank per your documentation.   Deployment fails with the following error:   ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is refe

Hi, I want to delete the /var/lib/ cloudera-service/host-monitor files. But when I search everyone is saying its not best to delete these files. But 90% of space is used by them. I know I will loose the chart data history, that wont be a problem. If I delete them, will my production cluster can work normally like before?If not what errors will I get? We are using CDH 5.10.0 parcles. I need your insight on this issue. 

Hi All Is Cloudera suggesting to use S3Guard as a solution for the consistency problem in multi-step ETL? Cause in the reference architecture, the suggested option is to load data from S3 into HDFS and then write back to S3? ThanksCK 

I'm working on a POC to take NN metadata backup. For this, i'm compressing the NN's metadata using a custom script that i will call through ambari API. (https://community.hortonworks.com/articles/139788/running-custom-scripts-through-ambari.html) . If i simply hardcode the datadirectory name it will work. However, i'm looking to pass them as an argument while submitting the request. Anyone aware of the method ?

Hello,I see the path /atsv2 in HDFS have large size, and it growing, this path contain embedded hbase data of Yarn ATS.Have Anyone can explain about this path, and how to purge old data?Thanks.

hi allby the following API we can delete the thrift server component on master01 and master03 nodes # curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/hdp/hosts/master01/host_components/SPARK2_THRIFTSERVER # curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/hdp/hosts/master03/host_components/SPARK2_THRIFTSERVERbut how to know by REST API that thrift server component , not defined on specific node ( after deletion )

hi allwe have two spark thrift servers component in our ambari clusterone thrift server is on master01 machineand the second is on master03 machinehow to stop the thrift server component on master01 and master03 machine by REST API

hi allhow to remove the spark thrift servers component from ambari by REST APIexample before removing the spark thrift serversexample after we delete the thrift servers from ambariexample of manual deletion:

hi allwe upgrade today the ambari version and HDP versionrpm -qa | grep -i ambariambari-metrics-monitor-2.6.2.2-1.x86_64ambari-metrics-collector-2.5.0.3-7.x86_64ambari-agent-2.6.2.2-1.x86_64ambari-server-2.6.2.2-1.x86_64ambari-metrics-grafana-2.6.2.2-1.x86_64ambari-metrics-hadoop-sink-2.6.2.2-1.x86_64when we login to ambari and to stack and version we not seen any info about the ambari new version - 2.6.2.2why? this info not in stak and version ?

hi allwhat is the normal gap between machines in ambari cluster in seconds?for example ( the following example described gap of 20 seconds )on master01Fri Jul 5 07:37:23 UTC 2019but on kafka machine we haveFri Jul 5 07:37:43 UTC 2019

hi allwe are in the middle of rolling upgradewe install the HDP 2.6.5 pkgs from ambarinow we perform rolling upgraderolling upgrade window say that spark thrift servers are down and in maintenance modethis is right we stop the spark thrift services since we are not use the SPARK2_THRIFTSERVER , and not want to do spark upgradeso only spark clients and spark history are up and running but rolling upgrade cant continue because ( according to the rolling upgrade message we must to start the spark thrift services and disable the maintenance mode )so how to continue from this stage?Reason: The following Service Components should be in a started state. Please invoke a service Stop and full Start and try again. SPARK2: SPARK2_THRIFTSERVER (in INSTALLED on host master01.sys674.com)Failed on: SPARK2

hi allwe notice about no option to un-install the installation of the rpm's ( when doing upgrade from 2.6.4 to 2.6.5 )why?

My requirement is to upgrade HDF (not HDP) . Basically my environment have two separate clusters - HDP and HDFI want to upgrade only HDF to 3.4.0 Version from HDF 3.0.2 . Ambari version currently installed is 2.6.1 and HDF version is 3.0.2 . Can you please help me to know if I can directly upgrade to 3.4.0 from 3.0.2 ?I tried looking into this but no luckhttps://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.4.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.htmlCan i directly perform HDF upgrade to 3.4.0 from 3.0.2.5 ? Or does it requires first to upgrade to some version first and then upgrade it to higher version hdf 3.4.0 ?

hi allwe are very confuse because un-consistence recommendation in hortonworks documentsfrom the link: ( https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/upgrading_hdp_stack.html )we can see that: "It is strongly recommended that you first upgrade to Ambari 2.6.2 "NoteBe sure to review the available HDP upgrade scenarios below. It is strongly recommended that you first upgrade to Ambari 2.6.2 before upgrading HDP unless otherwise noted. Ambari 2.6.2 supports patch and maintenance upgrades to HDP 2.6.5.x stack. After upgrading Ambari, be sure the cluster is operating normally and service checks are passing prior to attempting an HDP upgrade.Ambari 2.6.2 does not support IOP-HDP migration. Customers migrating from IOP should use Ambari 2.6.1 to migrate IOP to HDP 2.6.4, then upgrade Ambari to 2.6.2 and use it to upgrade HDP 2.6.4 to HDP 2.6.5.but from ( https://supportmatrix.hortonworks.com/ )we can see that HDP - 2.6.5 can be installed on ambari 2.6.1

I am trying to exchange a partition from staging db after merging the incremental data with the existing one as below:1. Created staging table with partition: CREATE TABLE stg.customers_testcontrol_staging(customer_id bigint,customer_name string, customer_number string,status string,attribute_category string,attribute1 string, attribute2 string, attribute3 string, attribute4 string, attribute5 string) PARTITIONED BY (source_name string) ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' Location('/apps/hive/warehouse/stg.db/customers_testcontrol_staging'2. Inserted data into above table after merging with the base table data  INSERT OVERWRITE TABLE finstg.customers_testcontrol_staging PARTITION (source_name) SELECT t1.* FROM (SELECT * FROM base.customers where source_name='ORACLE' UNION ALL SELECT * FROM external.customers_incre

"File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/conf.server ; /usr/hdp/current/hive-server2-hive2/bin/schematool -initSchema -dbType mysql -userName hive -passWord [PROTECTED] -verbose' returned 1. Your account has expired; please contact your system administrator"

If i use an extract text processor to extract a key and value from a text, then i always get 2 attributes with the same value.For example:If i use test: ([\w .]+) as my expression, then when i process a file with content "test: me" then the single data capture produces two attributes which as far as i can see are exactly the same.myattribute : "me"myattribute.1 : "me"Is there any way i can configure this to get just 1 attribute? Or is there a better way to extract key= value data from a file content?eg. attribute1:500, attribute2:600Many thanks in advance.

hi allwe prepare to upgrade our clusterwe have HDP version - 2.6.4 and ambari - 2.6.1we want to upgrade to the latest HDP 3.1 and latest ambari 2.7.2what are the hortonworks document that explain this upgrade ?is it safe to jump from old HDP and ambari version to the latest ? dose hortonwoks tested this scenario ? ( upgrade HDP 2.6.4 to HDP 3.1 )

I'm trying to upgrade HDP 3.0 to 3.1 but after Ambari upgrade HDP services are not listed in dashboard, instead cluster installation wizard come up.[root@host ~]# curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://host/api/v1/clusters/"{  "href" : "http://host/api/v1/clusters/",  "items" : [ ]}[root@host~]#while I can see services running in background[root@host ~]# jps27457 HMaster28711 TimelineReaderServer25675 jar30637 ZeppelinServer29427 Main6195 HRegionServer2324 AmbariServer31860 Main26869 ResourceManager29013 Main20150 Jps23895 SupportToolServer26391 Main10041 RemoteInterpreterServer4219 NameNode25212 QuorumPeerMain3679 DFSZKFailoverController

Hi guys,I have installed and configured cloudera manager, i decided to clean eveyrithing and install it again, and is failing when distributing parcels