With HIVE-13670 Till today we need to remember the complete Hive Connection String either you are using direct 1000 port or ZK connection string. After the above Jira we can optimize that by setting up the environment variable(/etc/profile) on the Edge nodes. export BEELINE_URL_HIVE="<jdbc url>" Example: export BEELINE_URL_HIVE="jdbc:hive2://<ZOOKEEPER QUORUM>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" Now just type beeline -u HIVE Even we can setup multiple connection strings just by setting different naming connections like BEELINE_URL_BATCH, BEELIVE_URL_LLAP. Hope this helps you. ... View more

The reason why Ambari is unable to start Namenode smoothly is bug and below is the workaround. Issue got fixed permanently in Ambari 2.5.x. Few lines of Error message from Ambari Ops logs: File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 55, in wrapper return function(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 562, in is_this_namenode_active raise Fail(format("The NameNode {namenode_id} is not listed as Active or Standby, waiting..."))resource_management.core.exceptions.Fail: The NameNode nn2 is not listed as Active or Standby, waiting... ROOT CAUSE: https://issues.apache.org/jira/browse/AMBARI-18786 RESOLUTION: Increase the timeout in /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py from this; @retry(times=5, sleep_time=5, backoff_factor=2,err_class=Fail) to this; @retry(times=25, sleep_time=25, backoff_factor=2,err_class=Fail) ... View more

Below script is used to get all the Hive Databases and underlining tables or views or INDEX_TABLES in a cluster into a csv file. This helps in evaluating total counts for metrics or identifying any tables: beeline -u 'jdbc:hive2://zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' --outputformat=csv2 -f table.hql > tableslist.csv In table.hql !tables Hope this helps you. ... View more

Today we are using below command to sync all the AD users to get access to Ambari Views/Operation server. ambari-server sync-ldap --groups groups.txt We can't schedule cron tab because we need to pass admin credentials at the time of sync. It's hard to run daily as the users get added to the group and deleted which are managed in AD. By using below curl command we can automate the Ambari LDAP sync. We can add this command to bash script and schedule it using confab. Whenever new users get added/deleted to the LDAP group, it will get automatically synced the same with Ambari Views. We need to run below command as Ambari user. This can avoid running ‘ambari-server sync-ldap’command. Here even no need to run any unset proxy commands. curl -k -u admin:<password> -H 'X-Requested-By: ambari' -X POST -d '[{"Event": {"specs": [{"principal_type": "users", "sync_type": "specific", "names": "sredd66,sreddyuw"},{"principal_type":"groups","sync_type":"specific", "names": "hadoopadmin,hadoopusers"}]}}]' https://ambarihostname:8080/api/v1/ldap_sync_events Steps to remove an LDAP group from Ambari and still have the group in LDAP: Command to GET/DELETE groups from Ambari by not touching AD. GET: curl --insecure -u admin:<password> -H 'X-Requested-By: ambari' -X GET https://ambarihostname:8080/api/v1/groups/ DELETE: curl --insecure -u admin:<password> -H 'X-Requested-By: ambari' -X DELETE https://ambarihostname:8080/api/v1/groups/<group name you wanted to delete> Let me know if you have any questions. ... View more

If you are not using any Hive Authorization like SQL Standard Authorization or Ranger and you don't want other to read/write data and allow the only owner to read & write, and allowing the group to read. Changing the warehouse directory permission. 1.From the command line in the Ambari server node, edit the file /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py Search for hive_apps_whs_dir which should go to this block: params.HdfsResource(params.hive_apps_whs_dir, type="directory", action="create_on_execute", owner=params.hive_user, group=params.user_group, mode=0755 ) Modify the value for mode from 0755 to the desired permission, for example, 0777. Save and close the file. Restart the Ambari server to propagate the change to all nodes in the cluster: ambari-server restart It may take a few seconds to update the file in the Ambari agents on all nodes. To verify if the change has been applied on a particular node, check the content of hive.py in /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py From the Ambari UI, restart HiveServer2 to apply the new permission to the warehouse directory. If multiple HiveServer2 instances are configured, any one instance can be restarted. Hope this helps you. ... View more

hdfs dfsadmin -setBalancerBandwidth 100000000 on all the DN and the client we ran the command below hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5 This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used. Hope this helps you. ... View more