Hi, Unable to start the Spark Livy Server 2 ExecutionFailed: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://random_host.net:8999/sessions | grep 200 ' returned 1. curl: option --negotiate: the installed libcurl version doesn't support this curl: try 'curl --help' or 'curl --manual' for more information ) I have tried to install libcurl . but it has latest package # yum install libcurl Loaded plugins: fastestmirror, langpacks Package matching libcurl-7.29.0-35.el7.centos.x86_64 already installed. Checking for update. Nothing to do Ambari Says it is started and running .But the error wont go. Refer image attached. The port is opened and running in the background: netstat -plant | grep 8999tcp6 0 0 :::8999 :::* LISTEN 4677/java I have Spark2 livy server in two machines. 1.Machine 1 Spark livy server 2 is running. 2.Machine 2 Spark Livy server 2 is running as per Ambari and terminal . However Ambari says error. I have verified both the machines logs , it shows both error are similar (ie Hadoop folder missing , java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.) n(java.io.FileNotFoundException and File does not exist: /livy2-recovery/v1/batch/st ) the error im getting from 2nd machine not from the 1st machine. I have disabled the alert and enabled the alert . still it doesn't work.I have ambari agent restarted twice . . Still it doesn't work. sp.png Attached log : sp.txt ... View more

How can I add two instances in hive server 2 . It's just adding the hive server through ambari. Do we need change any manual configuration on this . I cannot the see the tables from node 1 to node2. How can I achieve that ... View more

Hi, We are planning to do Hive server 2(HS2) High Availability through Ambari and Add multiple (two) Hiveserver 2 instances . Let's assume node 1 and node 2 . Node 1 is having the hive metastore and hiveserver 2 ( 1st instance) . I have enabled the second instance of Hiveserver 2 using the below link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_hadoop-high-availability/content/additional_hs2_installed_with_ambari.html In Hiveserver 2 Node 1: I have few tables lets say , Table1 and table 2 . after enabled the hiveserver 2 in node 2 . I can't find the hiveserver 2 folder in the node2 machine to make the configuration edits in hive-site.xml and as well as I can't view the tables from Node 1 in node 2 Hive server 2 machine . Do I need to configure anything apart from adding HS2 in node 2.Have restarted the services . 1) Do i need to configure Hive remote metastore to view the tables in node 1 . If so ,Kindly provide me a link please. 2) How can I achieve HS2 HA and Hive metastore HA . Please guide me on this . I cannot find a valid tutorials out there in Internet . thanks ... View more

Hi All, Can anyone provide the step by step guide to do the Hiveserver2 LDAP Integration with SSL . The official Documentation is not working properly . Please share any valuable links if there is any . Thanks. ... View more

I'm trying to copy the files from one cluster(prod) to another cluster(dev). The files which i'm trying to copy is bucketed and partitioned files from Hive tables in orc format. I'm getting error below. Source machine is Not Namenode HA and Destination machine is NN HA enabled. Error: java.io.IOException: File copy failed: hdfs://cluster:8020/user/backup/machineID=XEUS/delta_21551841_21551940/bucket_00003 --> hdfs://CLUSTTDEV:8020/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://cluster:8020/user/backup/machineID=XEUS/delta_21551841_21551940/bucket_00003 --> hdfs://CLUSTTDEV:8020/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-267577882-40.133.26.59-1515787116650:blk_1076168453_2430591 file=/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:290) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:250) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-267577882-40.133.26.59-1515787116650:blk_1076168453_2430591 file=/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:995) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:638) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:888) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:945) at java.io.DataInputStream.read(DataInputStream.java:100) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:77) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:285) ... 16 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. 18/05/10 02:25:30 INFO mapreduce.Job: map 30% reduce 0% 18/05/10 02:25:32 INFO mapreduce.Job: map 32% reduce 0% 18/05/10 02:25:39 INFO mapreduce.Job: map 33% reduce 0% hadoop distcp -pbugc hdfs://cluster:8020/user/backup/ hdfs://CLUSTTDEV:8020/user/backupdev/ hadoop distcp -skipcrccheck -update hdfs://cluster:8020/user/backup/ hdfs://CLUSTTDEV:8020/user/backupdev/ ... View more

I have created a fresh cluster , Hive is not allowing me to enter using the root user( Linux ) and even tried with HDFS user . However I'm able to login as user hive and able to run queries. How can I provide access to any user in Hive.For example,I have a local linux users it_dev it_admin. 1. How can I grant permissions to use hive from their login . They are Ldap users by default. 2.How can I grant permissions to specific databases and restrict them with ( create /delete/update). I'm sure this is very basic question . I haven't find any good document in Internet . Could anyone here please describe for everyone in the forum . I can understand this can be managed by Ranger . However we do not want Ranger in our system now . Please assist me ... View more

Problem Installing a Fresh Install of Ambari after removing the earlier version . We had Ambari 2.5.1 earlier then we completely removed 2.5.1 and started to install Ambari 2.5.2 We face few issues when we try to install Ambari : Attached logs.txt Snippet from the Logs : stderr: /var/lib/ambari-agent/data/errors-14.txt resource_management.core.exceptions.ExecutionFailed: Execution of 'useradd -m -u 1015 -G hadoop -g hadoop yarn' returned 12. useradd: cannot create directory /home/yarn Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-14.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-14.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-14.txt Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-14.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-14.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']Command failed after 1 tries As described in this page : https://community.hortonworks.com/questions/103915/problem-in-installing-ambarihdp-in-cluster-without.html I am able to execute the useradd -m -u 1015 -G hadoop -g hadoop yarn command . However I was unable to create a directory under /home/yarn Since it is mounted as NFS . I have added the details in the cat /etc/auto.home * -fstype=nfs homes:/global/export/home/& hdfs :/usr/local/home/hdfs yarn :/usr/local/home/yarn mapred :/usr/local/home/mapred activity_analyzer :/usr/local/home/activity_analyzer After updating the details to /etc/auto.home I have removed Ambari Server and Agent using the link : https://community.hortonworks.com/questions/1110/how-to-completely-remove-uninstall-ambari-and-hdp.html Have tried to reinstall Ambari . Again I have got the same error . df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/os-root 10G 2.4G 7.7G 24% / devtmpfs 910M 0 910M 0% /dev tmpfs 920M 12K 920M 1% /dev/shm tmpfs 920M 120M 801M 13% /run tmpfs 920M 0 920M 0% /sys/fs/cgroup /dev/sda1 488M 127M 326M 29% /boot /dev/mapper/os-var 4.0G 2.6G 1.5G 64% /var /dev/mapper/os-tmp 2.0G 33M 2.0G 2% /tmp /dev/mapper/apps-app 20G 45M 19G 1% /app tmpfs 184M 0 184M 0% /run/user/48258 homes:/global/export/home/user 2.9T 2.5T 337G 89% /home/user Still receving the same error . What can be done to resolve the issue . Is there any configuration changes has to be done to resolve the issue ... View more

I'm Unable to add HDF Package ( Nifi ) 3.0.3 to existing Cluster . Versions which I'm using HDP version 2.6.3 and Ambari 2.6.0. I'm using Cent OS 7.2. Steps I'm following to Install: -> Installed the M-Pack using the command ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-<version>.tar.gz --verbose -> ambari-server restart -> Added Base URL for HDF 3.0.3(Local and Public repo ) and Added Base URL for HDF 3.0.2 . Both ways It failed .. Followed the URL to install https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.3/bk_installing-hdf-and-hdp-ppc/content/ch_install-mpack.html And I see this Warning You cannot install HDF 3.0.3 on systems other than IBM Power Systems. For information about the latest release on systems other than IBM Power Systems, see the HDF 3.0.2 Release Notes. This is my local Repo file hdf.repo ( adjusted the public URL when i'm using the URL) #VERSION_NUMBER=3.0.3.0-6 [HDF-3.0.3.0] name=HDF Version - HDF-3.0.3.0 baseurl=http://192.168.1.8/repo/HDF/ gpgcheck=1 gpgkey=http://public-repo-1.hortonworks.com/HDF/centos7-ppc/3.x/updates/3.0.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 error-1.txt ... View more

Hi All, I'm unable to register blueprints to install Ambari multinode cluster . I'm getting the below error , I have used the below blueprints : https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-blueprints-1.html and getting the below error : }[root@master ~]# curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://master.hadoop.com:8080/api/v1/clusters/multinode1 -d @hostmapping.json { "status" : 400, "message" : "Topology validation failed: org.apache.ambari.server.topology.InvalidTopologyException: Component [MYSQL_SERVER] must explicitly be set in the blueprint when hive database is configured with the current settings. HIVE service validation failed." Kindly advise . Steps taken - Installed Mysql server and tried . Still could not install . Added mysql entry in JSON file . Still didnt work out ... View more

Unable to start Ambari Agent. I'm getting heartbeat lost for all the services on the server. Since it is Primary namenode. Couldn't identify the status of the services on the server.When I fire ambari-agent start/restart . It started and stopped suddenly .However when I grep ambari in running process but it is actually not running. How can I start ambari agent .. root 2970771 1 0 Nov08 ? 00:00:00 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start root 2970779 2970771 0 Nov08 ? 00:21:24 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/main.py start Symptoms: Using version Python 2.6 Logs didn't say anything other than this actually stopped logging . ValueError: Unknown format code 'd' for object of type 'float' INFO 2017-11-10 15:45:48,904 DataCleaner.py:120 - Data cleanup started INFO 2017-11-10 15:45:48,908 DataCleaner.py:122 - Data cleanup finished WARNING 2017-11-10 15:46:42,230 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package /alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts WARNING 2017-11-10 15:47:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package /alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks o r /var/lib/ambari-agent/cache/host_scripts ERROR 2017-11-10 15:47:42,428 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-10 15:49:42 .210824)" raised an exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job retval = job.func(*job.args, **job.kwargs) File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda> return lambda: alert_def.collect() File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect data['text'] = res_base_text.format(*res[1]) ValueError: Unknown format code 'd' for object of type 'float' File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job retval = job.func(*job.args, **job.kwargs) File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda> return lambda: alert_def.collect() File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect data['text'] = res_base_text.format(*res[1]) ValueError: Unknown format code 'd' for object of type 'float' WARNING 2017-11-11 11:52:42,221 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts WARNING 2017-11-11 11:53:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts ERROR 2017-11-11 11:53:42,416 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-11 11:55:42.210824)" raised an exception Traceback (most recent call last): @Jay Kumar SenSharma . Please any idea on this . ... View more

Unable to start Ambari Agent. I'm getting heartbeat lost for all the services on the server. Since it is Primary namenode. Couldn't identify the status of the services on the server.When I fire ambari-agent start/restart . It started and stopped suddenly .However when I grep ambari in running process but it is actually not running. How can I start ambari agent .. root 2970771 1 0 Nov08 ? 00:00:00 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start root 2970779 2970771 0 Nov08 ? 00:21:24 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/main.py start Symptoms: Using version Python 2.6 Logs didn't say anything other than this actually stopped logging . ValueError: Unknown format code 'd' for object of type 'float' INFO 2017-11-10 15:45:48,904 DataCleaner.py:120 - Data cleanup started INFO 2017-11-10 15:45:48,908 DataCleaner.py:122 - Data cleanup finished WARNING 2017-11-10 15:46:42,230 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package /alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts WARNING 2017-11-10 15:47:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package /alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks o r /var/lib/ambari-agent/cache/host_scripts ERROR 2017-11-10 15:47:42,428 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-10 15:49:42 .210824)" raised an exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job retval = job.func(*job.args, **job.kwargs) File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda> return lambda: alert_def.collect() File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect data['text'] = res_base_text.format(*res[1]) ValueError: Unknown format code 'd' for object of type 'float' File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job retval = job.func(*job.args, **job.kwargs) File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda> return lambda: alert_def.collect() File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect data['text'] = res_base_text.format(*res[1]) ValueError: Unknown format code 'd' for object of type 'float' WARNING 2017-11-11 11:52:42,221 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts WARNING 2017-11-11 11:53:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts ERROR 2017-11-11 11:53:42,416 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-11 11:55:42.210824)" raised an exception Traceback (most recent call last): @Jay Kumar SenSharma . Please any idea on this . ... View more

Hello there, In Resource manager URL, I can see 4 lost nodes and in Ambari . All the data nodes are live(up and running). Logged into the lost data node and I can see the node manager are up and running. In Namenode UI also I can see all the data nodes are live. However why Resource manager showing the false alert. How can I rectify the false alert in Resouce manager URL? ... View more