Member since
10-20-2017
63
Posts
0
Kudos Received
0
Solutions
06-25-2018
01:44 PM
Hi, Unable to start the Spark Livy Server 2 ExecutionFailed: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://random_host.net:8999/sessions | grep 200 ' returned 1. curl: option --negotiate: the installed libcurl version doesn't support this
curl: try 'curl --help' or 'curl --manual' for more information
) I have tried to install libcurl . but it has latest package # yum install libcurl
Loaded plugins: fastestmirror, langpacks
Package matching libcurl-7.29.0-35.el7.centos.x86_64 already installed. Checking for update.
Nothing to do Ambari Says it is started and running .But the error wont go. Refer image attached. The port is opened and running in the background: netstat -plant | grep 8999tcp6
0 0 :::8999 :::* LISTEN 4677/java I have Spark2 livy server in two machines. 1.Machine 1 Spark livy server 2 is running. 2.Machine 2 Spark Livy server 2 is running as per Ambari and terminal . However Ambari says error. I have verified both the machines logs , it shows both error are similar (ie Hadoop folder missing , java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.) n(java.io.FileNotFoundException and File does not exist: /livy2-recovery/v1/batch/st ) the error im getting from 2nd machine not from the 1st machine. I have disabled the alert and enabled the alert . still it doesn't work.I have ambari agent restarted twice . . Still it doesn't work. sp.png Attached log : sp.txt
... View more
Labels:
06-10-2018
04:09 AM
Thanks Sandeep . What about adding two instances through ambari . In Hiveserver 2 Node 1: I have few tables lets say , Table1 and table 2 . after enabled the hiveserver 2 in node 2 . I can't find the hiveserver 2 folder in the node2 machine to make the configuration edits in hive-site.xml and as well as I can't view the tables from Node 1 in node 2 Hive server 2 machine . Do I need to configure anything apart from adding HS2 in node 2.Have restarted the services . 1) Do i need to configure Hive remote metastore to view the tables in node 1 . If so ,Kindly provide me a link please.
... View more
06-09-2018
08:26 AM
How can I add two instances in hive server 2 . It's just adding the hive server through ambari. Do we need change any manual configuration on this . I cannot the see the tables from node 1 to node2. How can I achieve that
... View more
Labels:
06-09-2018
03:46 AM
Hi, We are planning to do Hive server 2(HS2) High Availability through Ambari and Add multiple (two) Hiveserver 2 instances . Let's assume node 1 and node 2 . Node 1 is having the hive metastore and hiveserver 2 ( 1st instance) . I have enabled the second instance of Hiveserver 2 using the below link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_hadoop-high-availability/content/additional_hs2_installed_with_ambari.html In Hiveserver 2 Node 1: I have few tables lets say , Table1 and table 2 . after enabled the hiveserver 2 in node 2 . I can't find the hiveserver 2 folder in the node2 machine to make the configuration edits in hive-site.xml and as well as I can't view the tables from Node 1 in node 2 Hive server 2 machine . Do I need to configure anything apart from adding HS2 in node 2.Have restarted the services . 1) Do i need to configure Hive remote metastore to view the tables in node 1 . If so ,Kindly provide me a link please. 2) How can I achieve HS2 HA and Hive metastore HA . Please guide me on this . I cannot find a valid tutorials out there in Internet . thanks
... View more
Labels:
06-09-2018
03:24 AM
Fantastic and detailed reply. I would try this out and reply if that works .Thanks a lot @Shu
... View more
06-08-2018
06:58 PM
Hi @Shu I was able to establish a connection between mysql and Hive . However the purpose is not solved yet . I was trying to update and insert values into Hive . For Example ( refer attached) . In the example : . On Initial load Hive has loaded the values of ID 1,2 and 3 . On the second load , We have to remove the values 3 and update the value 1 and insert the value 4 . How that is possible in Nifi . Is that at possible ? Could please guide me on this If you can. thanks ... ex.jpg
... View more
06-07-2018
07:26 PM
Hi, I'm unable to establish connection between Mysql and Hive due to DBCP connection pool .This is my configuration settings in DBCP connection pool ( please refer attached) . I'm getting error says (refer attached ) dbcp1.jpg dbcp2.jpg dbcp.png Query database table -> put hive streaming processors is used . and Query Database table is scheduled to run every three minutes . I have restarted Nifi by disabling the DBCP connection pool and tried . Still the same and failed . Also Is it possible to insert updated values + New values in Mysql to Hive ( which processor should I use) @Matt Clarke @Matt Burgess . Could you please have a look here please . what am I doing wrong . Thanks All
... View more
Labels:
05-31-2018
06:47 PM
Hi All, Can anyone provide the step by step guide to do the Hiveserver2 LDAP Integration with SSL . The official Documentation is not working properly . Please share any valuable links if there is any . Thanks.
... View more
Labels:
05-10-2018
07:35 AM
I'm trying to copy the files from one cluster(prod) to another cluster(dev). The files which i'm trying to copy is bucketed and partitioned files from Hive tables in orc format. I'm getting error below. Source machine is Not Namenode HA and Destination machine is NN HA enabled. Error: java.io.IOException: File copy failed: hdfs://cluster:8020/user/backup/machineID=XEUS/delta_21551841_21551940/bucket_00003 --> hdfs://CLUSTTDEV:8020/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://cluster:8020/user/backup/machineID=XEUS/delta_21551841_21551940/bucket_00003 --> hdfs://CLUSTTDEV:8020/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-267577882-40.133.26.59-1515787116650:blk_1076168453_2430591 file=/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:290) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:250) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-267577882-40.133.26.59-1515787116650:blk_1076168453_2430591 file=/user/backupdev/machineID=XEUS/delta_21551841_21551940/bucket_00003 at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:995) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:638) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:888) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:945) at java.io.DataInputStream.read(DataInputStream.java:100) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:77) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:285) ... 16 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. 18/05/10 02:25:30 INFO mapreduce.Job: map 30% reduce 0% 18/05/10 02:25:32 INFO mapreduce.Job: map 32% reduce 0% 18/05/10 02:25:39 INFO mapreduce.Job: map 33% reduce 0% hadoop distcp -pbugc hdfs://cluster:8020/user/backup/ hdfs://CLUSTTDEV:8020/user/backupdev/ hadoop distcp -skipcrccheck -update hdfs://cluster:8020/user/backup/ hdfs://CLUSTTDEV:8020/user/backupdev/
... View more
Labels:
05-07-2018
06:15 AM
I have created a fresh cluster , Hive is not allowing me to enter using the root user( Linux ) and even tried with HDFS user . However I'm able to login as user hive and able to run queries. How can I provide access to any user in Hive.For example,I have a local linux users it_dev it_admin. 1. How can I grant permissions to use hive from their login . They are Ldap users by default. 2.How can I grant permissions to specific databases and restrict them with ( create /delete/update). I'm sure this is very basic question . I haven't find any good document in Internet . Could anyone here please describe for everyone in the forum . I can understand this can be managed by Ranger . However we do not want Ranger in our system now . Please assist me
... View more
Labels:
05-05-2018
08:45 PM
Apologies for the delay , I wasn't able to reply back in mobile . though I have drafted the reply . . Yes It worked !! . But we have already created the Symlink and We have even edited the python code with the absolute path in Webhcat.py file . Both the above solution works for me . However my concern is , we have had a problem when we changed the symlink in other service. So we dont want any trouble later . Would it be alright to create symlink alone as permanent workaround . Or any other permanent solution. Just curiosity , How did you predicted after seeing the error. Kindly teach us how did you analyzed the error quickly 🙂
... View more
05-05-2018
11:04 AM
We are unable to start the WebHcat server. We are the getting the below error : This the error we are getting : Fail: Applying File['/usr/hdp/2.6.3.0/hive/conf/hive-site.xml'] failed, parent directory /usr/hdp/2.6.3.0/hive/conf doesn't exist Please see the bug has been patched in version 2.6 and later . but we are using 2.5.2 . However , FYI , we have another machine with same version where we are able to start/stop the webhcat server without any issues .
... View more
Labels:
05-03-2018
06:41 PM
I have edited the file cat /etc/auto.home. Will it not help . We were able to fresh install into another machine without any strgulle . since we have removed and reinstalled fresh , we are getting this error . I have limited access to changing the name of mount point and mounted as NFS . I would see the possibility of getting approved . Any alternative good ways to tackle it as a best practise
... View more
05-03-2018
06:25 PM
Thanks for the reply , Do you want me to comment out the Lines for validation . However , I wasn't able to create any directory under home in centos7 . Will commenting alone will help ? Could u please help me
... View more
05-03-2018
05:55 PM
Problem Installing a Fresh Install of Ambari after removing the earlier version . We had Ambari 2.5.1 earlier then we completely removed 2.5.1 and started to install Ambari 2.5.2 We face few issues when we try to install Ambari : Attached logs.txt Snippet from the Logs : stderr: /var/lib/ambari-agent/data/errors-14.txt resource_management.core.exceptions.ExecutionFailed: Execution of 'useradd -m -u 1015 -G hadoop -g hadoop yarn' returned 12. useradd: cannot create directory /home/yarn Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-14.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-14.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-14.txt Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-14.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-14.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']Command failed after 1 tries As described in this page : https://community.hortonworks.com/questions/103915/problem-in-installing-ambarihdp-in-cluster-without.html I am able to execute the useradd -m -u 1015 -G hadoop -g hadoop yarn command . However I was unable to create a directory under /home/yarn Since it is mounted as NFS . I have added the details in the cat /etc/auto.home * -fstype=nfs homes:/global/export/home/& hdfs :/usr/local/home/hdfs yarn :/usr/local/home/yarn mapred :/usr/local/home/mapred activity_analyzer :/usr/local/home/activity_analyzer After updating the details to /etc/auto.home I have removed Ambari Server and Agent using the link : https://community.hortonworks.com/questions/1110/how-to-completely-remove-uninstall-ambari-and-hdp.html Have tried to reinstall Ambari . Again I have got the same error . df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/os-root 10G 2.4G 7.7G 24% / devtmpfs 910M 0 910M 0% /dev tmpfs 920M 12K 920M 1% /dev/shm tmpfs 920M 120M 801M 13% /run tmpfs 920M 0 920M 0% /sys/fs/cgroup /dev/sda1 488M 127M 326M 29% /boot /dev/mapper/os-var 4.0G 2.6G 1.5G 64% /var /dev/mapper/os-tmp 2.0G 33M 2.0G 2% /tmp /dev/mapper/apps-app 20G 45M 19G 1% /app tmpfs 184M 0 184M 0% /run/user/48258 homes:/global/export/home/user 2.9T 2.5T 337G 89% /home/user Still receving the same error . What can be done to resolve the issue . Is there any configuration changes has to be done to resolve the issue
... View more
04-24-2018
08:33 PM
I have created a Shellscript and It requires three arguments to be passed . It was working fine till I execute it in a terminal . When I configured using Nifi ExecuteProcess ,nifi is ignoring the arguments . Example :/tmp/script.sh 1 2 8 Refer the attached image . How can I inform nifi to look at the arguments for shell script. Or kindly advice me to use anyother processor with detailed steps . Since this is not my strongest point . Thanks!!
... View more
Labels:
04-21-2018
11:42 AM
tar.pngHi All, I have only these options , I do not have good internet connection . However, I have created a repo . but I can see only tar files in the hdf repo tar ball. http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0 https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.2.0/HDF-3.0.2.0-centos7-tars-tarball.tar.gz If you download and add to the Ambari Manage -> versions-> add the local repo URL (http://192.168.1.8/repo/HDF/) and get this error Some of the repositories failed validation. Make changes to the base url or skip validation if you are sure that urls are correct HOwever When I look at the HDF repo directory it has only tar files of services . I do not think it is a valid repo .
... View more
04-20-2018
08:01 PM
When I installed , MPACK for 3.0.2 . I get this error below : sudo ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-3.0.2.0-76.tar.gz --verbose
Using python /usr/bin/python
Installing management pack
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Installing management pack /tmp/hdf-ambari-mpack-3.0.2.0-76.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/hdf-ambari-mpack-3.0.2.0-76.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 950, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 920, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 872, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 78, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack
(mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 697, in _install_mpack
tmp_root_dir = expand_mpack(tmp_archive_path)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 150, in expand_mpack
archive_root_dir = get_archive_root_dir(archive_path)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/tar_archive.py", line 78, in get_archive_root_dir
names = tar.getnames()
File "/usr/lib64/python2.7/tarfile.py", line 1813, in getnames
return [tarinfo.name for tarinfo in self.getmembers()]
File "/usr/lib64/python2.7/tarfile.py", line 1805, in getmembers
self._load() # all members, we first have to
File "/usr/lib64/python2.7/tarfile.py", line 2380, in _load
tarinfo = self.next()
File "/usr/lib64/python2.7/tarfile.py", line 2315, in next
self.fileobj.seek(self.offset)
File "/usr/lib64/python2.7/gzip.py", line 434, in seek
self.read(1024)
File "/usr/lib64/python2.7/gzip.py", line 261, in read
self._read(readsize)
File "/usr/lib64/python2.7/gzip.py", line 308, in _read
self._read_eof()
File "/usr/lib64/python2.7/gzip.py", line 347, in _read_eof
hex(self.crc)))
IOError: CRC check failed 0xdc6e2773 != 0xf9fc1ddL
... View more
04-20-2018
07:53 PM
How do I start with fresh install now when I have already installed mpack for 3.0.3 and 3.0.3 HDF . Do you have any command to remove all the files which I have installed.
... View more
04-20-2018
07:51 PM
thanks Felix , However ,I have tried installing the 3.0.2 Same error . please find the attached error for your reference when I installed with 3.0.2error-302.txt .
... View more
04-20-2018
07:48 PM
Thanks for the Quick response , I have tried using 3.0.2 tarball and repo . Still facing the same issue . However I have installed mpack for 3.0.3 . Dont know how to remove this Mpack .
... View more
04-20-2018
07:41 PM
I'm Unable to add HDF Package ( Nifi ) 3.0.3 to existing Cluster . Versions which I'm using HDP version 2.6.3 and Ambari 2.6.0.
I'm using Cent OS 7.2.
Steps I'm following to Install:
-> Installed the M-Pack using the command ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-<version>.tar.gz --verbose
-> ambari-server restart
-> Added Base URL for HDF 3.0.3(Local and Public repo ) and Added Base URL for HDF 3.0.2 . Both ways It failed .. Followed the URL to install https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.3/bk_installing-hdf-and-hdp-ppc/content/ch_install-mpack.html
And I see this Warning
You cannot install HDF 3.0.3 on systems other than IBM Power Systems. For information about the latest release on systems other than IBM Power Systems, see the HDF 3.0.2 Release Notes.
This is my local Repo file hdf.repo ( adjusted the public URL when i'm using the URL) #VERSION_NUMBER=3.0.3.0-6
[HDF-3.0.3.0]
name=HDF Version - HDF-3.0.3.0
baseurl=http://192.168.1.8/repo/HDF/
gpgcheck=1
gpgkey=http://public-repo-1.hortonworks.com/HDF/centos7-ppc/3.x/updates/3.0.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1 error-1.txt
... View more
Labels:
01-27-2018
09:39 PM
Hi All, I'm unable to register blueprints to install Ambari multinode cluster . I'm getting the below error , I have used the below blueprints : https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-blueprints-1.html and getting the below error : }[root@master ~]# curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://master.hadoop.com:8080/api/v1/clusters/multinode1 -d @hostmapping.json
{
"status" : 400,
"message" : "Topology validation failed: org.apache.ambari.server.topology.InvalidTopologyException: Component [MYSQL_SERVER] must explicitly be set in the blueprint when hive database is configured with the current settings. HIVE service validation failed." Kindly advise . Steps taken - Installed Mysql server and tried . Still could not install . Added mysql entry in JSON file . Still didnt work out
... View more
Labels:
11-15-2017
12:09 AM
can understand , fuser is taking long time to respond to ambari agent. What is the fix other than restart a server.any idea
... View more
11-13-2017
04:51 PM
@Jay Kumar SenSharma, Sir . Do we need any more info to proceed on this . can you please help me.
... View more
11-11-2017
10:06 PM
Yes . The changes made to the Amabri Metrics are because we were unable to start the Ambari metrics few days back. Hence we changed the port numbers as below. timeline.metrics.service.webapp.address--> 0.0.0.0:7188 and hbase.zookeeper.property.clientPort --> 2181 from 61181 . It is distributed environment. Ambari Version 2.1.0 Hostname changed <affected host> 11 Nov 2017 04:18:30,378 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host <affected host>
11 Nov 2017 04:18:30,379 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component METRICS_MONITOR on <affected host>
11 Nov 2017 04:18:30,379 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component FLUME_HANDLER on <affected host>
11 Nov 2017 04:18:30,379 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HBASE_REST_SERVER on <affected host>
11 Nov 2017 04:18:30,379 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component HBASE_MASTER on <affected host>
11 Nov 2017 04:18:30,379 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component ZKFC on <affected host>
11 Nov 2017 04:18:30,380 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component NAMENODE on <affected host>
11 Nov 2017 04:18:30,380 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:169 - Setting component state to UNKNOWN for component ZOOKEEPER_SERVER on <affected host>
11 Nov 2017 04:18:53,910 INFO [AlertNoticeDispatchService] AlertNoticeDispatchService:279 - There are 5 pending alert notices about to be dispatched...
11 Nov 2017 04:18:54,107 INFO [alert-dispatch-32] EmailDispatcher:88 - Sending email: XXXXXXXXXXXXXXXXXXXXXXX
11 Nov 2017 12:40:53,970 ERROR [qtp-client-6767] MetricsPropertyProvider:185 - Error getting timeline metrics. Can not connect to collector, socket error.
11 Nov 2017 12:41:03,981 ERROR [qtp-client-6767] MetricsPropertyProvider:185 - Error getting timeline metrics. Can not connect to collector, socket error.
ERROR [qtp-client-3412] MetricsReportPropertyProvider:223 - Error getting timeline metrics. Can not connect to collector, socket error.
The alert_ambari_metrics_monitor.py is same path for both working ambari-agent host and non ambari-agent host are same .
... View more
11-11-2017
04:58 PM
Unable to start Ambari Agent. I'm getting heartbeat lost for all the services on the server. Since it is Primary namenode. Couldn't identify the status of the services on the server.When I fire ambari-agent start/restart . It started and stopped suddenly .However when I grep ambari in running process but it is actually not running. How can I start ambari agent .. root 2970771 1 0 Nov08 ? 00:00:00 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start root 2970779 2970771 0 Nov08 ? 00:21:24 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/main.py start Symptoms: Using version Python 2.6 Logs didn't say anything other than this actually stopped logging . ValueError: Unknown format code 'd' for object of type 'float'
INFO 2017-11-10 15:45:48,904 DataCleaner.py:120 - Data cleanup started
INFO 2017-11-10 15:45:48,908 DataCleaner.py:122 - Data cleanup finished
WARNING 2017-11-10 15:46:42,230 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package
/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
WARNING 2017-11-10 15:47:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package
/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks o
r /var/lib/ambari-agent/cache/host_scripts
ERROR 2017-11-10 15:47:42,428 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-10 15:49:42
.210824)" raised an exception
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda>
return lambda: alert_def.collect()
File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect
data['text'] = res_base_text.format(*res[1])
ValueError: Unknown format code 'd' for object of type 'float'
File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda>
return lambda: alert_def.collect()
File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect
data['text'] = res_base_text.format(*res[1])
ValueError: Unknown format code 'd' for object of type 'float'
WARNING 2017-11-11 11:52:42,221 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
WARNING 2017-11-11 11:53:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
ERROR 2017-11-11 11:53:42,416 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-11 11:55:42.210824)" raised an exception
Traceback (most recent call last):
@Jay Kumar SenSharma . Please any idea on this .
... View more
Labels:
11-11-2017
12:46 PM
Unable to start Ambari Agent. I'm getting heartbeat lost for all the services on the server. Since it is Primary namenode. Couldn't identify the status of the services on the server.When I fire ambari-agent start/restart . It started and stopped suddenly .However when I grep ambari in running process but it is actually not running. How can I start ambari agent .. root 2970771 1 0 Nov08 ? 00:00:00 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start root 2970779 2970771 0 Nov08 ? 00:21:24 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/ambari_agent/main.py start Symptoms: Using version Python 2.6 Logs didn't say anything other than this actually stopped logging . ValueError: Unknown format code 'd' for object of type 'float'
INFO 2017-11-10 15:45:48,904 DataCleaner.py:120 - Data cleanup started
INFO 2017-11-10 15:45:48,908 DataCleaner.py:122 - Data cleanup finished
WARNING 2017-11-10 15:46:42,230 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package
/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
WARNING 2017-11-10 15:47:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package
/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks o
r /var/lib/ambari-agent/cache/host_scripts
ERROR 2017-11-10 15:47:42,428 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-10 15:49:42
.210824)" raised an exception
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda>
return lambda: alert_def.collect()
File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect
data['text'] = res_base_text.format(*res[1])
ValueError: Unknown format code 'd' for object of type 'float'
File "/usr/lib/python2.6/site-packages/ambari_agent/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 114, in <lambda>
return lambda: alert_def.collect()
File "/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py", line 153, in collect
data['text'] = res_base_text.format(*res[1])
ValueError: Unknown format code 'd' for object of type 'float'
WARNING 2017-11-11 11:52:42,221 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
WARNING 2017-11-11 11:53:42,220 base_alert.py:140 - [Alert][ams_metrics_monitor_process] Unable to execute alert. Unable to find 'AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
ERROR 2017-11-11 11:53:42,416 scheduler.py:520 - Job "452de60e-d34c-41d8-9748-bcff4784ebe2 (trigger: interval[0:02:00], next run at: 2017-11-11 11:55:42.210824)" raised an exception
Traceback (most recent call last):
@Jay Kumar SenSharma . Please any idea on this .
... View more
Labels:
11-01-2017
05:08 PM
Hello there, In Resource manager URL, I can see 4 lost nodes and in Ambari . All the data nodes are live(up and running). Logged into the lost data node and I can see the node manager are up and running. In Namenode UI also I can see all the data nodes are live. However why Resource manager showing the false alert. How can I rectify the false alert in Resouce manager URL?
... View more
- « Previous
-
- 1
- 2
- Next »