Support Questions
Find answers, ask questions, and share your expertise

YARN Apptimeline server not starting,YARN App Timeline server not starting

YARN Apptimeline server not starting,YARN App Timeline server not starting

New Contributor

I installed HDP 2.5.0 on Centos 7. I am having issues in starting app timeline server. On Web UI, it shows started but after refresh its shows in stopped state. I checked the yarn-yarn-timelineserver-localhost.localdomain.log file. Its failing on starting application history server.

Below log for your reference.

2016-11-03 13:13:24,408 INFO  impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(299)) - Timeline service address: http://localhost.localdomain:8188/ws/v1/timeline/
2016-11-03 13:13:28,188 INFO  timeline.RollingLevelDBTimelineStore (RollingLevelDBTimelineStore.java:serviceInit(318)) - Using leveldb path /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store
2016-11-03 13:13:28,299 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
2016-11-03 13:13:28,313 WARN  service.AbstractService (ServiceOperations.java:stopQuietly(82)) - When stopping the service org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore : java.lang.NullPointerException
java.lang.NullPointerException
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceStop(RollingLevelDBTimelineStore.java:363)
	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
2016-11-03 13:13:28,331 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service EntityGroupFSTimelineStore failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	... 8 more

Could someone please check and guide me to resolve this issue.

I tried to run the sqoop import statement and it is failing with below error, i think the below error is related to App Timeline server not running issue.

16/11/03 10:35:42 INFO mapreduce.Job: Job job_1478193879443_0002 failed with state FAILED due to: Application application_1478193879443_0002 failed 2 times due to AM Container for appattempt_1478193879443_0002_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1478193879443_0002 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: hadoop/lib/native/libhdfs.so: Cannot utime: Operation not permitted tar: hadoop/lib/native/libsnappy.so.1: Cannot utime: Operation not permitted tar: hadoop/lib/native/libsnappy.so: Cannot utime: Operation not permitted tar: hadoop/lib/native/libhadoop.so: Cannot utime: Operation not permitted tar: Exiting with failure status due to previous errors Failing this attempt. Failing the application. 16/11/03 10:35:42 INFO mapreduce.Job: Counters: 0 16/11/03 10:35:42 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 16/11/03 10:35:42 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 407.92 seconds (0 bytes/sec) 16/11/03 10:35:42 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 16/11/03 10:35:42 INFO mapreduce.ImportJobBase: Retrieved 0 records. 16/11/03 10:35:42 ERROR tool.ImportTool: Error during import: Import job failed!

,

I installed HDP 2.5.0 on Centos 7. I am having issues in starting app timeline server. On Web UI, it shows started but after refresh its shows in stopped state. I checked the yarn-yarn-timelineserver-localhost.localdomain.log file. Its failing on starting application history server.

Below log for your reference.

2016-11-03 13:13:24,408 INFO  impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(299)) - Timeline service address: http://localhost.localdomain:8188/ws/v1/timeline/
2016-11-03 13:13:28,188 INFO  timeline.RollingLevelDBTimelineStore (RollingLevelDBTimelineStore.java:serviceInit(318)) - Using leveldb path /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store
2016-11-03 13:13:28,299 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
2016-11-03 13:13:28,313 WARN  service.AbstractService (ServiceOperations.java:stopQuietly(82)) - When stopping the service org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore : java.lang.NullPointerException
java.lang.NullPointerException
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceStop(RollingLevelDBTimelineStore.java:363)
	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
2016-11-03 13:13:28,331 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service EntityGroupFSTimelineStore failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:212)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
	at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument
	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
	at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:319)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	... 8 more

Could someone please check and guide me to resolve this issue.

I tried to run the sqoop import statement and it is failing with below error, i think the below error is related to App Timeline server not running issue.

16/11/03 10:35:42 INFO mapreduce.Job: Job job_1478193879443_0002 failed with state FAILED due to: Application application_1478193879443_0002 failed 2 times due to AM Container for appattempt_1478193879443_0002_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1478193879443_0002 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: hadoop/lib/native/libhdfs.so: Cannot utime: Operation not permitted tar: hadoop/lib/native/libsnappy.so.1: Cannot utime: Operation not permitted tar: hadoop/lib/native/libsnappy.so: Cannot utime: Operation not permitted tar: hadoop/lib/native/libhadoop.so: Cannot utime: Operation not permitted tar: Exiting with failure status due to previous errors Failing this attempt. Failing the application. 16/11/03 10:35:42 INFO mapreduce.Job: Counters: 0 16/11/03 10:35:42 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 16/11/03 10:35:42 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 407.92 seconds (0 bytes/sec) 16/11/03 10:35:42 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 16/11/03 10:35:42 INFO mapreduce.ImportJobBase: Retrieved 0 records. 16/11/03 10:35:42 ERROR tool.ImportTool: Error during import: Import job failed!

4 REPLIES 4

Re: YARN Apptimeline server not starting,YARN App Timeline server not starting

Mentor

can you check your directory permissions and paths? Directory

hadoop/lib/native/

Also make sure your compression libs are installed

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/install...

Re: YARN Apptimeline server not starting,YARN App Timeline server not starting

New Contributor

Thanks Artem Ervits for the response!

I checked the permissions on the /lib/native/ folder and it seems to be correct. User hdfs has full permissions on this folder. Also, checked the libraries and these are also installed.

Still sqoop import command is falling with same error.

App timeline server showing same error.

Re: YARN Apptimeline server not starting,YARN App Timeline server not starting

Mentor

@Arvind Kumar please review the properties for Application Timeline server, it is possible that you have ATS misconfigured. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_yarn-resource-management/content/ch_time...

look at the next page of the document

Changed properties:

Property: yarn.timeline-service.store-class
	Old value: org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore
	New value: org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore

Property: yarn.timeline-service.leveldb-timeline-store.path
	Old value: ${yarn.log.dir}/timeline
	New value: <Data disk>/ats/leveldb/



Re: YARN Apptimeline server not starting,YARN App Timeline server not starting

New Contributor

@Artem Ervits I checked the configurations but did not see any issues. Also, noticed the mentioned properties, below for your reference.

<property> <name>yarn.timeline-service.store-class</name> <value>org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore</value> </property>

<property> <name>yarn.timeline-service.leveldb-timeline-store.path</name> <value>/media/psf/Home/hadoop/yarn/timeline</value> </property>

I also tried changing the leveldb-timeline-store path, but got same error.

Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /media/psf/Home/hadoop/yarn/timeline/ats/leveldb/leveldb-timeline-store/domain-ldb/MANIFEST-000001: Invalid argument at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at

org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:319) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 8 more 16/11/22 10:18:14 INFO util.ExitUtil: Exiting with status -1 16/11/22 10:18:14 INFO timeline.HadoopTimelineMetricsSink: Closing HadoopTimelineMetricSink. Flushing metrics to collector... 16/11/22 10:18:14 INFO applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at localhost.localdomain/127.0.0.1 ************************************************************/