Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

index job fails with ClassNotFoundException

index job fails with ClassNotFoundException

Guru

Tried submitted a task to indexer. Submit worked but the task failed with ClassNotFoundException. We use ComponentGroupsMapping for getting groups for users (where we use unixgroupmapping and ADGroupMapping)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.CompositeGroupsMapping not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) ~[?:?]
	at org.apache.hadoop.security.Groups.<init>(Groups.java:99) ~[?:?]
	at org.apache.hadoop.security.Groups.<init>(Groups.java:95) ~[?:?]
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352) ~[?:?]
	at io.druid.security.kerberos.DruidKerberosUtil.authenticateIfRequired(DruidKerberosUtil.java:101) ~[?:?]
	at io.druid.security.kerberos.KerberosHttpClient.inner_go(KerberosHttpClient.java:97) ~[?:?]
	at io.druid.security.kerberos.KerberosHttpClient.go(KerberosHttpClient.java:67) ~[?:?]
	at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) ~[http-client-1.0.4.jar:?]
	at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:101) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]
	at io.druid.indexing.common.task.HadoopIndexTask.isReady(HadoopIndexTask.java:137) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]
	at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:168) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]
5 REPLIES 5

Re: index job fails with ClassNotFoundException

Expert Contributor

@Ravi Mutyala can you please add the task json spec and the full task logs from the overlord console? Thanks

Highlighted

Re: index job fails with ClassNotFoundException

Guru

Re: index job fails with ClassNotFoundException

New Contributor

can you try setting hadoop.mapreduce.job.classloader=true in your mapreduce configs ?

Re: index job fails with ClassNotFoundException

Guru

I see -Dhadoop.mapreduce.job.classloader=true in druid.indexer.runner.javaOpts. I also added in mapreduce-site.xml but same issue.

Below is what I can see from overlord log.

2017-10-18T15:22:40,938 INFO [TaskQueue-Manager] io.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z] version[2017-10-18T15:22:40.938Z] for task: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 INFO [TaskQueue-Manager] io.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 WARN [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - No worker selections strategy set. Using default. 2017-10-18T15:22:40,941 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[HOST:8091] to add task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] 2017-10-18T15:22:40,944 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z switched from pending to running (on [HOST:8091]) 2017-10-18T15:22:40,944 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] status changed to [RUNNING]. 2017-10-18T15:22:40,951 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote RUNNING status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='null', port=-1}] 2017-10-18T15:22:40,965 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote RUNNING status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='HOST', port=8100}] 2017-10-18T15:22:40,965 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] location changed to [TaskLocation{host='HOST', port=8100}]. 2017-10-18T15:22:46,906 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote FAILED status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='HOST', port=8100}] 2017-10-18T15:22:46,906 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] completed task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] with status[FAILED] 2017-10-18T15:22:46,907 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:46,907 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on worker[HOST:8091] 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] from activeTasks 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] from TaskLock[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, dataSource=wikiticker, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, version=2017-10-18T15:22:40.938Z} 2017-10-18T15:22:46,911 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[5]: TaskLock{groupId=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, dataSource=wikiticker, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, version=2017-10-18T15:22:40.938Z} 2017-10-18T15:22:46,914 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z to status: TaskStatus{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, status=FAILED, duration=5951} 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Task done: HadoopIndexTask{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, type=index_hadoop, dataSource=wikiticker} 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Task FAILED: HadoopIndexTask{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, type=index_hadoop, dataSource=wikiticker} (5951 run duration) 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] status changed to [FAILED]. 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] went bye bye. 2017-10-18T15:23:27,138 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

Re: index job fails with ClassNotFoundException

New Contributor