Support Questions

ravi1 · ‎10-16-2017

Tried submitted a task to indexer. Submit worked but the task failed with ClassNotFoundException. We use ComponentGroupsMapping for getting groups for users (where we use unixgroupmapping and ADGroupMapping)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.CompositeGroupsMapping not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) ~[?:?]
	at org.apache.hadoop.security.Groups.<init>(Groups.java:99) ~[?:?]
	at org.apache.hadoop.security.Groups.<init>(Groups.java:95) ~[?:?]
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352) ~[?:?]
	at io.druid.security.kerberos.DruidKerberosUtil.authenticateIfRequired(DruidKerberosUtil.java:101) ~[?:?]
	at io.druid.security.kerberos.KerberosHttpClient.inner_go(KerberosHttpClient.java:97) ~[?:?]
	at io.druid.security.kerberos.KerberosHttpClient.go(KerberosHttpClient.java:67) ~[?:?]
	at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) ~[http-client-1.0.4.jar:?]
	at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:101) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]
	at io.druid.indexing.common.task.HadoopIndexTask.isReady(HadoopIndexTask.java:137) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]
	at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:168) ~[druid-indexing-service-0.9.2.2.6.1.0-129.jar:0.9.2.2.6.1.0-129]

sbouguerra · ‎10-16-2017

@Ravi Mutyala can you please add the task json spec and the full task logs from the overlord console? Thanks

ravi1 · ‎10-17-2017

wikiticker-indexjson.txt

Task json spec attached.

nbangarwa · ‎10-18-2017

can you try setting hadoop.mapreduce.job.classloader=true in your mapreduce configs ?

ravi1 · ‎10-18-2017

I see -Dhadoop.mapreduce.job.classloader=true in druid.indexer.runner.javaOpts. I also added in mapreduce-site.xml but same issue.

Below is what I can see from overlord log.

2017-10-18T15:22:40,938 INFO [TaskQueue-Manager] io.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z] version[2017-10-18T15:22:40.938Z] for task: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 INFO [TaskQueue-Manager] io.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:40,940 WARN [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - No worker selections strategy set. Using default. 2017-10-18T15:22:40,941 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[HOST:8091] to add task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] 2017-10-18T15:22:40,944 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z switched from pending to running (on [HOST:8091]) 2017-10-18T15:22:40,944 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] status changed to [RUNNING]. 2017-10-18T15:22:40,951 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote RUNNING status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='null', port=-1}] 2017-10-18T15:22:40,965 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote RUNNING status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='HOST', port=8100}] 2017-10-18T15:22:40,965 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] location changed to [TaskLocation{host='HOST', port=8100}]. 2017-10-18T15:22:46,906 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] wrote FAILED status for task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on [TaskLocation{host='HOST', port=8100}] 2017-10-18T15:22:46,906 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[HOST:8091] completed task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] with status[FAILED] 2017-10-18T15:22:46,907 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_hadoop_wikiticker_2017-10-18T15:22:40.933Z 2017-10-18T15:22:46,907 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] on worker[HOST:8091] 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] from activeTasks 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] from TaskLock[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] 2017-10-18T15:22:46,908 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, dataSource=wikiticker, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, version=2017-10-18T15:22:40.938Z} 2017-10-18T15:22:46,911 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[5]: TaskLock{groupId=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, dataSource=wikiticker, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, version=2017-10-18T15:22:40.938Z} 2017-10-18T15:22:46,914 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_hadoop_wikiticker_2017-10-18T15:22:40.933Z to status: TaskStatus{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, status=FAILED, duration=5951} 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Task done: HadoopIndexTask{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, type=index_hadoop, dataSource=wikiticker} 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Task FAILED: HadoopIndexTask{id=index_hadoop_wikiticker_2017-10-18T15:22:40.933Z, type=index_hadoop, dataSource=wikiticker} (5951 run duration) 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] status changed to [FAILED]. 2017-10-18T15:22:46,915 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_hadoop_wikiticker_2017-10-18T15:22:40.933Z] went bye bye. 2017-10-18T15:23:27,138 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

kumarbrahma · ‎10-24-2017

I was able to resolve this issue. Please refer to the link below:

https://community.hortonworks.com/questions/141715/router-error-in-druid-with-kerberos.html?childToV...

Support Questions

index job fails with ClassNotFoundException