Member since
09-16-2016
10
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 27 | 03-17-2017 01:10 PM |
03-14-2018
07:40 AM
1 Kudo
Hey @kirk chou- sorry for not having posted this earlier. This was due to differences in system $PATH between the RHEL6 and the RHEL7 hosts (the 'usr/bin/ln' vs '/bin/ln' command in this case). Oozie forcefully overrides the ShellAction child tasks execution context, especially $PATH, as defined in the application master nodes. If values in $PATH on the application master node differ from $PATH on the worker node, then the task will fail on the worker node. Hope this helps, -Regis
... View more
01-05-2018
02:00 PM
Hi Community, I have setup a HDP 2.6.3 kerberized cluster, that comprises Atlas, Kafka, Storm. Now that I have kerberized the cluster, when I push a topology (storm jar mytopology.jar f.q.d.n.MyTopology topologyname), the AtlasHook has a problem determining keytab or ticket-cache for KafkaClient. 5667 [main] WARN o.a.a.h.AtlasHook - Error in determining keytab for KafkaClient-JAAS config
java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider This is really just a warning - in Atlas, I see the topology has been updated after completion of the command. I suspected that I had conflicting jars files in the classpath but it looks like it's not the case. This is the generated client_jaas.conf file StormClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="nimbus";
}; KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
}; The full stack trace look like follows. 1322 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7645321177559766236:-6292660308148401340
1827 [main] INFO o.a.s.m.n.Login - successfully logged in.
1904 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : gvscc-gw.novalocal:6627
2806 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
2812 [main] INFO o.a.s.m.n.Login - successfully logged in.
2849 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : gvscc-gw.novalocal:6627
2901 [main] INFO o.a.s.StormSubmitter - Uploading dependencies - jars...
2902 [main] INFO o.a.s.StormSubmitter - Uploading dependencies - artifacts...
2902 [main] INFO o.a.s.StormSubmitter - Dependency Blob keys - jars : [] / artifacts : []
2948 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /tmp/d606bdd4f21411e7af34fa174e2f5dce.jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-0d7bf304-4c62-4454-864a-571b4ada6662.jar
Start uploading file '/tmp/d606bdd4f21411e7af34fa174e2f5dce.jar' to '/hadoop/storm/nimbus/inbox/stormjar-0d7bf304-4c62-4454-864a-571b4ada6662.jar' (592151 bytes)
[==================================================] 592151 / 592151
File '/tmp/d606bdd4f21411e7af34fa174e2f5dce.jar' uploaded to '/hadoop/storm/nimbus/inbox/stormjar-0d7bf304-4c62-4454-864a-571b4ada6662.jar' (592151 bytes)
3002 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-0d7bf304-4c62-4454-864a-571b4ada6662.jar
3002 [main] INFO o.a.s.StormSubmitter - Submitting topology exclamate in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7645321177559766236:-6292660308148401340","topology.workers":3,"topology.debug":true}
3548 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: exclamate
3548 [main] INFO o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
4875 [main] INFO o.a.a.ApplicationProperties - Looking for atlas-application.properties in classpath
4876 [main] INFO o.a.a.ApplicationProperties - Loading atlas-application.properties from file:/etc/storm/2.6.3.0-235/0/atlas-application.properties
log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
5667 [main] WARN o.a.a.h.AtlasHook - Error in determining keytab for KafkaClient-JAAS config
java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:101) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:95) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:291) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:846) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:816) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.isLoginKeytabBased(UserGroupInformation.java:1376) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.atlas.hook.AtlasHook.isLoginKeytabBased(AtlasHook.java:232) [atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:80) [atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_72]
at java.lang.Class.forName(Class.java:348) [?:1.8.0_72]
at org.apache.atlas.storm.hook.StormAtlasHook.initialize(StormAtlasHook.java:78) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.storm.hook.StormAtlasHook.<init>(StormAtlasHook.java:46) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_72]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_72]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_72]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_72]
at java.lang.Class.newInstance(Class.java:442) [?:1.8.0_72]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:426) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:407) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at com.vrsn.vscc.ExclamationTopology.main(ExclamationTopology.java:60) [d606bdd4f21411e7af34fa174e2f5dce.jar:?]
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
... 25 more
5700 [main] WARN o.a.a.h.AtlasHook - Error in determining ticket-cache for KafkaClient-JAAS config
java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:101) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:95) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:291) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:846) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:816) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.isLoginTicketBased(UserGroupInformation.java:1384) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.atlas.hook.AtlasHook.isLoginTicketBased(AtlasHook.java:244) [atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:81) [atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_72]
at java.lang.Class.forName(Class.java:348) [?:1.8.0_72]
at org.apache.atlas.storm.hook.StormAtlasHook.initialize(StormAtlasHook.java:78) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.storm.hook.StormAtlasHook.<init>(StormAtlasHook.java:46) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_72]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_72]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_72]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_72]
at java.lang.Class.newInstance(Class.java:442) [?:1.8.0_72]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:426) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:407) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at com.vrsn.vscc.ExclamationTopology.main(ExclamationTopology.java:60) [d606bdd4f21411e7af34fa174e2f5dce.jar:?]
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
... 25 more
5801 [main] WARN o.a.a.h.AtlasHook - Error in determining current user
java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:101) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.<init>(Groups.java:95) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:291) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:846) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:816) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.atlas.hook.AtlasHook.getUser(AtlasHook.java:221) ~[atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.hook.AtlasHook.getUser(AtlasHook.java:190) ~[atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:92) [atlas-notification-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_72]
at java.lang.Class.forName(Class.java:348) [?:1.8.0_72]
at org.apache.atlas.storm.hook.StormAtlasHook.initialize(StormAtlasHook.java:78) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at org.apache.atlas.storm.hook.StormAtlasHook.<init>(StormAtlasHook.java:46) [storm-bridge-shim-0.8.0.2.6.3.0-235.jar:0.8.0.2.6.3.0-235]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_72]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_72]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_72]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_72]
at java.lang.Class.newInstance(Class.java:442) [?:1.8.0_72]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:426) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:407) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235]
at com.vrsn.vscc.ExclamationTopology.main(ExclamationTopology.java:60) [d606bdd4f21411e7af34fa174e2f5dce.jar:?]
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
... 26 more
5802 [main] INFO o.a.a.h.AtlasHook - Created Atlas Hook
5816 [main] INFO o.a.s.m.n.Login - successfully logged in.
5849 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : gvscc-gw.novalocal:6627
6103 [main] INFO o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
6103 [main] INFO o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology: exclamate
6164 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: storm_topology
7080 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
acks = 1
batch.size = 16384
block.on.buffer.full = false
bootstrap.servers = [gvscc-wn2.novalocal:6667]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.fetch.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXTSASL
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
timeout.ms = 30000
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
7089 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
acks = 1
batch.size = 16384
block.on.buffer.full = false
bootstrap.servers = [gvscc-wn2.novalocal:6667]
buffer.memory = 33554432
client.id = producer-1
compression.type = none
connections.max.idle.ms = 540000
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.fetch.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXTSASL
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
timeout.ms = 30000
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
7117 [main] INFO o.a.k.c.s.a.AbstractLogin - Successfully logged in.
7127 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT refresh thread started.
7138 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT valid starting at: 2018-01-05T12:35:18.000+0000
7139 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT expires: 2018-01-06T12:35:18.000+0000
7139 [kafka-kerberos-refresh-thread] INFO o.a.k.c.s.k.KerberosLogin - TGT refresh sleeping until: 2018-01-06T08:05:31.527+0000
7141 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'key.deserializer' was supplied but isn't a known config.
7141 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'value.deserializer' was supplied but isn't a known config.
7142 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'hook.group.id' was supplied but isn't a known config.
7142 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'zookeeper.connection.timeout.ms' was supplied but isn't a known config.
7143 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'zookeeper.session.timeout.ms' was supplied but isn't a known config.
7143 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'enable.auto.commit' was supplied but isn't a known config.
7144 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
7144 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'zookeeper.sync.time.ms' was supplied but isn't a known config.
7144 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'session.timeout.ms' was supplied but isn't a known config.
7144 [main] WARN o.a.k.c.p.ProducerConfig - The configuration 'auto.offset.reset' was supplied but isn't a known config.
7146 [main] INFO o.a.k.c.u.AppInfoParser - Kafka version : 0.10.1.2.6.3.0-235
7146 [main] INFO o.a.k.c.u.AppInfoParser - Kafka commitId : ba0af6800a08d2f8 This Atlas bug seems to be very similar https://issues.apache.org/jira/browse/ATLAS-1741 Anyone else saw that issue before ? Thanks for your support! -Regis
... View more
Labels:
09-26-2017
09:47 AM
Hi all - we have a cluster of RHEL6 and RHEL7 nodes. When oozie launches a workflow and uses a RHEL7 node as application master, the tasks dispatched to RHEL6 nodes fail to execute the launch_container.sh script. As per the log, it looks like the PATH could be not set properly, see below, as it cannot find the "ln" command. Stack trace: ExitCodeException exitCode=127: /data/d9/yarn/nm/usercache/hdfs/appcache/application_1506341577822_0486/container_e102_1506341577822_0486_01_000004/launch_container.sh: line 30: ln: command not found
at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
at org.apache.hadoop.util.Shell.run(Shell.java:487)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 127
The log from the job looks like below 2017-09-25 19:05:12,937 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1506341577822_0486_m_000000 Task Transitioned from SCHEDULED to RUNNING
2017-09-25 19:05:13,701 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1506341577822_0486: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:76800, vCores:0> knownNMs=2
2017-09-25 19:05:14,707 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e102_1506341577822_0486_01_000002
2017-09-25 19:05:14,708 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-09-25 19:05:14,710 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1506341577822_0486_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2017-09-25 19:05:14,710 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1506341577822_0486_m_000000_0: Exception from container-launch. When MR jobs are not triggered by Oozie, all is fine on those RHEL7 nodes. I'm able to reproduce the issue with HDP 2.3.4 and 2.3.6. I could not find any known Oozie issue that got fixed in later versions. Any help or pointers welcome. Best, -Regis
... View more
Labels:
03-17-2017
01:10 PM
My bad, the example pig script in the apache example assumed that quota is int - in my case, it's a long. changing the type fixed the issue.
... View more
03-17-2017
10:29 AM
Hi, running Hadoop 2.7.1, I use oiv to make an fsimage accessible via WebHDFS hdfs oiv -i fsimage_0000000000068206269 17/03/17 06:11:53 INFO offlineImageViewer.WebImageViewer: WebImageViewer started. Listening on /127.0.0.1:5978. Press Ctrl+C to stop the viewer. I can use a curl client to get information about some files in the fsimage successfully. curl -i "http://127.0.0.1:5978/webhdfs/v1/user/name/?op=GETFILESTATUS" The operation that returns the info that I need is GETCONTENTSUMMARY (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Content_Summary_of_a_Directory) but when I run that command curl -i "http://127.0.0.1:5978/webhdfs/v1/user/name/?op=GETCONTENTSUMMARY" I get HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Content-Length: 168
Connection: close
{"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid value for webhdfs parameter \"op\""}} The exact same webhdfs command issued to the namenode executes successfully ... Looks like the GETCONTENTSUMMARY operation is not supported with fsimage - could someone confirm? Best, -Regis
... View more
Labels:
03-16-2017
07:37 PM
1 Kudo
All, I would like to generate a list of all HDFS directories for which a quota has been set, and report the quota size. I have used dfs -count successfully from the command line - the shortcoming is, it is expensive, and running it recursively on every folder of a large HDFS Production cluster is probably not a good idea. I tried a different approach using fsimage, and oiv to Delimited format. However, the namespace quota and diskspace quota values are consistently -1, 0 or blank. I cannot seem to get the quota value anywhere. If you have pointers to why this is happening, or an alternative approach to achieve this, I'd love to hear it 🙂 Kindest regards, -Regis
... View more
- Tags:
- fsimage
- Hadoop Core
Labels:
09-29-2016
02:41 PM
We do use Ambari 2.2 - Smartsense 1.3 needs to be deployed to each and every node individually if using Ambari 2.2.
... View more
09-29-2016
09:46 AM
1 Kudo
All, I have installed SmartSense 1.3, including the Zeppelin instance. None of the Zeppelin notebooks work as each request/paragraph ends up trying to reach out to a hbase instance, see log below: Wed Sep 28 18:23:30 EDT 2016, RpcRetryingCaller{globalStartTime=1475100879458, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused I don't have an hbase instance running anywhere in my cluster - so, not sure if that is a requirement for SmartSense activity explorer / analyzer to perform their job? Any feedback welcome. Thanks! -Regis
... View more
09-16-2016
02:07 PM
@Artem Ervits Restarted the ambari-server, still no quick links. yes I can capture bundles.
... View more
09-16-2016
11:57 AM
1 Kudo
Hi, I have installed Smartsense 1.3 and use Ambari 2.2.2.0. When I go to the SmartSense Service to access the Activity Explorer, there are no quicklinks available in the summary section. As per documentation below, it should be available. http://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.3.0/bk_user-guide/content/activity_explorer.html Anyone seeing the same behaviour? Thanks, -Regis
... View more
Labels:
