Member since
04-30-2015
61
Posts
4
Kudos Received
0
Solutions
01-02-2017
06:12 AM
below are my source and sink properties. producer: name=local-file-source connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector tasks.max=1 file=/tmp/test.txt (with 777 mod) topic=newtest Sink: name=local-file-sink connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector tasks.max=1 file=/tmp/test.sink.txt (with 777 mod) topics=newtest
... View more
01-02-2017
06:05 AM
Hi - i'm trying for kafka file import and export but its failing with timed out. ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding messages, 1 left ({ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da=ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da}) (org.apache.kafka.connect.runtime.WorkerSourceTask:239) [2017-01-02 05:51:08,891] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112) i check both kafka server and zookeeper and those are running fine. no other error am seeing in logs.. please help me in fixing the issue thanks,sathish
... View more
- Tags:
- Kafka
- kafka-spout
Labels:
12-30-2016
06:44 AM
@Rajkumar Singh i started receiving below error messages now... while initiating producer... 1.no config changes done, 2.keytab looks fine and kinit successful. 3. ive created new topic thinking that the old one's corrupt but its giving me same error. ERROR fetching topic metadata for topics [Set(newtest)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,6667))] failed (kafka.utils.CoreUtils$) kafka.common.KafkaException: fetching topic metadata for topics [Set(newtest)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,6667))] failed caused : java.nio.channels.ClosedChannelException: thanks, sathish
... View more
12-29-2016
06:50 AM
Exception from container-launch. Container id: container_e64_1481762217559_27152_01_000002 Exit code: 127 Stack trace: ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is ajamwal main : requested yarn user is ajamwal Container exited with a non-zero exit code 127
... View more
12-29-2016
06:41 AM
also am seeing below messages from nodemanager 2016-12-28 08:28:26,005 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1481762217559_27152 2016-12-28 08:28:26,005 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,481 INFO ipc.Server (Server.java:saslProcess(1441)) - Auth successful for appattempt_1481762217559_27152_000001 (auth:SIMPLE) 2016-12-28 08:28:26,491 INFO authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(135)) - Authorization successful for appattempt_1481762217559_27152_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB 2016-12-28 08:28:26,491 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,492 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=ajamwal IP=10.246.73.94 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1481762217559_27152 CONTAINERID=container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,817 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:processHeartbeat(674)) - Unknown localizer with localizerId container_e64_1481762217559_27152_01_000008 is sending heartbeat. Ordering it to DIE 2016-12-28 08:28:26,818 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:processHeartbeat(674)) - Unknown localizer with localizerId container_e64_1481762217559_27152_01_000008 is sending heartbeat. Ordering it to DIE 2016-12-28 08:28:27,227 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(1131)) - Localizer failed java.io.IOException: java.lang.InterruptedException at org.apache.hadoop.util.Shell.runCommand(Shell.java:579) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:258) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1113) 2016-12-28 08:28:28,016 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_e64_1481762217559_27152_01_000008]
... View more
12-29-2016
06:15 AM
am seeing few thread messages too..
INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node xyz_123.com
2016-12-28 08:28:34,579 INFO [Thread-77] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2016-12-28 08:28:34,582 INFO [Thread-77] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1481762217559_27152_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
... View more
12-29-2016
06:08 AM
@gsharm
i dont see anything related to application specific error messages ... all i could see is below error messages. 2016-12-28 08:28:12,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1481762217559_27152_m_000000_1: Exception from container-launch.
Container id: container_e64_1481762217559_27152_01_000007
Exit code: 127
Stack trace: ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
at org.apache.hadoop.util.Shell.run(Shell.java:487)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
... View more
12-28-2016
11:04 AM
i'm facing frequent container exception on particular cluster but same set of jobs ran fine in other cluster. Exception from container-launch. Container id: container_e64_1481762217559_27152_01_000002 Exit code: 127 Stack trace: ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is xyz main : requested yarn user is xyz
... View more
Labels:
12-22-2016
09:14 AM
1 Kudo
@Rajkumar Singh can you please let me know if there is any doc which talks more about kafka ? also i wanted to understand more about data streaming between kafka clusters... and how kafka clusters communicates with each other ? thanks, sathish
... View more
12-22-2016
05:26 AM
1 Kudo
@Rajkumar Singh - how about kafka connect. kafka connect is the official one i believe... how those two (nifi & kafka connect) differs from one another.. ? thanks, sathish
... View more
12-22-2016
04:47 AM
@Rajkumar Singh i should have been more specific on my asks... sorry for that... actually i want to ingest/load/insert data from kafka producer to hdfs consumer... is there a way to do it with normal kafka commands or any tools available to do it ? thanks, sathish
... View more
12-21-2016
06:57 AM
Hi - i wanna transfer a complete file from one system to other using KAFKA ? can you please help in doing it ? thanks, sathish
... View more
Labels:
12-21-2016
06:19 AM
okay.. thanks for your help @Rajkumar Singh
... View more
12-21-2016
06:07 AM
@Rajkumar Singh i just wanna see the content of a topic(it may be a message,file,log or anything). how do i do it ? is there any webui or command line available to do so? please let me know.. thanks, sathish
... View more
12-21-2016
05:53 AM
@Rajkumar Singhsure.. will do that ... i have one more query (may be a silly one ) .. how can i browse the topic or can i see the content of the topic ? is there any specific tool to do so or can it be done through HDFS itself ? thanks, sathish
... View more
12-21-2016
05:17 AM
yes.. i removed hard coded hostnames and changed it to localhost:6667(default one), restarted services from ambari.. its running fine now.. thank you very much for the help @Rajkumar Singh
... View more
12-20-2016
12:09 PM
now i'm able to bring up the brokers on diff nodes through command line but ambari still has issues.. i think, i will have to re-visit my configurations on ambari-kafka.. bcoz its not taking the listeners and hostname properties for multinode. i will have to see how can i setup properties for each individual kafka-brokers... may be i ill have to set the listeners and hostname to 0.0.0.0:6667... thanks, sathish
... View more
12-20-2016
06:14 AM
also, when i create new kafka broker through ambari the config parameters are still be referring to the first kafka-broker... means, listeners and advertised.host.name are referring to the old/first broker hostname/Ip... i'm getting error "unable to connect to zookeeper" if i add multi nodes on those properties, thanks, sathish
... View more
12-20-2016
06:00 AM
can you please guide me in setting up multinode brokers... what are the properties should be changed while setting up the multi-broker thanks, sathish
... View more
12-20-2016
05:59 AM
version is 2.2.2 and i havent added any new properties in server.properties, all ive done is added new hosts with portno on existing properties. thanks, sathish
... View more
12-20-2016
05:53 AM
just to add, there are no issues if its single node broker through ambari see the below log details , but Issues are occurring only if its multi node broker. [2016-12-20 05:40:04,272] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 13 milliseconds. (kafka.coordinator.GroupMetadataManager) [2016-12-20 05:40:04,345] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper) [2016-12-20 05:40:04,348] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper) [2016-12-20 05:40:04,352] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2016-12-20 05:40:04,366] INFO Creating /brokers/ids/1001 (is it secure? true) (kafka.utils.ZKCheckedEphemeral) [2016-12-20 05:40:04,394] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral) [2016-12-20 05:40:04,396] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(10.246.131.99,6667,PLAINTEXTSASL) (kafka.utils.ZkUtils) [2016-12-20 05:40:04,402] INFO [Kafka Server 1001], started (kafka.server.KafkaServer) thanks, sathish
... View more
12-20-2016
05:17 AM
1 Kudo
@Rajkumar Singh there is no parameter advertised.listeners in ambari-kafka config but it exists in server.properties. however advertised.listeners property has got the same value as "listeners" ( both has got same values). thanks, sathish
... View more
12-19-2016
08:40 AM
and now ive changed all the protocols,port nos (9091,92,93) and started kafka through ambari but it started giving me the below error FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 25000 thanks, sathish
... View more
12-19-2016
08:35 AM
also i tried with diff host name with same protocol and i got below error Each listener must have a different protocol java.lang.IllegalArgumentException: requirement failed: Each listener must have a different protocol at scala.Predef$.require(Predef.scala:233) at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:907) at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696) at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:58) at kafka.Kafka.main(Kafka.scala)
... View more
12-19-2016
08:29 AM
Hi Raj - ive tried removing advertised listeners but it gets generated automatically.. also still am getting same error "Each listener must have a different port" thanks, sathish
... View more
12-19-2016
06:00 AM
below are my configurations listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667 advertised.listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667 below is the error [2016-12-19 05:57:09,852] FATAL (kafka.Kafka$) java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port at scala.Predef$.require(Predef.scala:233) at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:906) at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696) at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:58) at kafka.Kafka.main(Kafka.scala)
... View more
12-19-2016
05:49 AM
1 Kudo
i configure multibroker kafka through ambari but failed to startup the kafka broker. however, it was successful when tried doing it through command line (with diff protocol,diff portnos, diff nodes) and now i'm very much confused about the listeners property because by default amabri bound to 6667 port, what should be used for other listeners if im configuring multibroker. all most i tried all the combinations but non of them helped me while starting it from ambari and it gives me unable to connect to zookeeoer. example : 1. ( diff protocol, diff portnos, diff nodes) - this worked from command line but not through ambari 2.(diff protocol, diff portnos,same node) - this worked from command line but not through ambari 3. its not working neither command line nor ambari if this has been set to 'same protocol' or 'same port' - please help me sathish
... View more
Labels:
12-16-2016
05:23 AM
Just to add - also please see the below config from server.properties dvertised.host.name=10.246.131.35 advertised.listeners=PLAINTEXTSASL://tstr400367.abz-xyz.com:9092,SSL://tstr400367.abz-xyz.com:9091 authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer auto.create.topics.enable=true auto.leader.rebalance.enable=true compression.type=producer controlled.shutdown.enable=true controlled.shutdown.max.retries=3 controlled.shutdown.retry.backoff.ms=5000 controller.message.queue.size=10 controller.socket.timeout.ms=30000 default.replication.factor=1 delete.topic.enable=false external.kafka.metrics.exclude.prefix=kafka.network.RequestMetrics,kafka.server.DelayedOperationPurgatory,kafka.server.BrokerTopicMetrics.BytesRejectedPerSec external.kafka.metrics.include.prefix=kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetCommit.98percentile,kafka.network.RequestMetrics.ResponseQueueTimeMs.request.Offsets.95percentile,kafka.network.RequestMetrics.ResponseSendTimeMs.request.Fetch.95percentile,kafka.network.RequestMetrics.RequestsPerSec.request fetch.purgatory.purge.interval.requests=10000 kafka.ganglia.metrics.group=kafka kafka.ganglia.metrics.host=localhost kafka.ganglia.metrics.port=8671 kafka.ganglia.metrics.reporter.enabled=true kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter kafka.timeline.metrics.host=tstr400378.abz-xyz.com kafka.timeline.metrics.maxRowCacheSize=10000 kafka.timeline.metrics.port=6188 kafka.timeline.metrics.protocol=http kafka.timeline.metrics.reporter.enabled=true kafka.timeline.metrics.reporter.sendInterval=5900 kafka.timeline.metrics.truststore.password=bigdata kafka.timeline.metrics.truststore.path=/etc/security/clientKeys/all.jks kafka.timeline.metrics.truststore.type=jks leader.imbalance.check.interval.seconds=300 leader.imbalance.per.broker.percentage=10 listeners=PLAINTEXTSASL://tstr400367.abz-xyz.com:9092,SSL://tstr400367.abz-xyz.com:9091 log.cleanup.interval.mins=10 log.dirs=/u/applic/data/hdfs1/kafka-logs log.index.interval.bytes=4096 log.index.size.max.bytes=10485760 log.retention.bytes=-1 log.retention.hours=168 log.roll.hours=168 log.segment.bytes=1073741824 message.max.bytes=1000000 min.insync.replicas=1 num.io.threads=8 num.network.threads=3 num.partitions=1 num.recovery.threads.per.data.dir=1 num.replica.fetchers=1 offset.metadata.max.bytes=4096 offsets.commit.required.acks=-1 offsets.commit.timeout.ms=5000 offsets.load.buffer.size=5242880 offsets.retention.check.interval.ms=600000 offsets.retention.minutes=86400000 offsets.topic.compression.codec=0 offsets.topic.num.partitions=50 offsets.topic.replication.factor=3 offsets.topic.segment.bytes=104857600 principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal producer.purgatory.purge.interval.requests=10000 queued.max.requests=500 replica.fetch.max.bytes=1048576 replica.fetch.min.bytes=1 replica.fetch.wait.max.ms=500 replica.high.watermark.checkpoint.interval.ms=5000 replica.lag.max.messages=4000 replica.lag.time.max.ms=10000 replica.socket.receive.buffer.bytes=65536 replica.socket.timeout.ms=30000 security.inter.broker.protocol=PLAINTEXTSASL socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 super.users=user:kafka zookeeper.connect=tstr400364.abz-xyz.com:2181,tstr400360.abz-xyz.com:2181,tstr400362.abz-xyz.com:2181,tstr400363.abz-xyz.com:2181,tstr400361.abz-xyz.com:2181,tstr400367.abz-xyz.com:2181 zookeeper.connection.timeout.ms=25000 zookeeper.session.timeout.ms=30000 zookeeper.sync.time.ms=2000
... View more
12-16-2016
05:17 AM
no.. it didnt help and still is the same issue. thanks, sathish
... View more
12-16-2016
05:04 AM
thanks for the reply Raj. yes, its kerberized cluster and right now below are the values for kafka client KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; };
... View more
- « Previous
-
- 1
- 2
- Next »