About Sathish Jeganathan

Sathish Jeganathan · ‎01-02-2017

below are my source and sink properties. producer: name=local-file-source connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector tasks.max=1 file=/tmp/test.txt (with 777 mod) topic=newtest Sink: name=local-file-sink connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector tasks.max=1 file=/tmp/test.sink.txt (with 777 mod) topics=newtest

Sathish Jeganathan · ‎01-02-2017

Hi - i'm trying for kafka file import and export but its failing with timed out. ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding messages, 1 left ({ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da=ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da}) (org.apache.kafka.connect.runtime.WorkerSourceTask:239) [2017-01-02 05:51:08,891] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112) i check both kafka server and zookeeper and those are running fine. no other error am seeing in logs.. please help me in fixing the issue thanks,sathish

Sathish Jeganathan · ‎12-30-2016

@Rajkumar Singh i started receiving below error messages now... while initiating producer... 1.no config changes done, 2.keytab looks fine and kinit successful. 3. ive created new topic thinking that the old one's corrupt but its giving me same error. ERROR fetching topic metadata for topics [Set(newtest)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,6667))] failed (kafka.utils.CoreUtils$) kafka.common.KafkaException: fetching topic metadata for topics [Set(newtest)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,6667))] failed caused : java.nio.channels.ClosedChannelException: thanks, sathish

Sathish Jeganathan · ‎12-29-2016

Exception from container-launch. Container id: container_e64_1481762217559_27152_01_000002 Exit code: 127 Stack trace: ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is ajamwal main : requested yarn user is ajamwal Container exited with a non-zero exit code 127

Sathish Jeganathan · ‎12-29-2016

also am seeing below messages from nodemanager 2016-12-28 08:28:26,005 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1481762217559_27152 2016-12-28 08:28:26,005 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,481 INFO ipc.Server (Server.java:saslProcess(1441)) - Auth successful for appattempt_1481762217559_27152_000001 (auth:SIMPLE) 2016-12-28 08:28:26,491 INFO authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(135)) - Authorization successful for appattempt_1481762217559_27152_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB 2016-12-28 08:28:26,491 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,492 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=ajamwal IP=10.246.73.94 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1481762217559_27152 CONTAINERID=container_e64_1481762217559_27152_01_000008 2016-12-28 08:28:26,817 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:processHeartbeat(674)) - Unknown localizer with localizerId container_e64_1481762217559_27152_01_000008 is sending heartbeat. Ordering it to DIE 2016-12-28 08:28:26,818 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:processHeartbeat(674)) - Unknown localizer with localizerId container_e64_1481762217559_27152_01_000008 is sending heartbeat. Ordering it to DIE 2016-12-28 08:28:27,227 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(1131)) - Localizer failed java.io.IOException: java.lang.InterruptedException at org.apache.hadoop.util.Shell.runCommand(Shell.java:579) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:258) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1113) 2016-12-28 08:28:28,016 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_e64_1481762217559_27152_01_000008]

Sathish Jeganathan · ‎12-29-2016

am seeing few thread messages too.. INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node xyz_123.com 2016-12-28 08:28:34,579 INFO [Thread-77] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2016-12-28 08:28:34,582 INFO [Thread-77] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1481762217559_27152_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0

Sathish Jeganathan · ‎12-29-2016

@gsharm i dont see anything related to application specific error messages ... all i could see is below error messages. 2016-12-28 08:28:12,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1481762217559_27152_m_000000_1: Exception from container-launch. Container id: container_e64_1481762217559_27152_01_000007 Exit code: 127 Stack trace: ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Sathish Jeganathan · ‎12-28-2016

i'm facing frequent container exception on particular cluster but same set of jobs ran fine in other cluster. Exception from container-launch. Container id: container_e64_1481762217559_27152_01_000002 Exit code: 127 Stack trace: ExitCodeException exitCode=127: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is xyz main : requested yarn user is xyz

Sathish Jeganathan · ‎12-22-2016

@Rajkumar Singh can you please let me know if there is any doc which talks more about kafka ? also i wanted to understand more about data streaming between kafka clusters... and how kafka clusters communicates with each other ? thanks, sathish

Sathish Jeganathan · ‎12-22-2016

@Rajkumar Singh - how about kafka connect. kafka connect is the official one i believe... how those two (nifi & kafka connect) differs from one another.. ? thanks, sathish

Sathish Jeganathan · ‎12-22-2016

@Rajkumar Singh i should have been more specific on my asks... sorry for that... actually i want to ingest/load/insert data from kafka producer to hdfs consumer... is there a way to do it with normal kafka commands or any tools available to do it ? thanks, sathish

Sathish Jeganathan · ‎12-21-2016

Hi - i wanna transfer a complete file from one system to other using KAFKA ? can you please help in doing it ? thanks, sathish

Sathish Jeganathan · ‎12-21-2016

okay.. thanks for your help @Rajkumar Singh

Sathish Jeganathan · ‎12-21-2016

@Rajkumar Singh i just wanna see the content of a topic(it may be a message,file,log or anything). how do i do it ? is there any webui or command line available to do so? please let me know.. thanks, sathish

Sathish Jeganathan · ‎12-21-2016

@Rajkumar Singhsure.. will do that ... i have one more query (may be a silly one ) .. how can i browse the topic or can i see the content of the topic ? is there any specific tool to do so or can it be done through HDFS itself ? thanks, sathish

Sathish Jeganathan · ‎12-21-2016

yes.. i removed hard coded hostnames and changed it to localhost:6667(default one), restarted services from ambari.. its running fine now.. thank you very much for the help @Rajkumar Singh

Sathish Jeganathan · ‎12-20-2016

now i'm able to bring up the brokers on diff nodes through command line but ambari still has issues.. i think, i will have to re-visit my configurations on ambari-kafka.. bcoz its not taking the listeners and hostname properties for multinode. i will have to see how can i setup properties for each individual kafka-brokers... may be i ill have to set the listeners and hostname to 0.0.0.0:6667... thanks, sathish

Sathish Jeganathan · ‎12-20-2016

also, when i create new kafka broker through ambari the config parameters are still be referring to the first kafka-broker... means, listeners and advertised.host.name are referring to the old/first broker hostname/Ip... i'm getting error "unable to connect to zookeeper" if i add multi nodes on those properties, thanks, sathish

Sathish Jeganathan · ‎12-20-2016

can you please guide me in setting up multinode brokers... what are the properties should be changed while setting up the multi-broker thanks, sathish

Sathish Jeganathan · ‎12-20-2016

version is 2.2.2 and i havent added any new properties in server.properties, all ive done is added new hosts with portno on existing properties. thanks, sathish

Sathish Jeganathan · ‎12-20-2016

just to add, there are no issues if its single node broker through ambari see the below log details , but Issues are occurring only if its multi node broker. [2016-12-20 05:40:04,272] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 13 milliseconds. (kafka.coordinator.GroupMetadataManager) [2016-12-20 05:40:04,345] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper) [2016-12-20 05:40:04,348] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper) [2016-12-20 05:40:04,352] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2016-12-20 05:40:04,366] INFO Creating /brokers/ids/1001 (is it secure? true) (kafka.utils.ZKCheckedEphemeral) [2016-12-20 05:40:04,394] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral) [2016-12-20 05:40:04,396] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(10.246.131.99,6667,PLAINTEXTSASL) (kafka.utils.ZkUtils) [2016-12-20 05:40:04,402] INFO [Kafka Server 1001], started (kafka.server.KafkaServer) thanks, sathish

Sathish Jeganathan · ‎12-20-2016

@Rajkumar Singh there is no parameter advertised.listeners in ambari-kafka config but it exists in server.properties. however advertised.listeners property has got the same value as "listeners" ( both has got same values). thanks, sathish

Sathish Jeganathan · ‎12-19-2016

and now ive changed all the protocols,port nos (9091,92,93) and started kafka through ambari but it started giving me the below error FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 25000 thanks, sathish

Sathish Jeganathan · ‎12-19-2016

also i tried with diff host name with same protocol and i got below error Each listener must have a different protocol java.lang.IllegalArgumentException: requirement failed: Each listener must have a different protocol at scala.Predef$.require(Predef.scala:233) at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:907) at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696) at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:58) at kafka.Kafka.main(Kafka.scala)

Sathish Jeganathan · ‎12-19-2016

Hi Raj - ive tried removing advertised listeners but it gets generated automatically.. also still am getting same error "Each listener must have a different port" thanks, sathish

Sathish Jeganathan · ‎12-19-2016

below are my configurations listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667 advertised.listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667 below is the error [2016-12-19 05:57:09,852] FATAL (kafka.Kafka$) java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port at scala.Predef$.require(Predef.scala:233) at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:906) at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699) at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696) at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:58) at kafka.Kafka.main(Kafka.scala)

Sathish Jeganathan · ‎12-19-2016

i configure multibroker kafka through ambari but failed to startup the kafka broker. however, it was successful when tried doing it through command line (with diff protocol,diff portnos, diff nodes) and now i'm very much confused about the listeners property because by default amabri bound to 6667 port, what should be used for other listeners if im configuring multibroker. all most i tried all the combinations but non of them helped me while starting it from ambari and it gives me unable to connect to zookeeoer. example : 1. ( diff protocol, diff portnos, diff nodes) - this worked from command line but not through ambari 2.(diff protocol, diff portnos,same node) - this worked from command line but not through ambari 3. its not working neither command line nor ambari if this has been set to 'same protocol' or 'same port' - please help me sathish

Sathish Jeganathan · ‎12-16-2016

Just to add - also please see the below config from server.properties dvertised.host.name=10.246.131.35 advertised.listeners=PLAINTEXTSASL://tstr400367.abz-xyz.com:9092,SSL://tstr400367.abz-xyz.com:9091 authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer auto.create.topics.enable=true auto.leader.rebalance.enable=true compression.type=producer controlled.shutdown.enable=true controlled.shutdown.max.retries=3 controlled.shutdown.retry.backoff.ms=5000 controller.message.queue.size=10 controller.socket.timeout.ms=30000 default.replication.factor=1 delete.topic.enable=false external.kafka.metrics.exclude.prefix=kafka.network.RequestMetrics,kafka.server.DelayedOperationPurgatory,kafka.server.BrokerTopicMetrics.BytesRejectedPerSec external.kafka.metrics.include.prefix=kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetCommit.98percentile,kafka.network.RequestMetrics.ResponseQueueTimeMs.request.Offsets.95percentile,kafka.network.RequestMetrics.ResponseSendTimeMs.request.Fetch.95percentile,kafka.network.RequestMetrics.RequestsPerSec.request fetch.purgatory.purge.interval.requests=10000 kafka.ganglia.metrics.group=kafka kafka.ganglia.metrics.host=localhost kafka.ganglia.metrics.port=8671 kafka.ganglia.metrics.reporter.enabled=true kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter kafka.timeline.metrics.host=tstr400378.abz-xyz.com kafka.timeline.metrics.maxRowCacheSize=10000 kafka.timeline.metrics.port=6188 kafka.timeline.metrics.protocol=http kafka.timeline.metrics.reporter.enabled=true kafka.timeline.metrics.reporter.sendInterval=5900 kafka.timeline.metrics.truststore.password=bigdata kafka.timeline.metrics.truststore.path=/etc/security/clientKeys/all.jks kafka.timeline.metrics.truststore.type=jks leader.imbalance.check.interval.seconds=300 leader.imbalance.per.broker.percentage=10 listeners=PLAINTEXTSASL://tstr400367.abz-xyz.com:9092,SSL://tstr400367.abz-xyz.com:9091 log.cleanup.interval.mins=10 log.dirs=/u/applic/data/hdfs1/kafka-logs log.index.interval.bytes=4096 log.index.size.max.bytes=10485760 log.retention.bytes=-1 log.retention.hours=168 log.roll.hours=168 log.segment.bytes=1073741824 message.max.bytes=1000000 min.insync.replicas=1 num.io.threads=8 num.network.threads=3 num.partitions=1 num.recovery.threads.per.data.dir=1 num.replica.fetchers=1 offset.metadata.max.bytes=4096 offsets.commit.required.acks=-1 offsets.commit.timeout.ms=5000 offsets.load.buffer.size=5242880 offsets.retention.check.interval.ms=600000 offsets.retention.minutes=86400000 offsets.topic.compression.codec=0 offsets.topic.num.partitions=50 offsets.topic.replication.factor=3 offsets.topic.segment.bytes=104857600 principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal producer.purgatory.purge.interval.requests=10000 queued.max.requests=500 replica.fetch.max.bytes=1048576 replica.fetch.min.bytes=1 replica.fetch.wait.max.ms=500 replica.high.watermark.checkpoint.interval.ms=5000 replica.lag.max.messages=4000 replica.lag.time.max.ms=10000 replica.socket.receive.buffer.bytes=65536 replica.socket.timeout.ms=30000 security.inter.broker.protocol=PLAINTEXTSASL socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 super.users=user:kafka zookeeper.connect=tstr400364.abz-xyz.com:2181,tstr400360.abz-xyz.com:2181,tstr400362.abz-xyz.com:2181,tstr400363.abz-xyz.com:2181,tstr400361.abz-xyz.com:2181,tstr400367.abz-xyz.com:2181 zookeeper.connection.timeout.ms=25000 zookeeper.session.timeout.ms=30000 zookeeper.sync.time.ms=2000

Sathish Jeganathan · ‎12-16-2016

no.. it didnt help and still is the same issue. thanks, sathish

Sathish Jeganathan · ‎12-16-2016

thanks for the reply Raj. yes, its kerberized cluster and right now below are the values for kafka client KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; };

Online	Offline
Last Visited	‎05-01-2015 03:16 AM

Date Registered	‎04-30-2015 11:54 PM
Last Visited	‎05-01-2015 03:16 AM
Total Messages Posted	61
Total Kudos Received	4

Re: kafka connect file import and export

kafka connect file import and export

Re: Kafka multi broker setup through ambari

Re: container exception Exit code: 127

Re: container exception Exit code: 127

Re: container exception Exit code: 127

Re: container exception Exit code: 127

container exception Exit code: 127

Re: transfer file using kafka

Re: transfer file using kafka

Re: transfer file using kafka

transfer file using kafka

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Re: Kafka multi broker setup through ambari

Kafka multi broker setup through ambari

Re: kafka- producer not pushing data to kafka- bro...

Re: kafka- producer not pushing data to kafka- bro...

Re: kafka- producer not pushing data to kafka- bro...