question Re: kafka connect file import and export in Archives of Support Questions (Read Only)

kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 14:05:40 GMT

Hi - i'm trying for kafka file import and export but its failing with timed out.

ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding messages, 1 left ({ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da=ProducerRecord(topic=newtest, partition=null, key=[B@63d673d3, value=[B@144e54da}) (org.apache.kafka.connect.runtime.WorkerSourceTask:239)

[2017-01-02 05:51:08,891] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112)

i check both kafka server and zookeeper and those are running fine.

no other error am seeing in logs..

please help me in fixing the issue

thanks,sathish

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 14:12:47 GMT

below are my source and sink properties.

producer:

name=local-file-source

connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector

tasks.max=1

file=/tmp/test.txt (with 777 mod)

topic=newtest

Sink:

name=local-file-sink

connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector

tasks.max=1

file=/tmp/test.sink.txt (with 777 mod)

topics=newtest

Re: kafka connect file import and export

sandyy006 — Mon, 02 Jan 2017 16:58:52 GMT

@sathish jeganathan

Check connect-standalone.properties file and see if you have the valid broker hostname and port (bootstrap.servers).

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 17:06:35 GMT

yes.. its localhost:6667. initially it was localhost:9092 but i changed to 6667

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Mon, 02 Jan 2017 17:10:53 GMT

@sathish jeganathan Please replace localhost with the broker hostname and re-run the connect command.

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 17:13:15 GMT

@Sandeep Nemuri

yup., i've tried that too.. but still same error

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Mon, 02 Jan 2017 17:19:53 GMT

Can you change the log4j properties to DEBUG mode in connect-log4j.properties and see what is the error it is throwing ?

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 17:57:26 GMT

there are no such parms for kafak connect-log4j.properties.. can you please let me know the parameter

thanks,

sathish

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 17:58:52 GMT

right now i see all below parms in connect-log4j properties

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR

log4j.logger.org.I0Itec.zkclient=ERROR

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Mon, 02 Jan 2017 18:00:00 GMT

Here is the content of connect-log4j.properties

log4j.rootLogger=DEBUG, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.org.apache.zookeeper=DEBUG
log4j.logger.org.I0Itec.zkclient=DEBUG

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 18:02:54 GMT

yes.. i've started in debug mode.. please gimme some time .. am going through the logs now

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 18:11:34 GMT

see the below error msg's ..it says connection refused for the broker host.

DEBUG Connection with tstr400367.abc-test.com/10.246.131.35 disconnected (org.apache.kafka.common.network.Selector:307)

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:54)

at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:72)

at org.apache.kafka.common.network.Selector.poll(Selector.java:274)

at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256)

at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)

at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)

at java.lang.Thread.run(Thread.java:745)

i've mentioned proper node name with port id, now am not sure what to check

thanks,

sathish

Re: kafka connect file import and export

Sathish Jeganathan — Mon, 02 Jan 2017 18:49:09 GMT

@Sandeep Nemuri

i've setup port 6667 with sec.protocol to plaintextsasl.. but kafka connect by default running with producer properties (security.protocol = PLAINTEXT) . how can i override the parms for kafka connect. i've updated the parms in standalone.properties but kafka connect is not taking the parm while starting it. how should change the producer properties for kafka connect ?

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Mon, 02 Jan 2017 20:06:31 GMT

@sathish jeganathan

You need to add below properties in /etc/kafka/conf/connect-standalone.properties if you are using SASL.

producer.security.protocol=SASL_PLAINTEXT
producer.sasl.kerberos.service.name=kafka
consumer.security.protocol=SASL_PLAINTEXT
consumer.sasl.kerberos.service.name=kafka

Re: kafka connect file import and export

Sathish Jeganathan — Tue, 03 Jan 2017 12:35:52 GMT

yes.. its was problem with the security protocol.. i've changed them now and it started working now. is there any link or doc for parameter reference ?

thanks,

sathish

Re: kafka connect file import and export

Sathish Jeganathan — Tue, 03 Jan 2017 14:57:26 GMT

@Sandeep Nemuri

right now I'm testing with RDMS source(mysql) and kafka connect is failing for "connector.class". how can i find the correct connector class for rdms(mysql database).

i've tried with org.apache.kafka.connect.jdbc.JdbcSourceConnector,io.confulent.connect.jdbc.JdbcSourceConnector and both are not exist.

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Tue, 03 Jan 2017 16:38:27 GMT

@sathish jeganathan

I'm not sure about the mysql connector. But the class you are using belongs to https://github.com/confluentinc/kafka-connect-jdbc which is not shipped with HDP.

Re: kafka connect file import and export

Sathish Jeganathan — Tue, 03 Jan 2017 16:48:38 GMT

its not specific to mysql... connector class for "rdbms" source.

thanks,

sathish

Re: kafka connect file import and export

Sathish Jeganathan — Tue, 03 Jan 2017 16:56:32 GMT

i should have been more specific.. i'm looking for kafka connector class for rdbms as source. can you please let me know the connector class or any doc for reference...

thanks,

sathish

Re: kafka connect file import and export

sandyy006 — Tue, 03 Jan 2017 22:33:02 GMT

@sathish jeganathan

I am not sure about the connector, Request you to ask a new question for the same.