when create kafka connector, use localhost:8083, always get an error: curl(7) connection refused

[#1]

Hi, gurus, I have kafka 3 nodes cluster and try to create a kafka connector to mongodb.

I have followed this link, https://github.com/startappdev/kafka-connect-mongodb.

Every step is ok, till create a connector by curl, curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://localhost:8083/connectors

an error after this:

curl(7) connection refused. my port 8083 is not used by anyone on my localhost, I used 2181 for zeekeeper, 6667 for producer, what this 8083 does here?

hope someone setup this before so can share some details with me.

thank you very much.

Robin

[#2]

Hi @Robin Dong ,

port 8083 is the default port for the KafkaConnect Worker, if started in distributed mode....which is the case in the URL you are referring to. You can set this port to another one in the properties file you provide as parameter to the connect-distributed.sh cmdline call (the property is called rest.port , see here ).

In distributed mode you have to use the REST API to configure your Connectors, that's the only option. You can of course also start investigating into using Connect by starting with standalone mode. Then you do not need a REST call to configure your connector, you can just provide the connector.properties file as additional parameter at starting time of the ConnectWorker to the connect-standalone.sh script (ref. here)

Please try to replace 'localhost' by the FQDN of the host , where the Connect worker was started, and of course check if this start was successfull by looking at the listening ports e.g.

netstat -tulpn | grep 8083

HTH, Gerd

---------

If you find this post useful, any vote/reward highly appreciated 😉

[#3]

Thank you very much. I am trying to connect-standalone, and replace localhost to public IP, I still got the same error:

1) ./bin/connect-standalone.sh /tmp/connect-standalone .properties [2017-07-04 16:20:19,573] INFO Usage: ConnectStandalone worker.properties connec tor1.properties [connector2.properties ...] (org.apache.kafka.connect.cli.Connec tStandalone:61)

2) curl -X POST -H "Content-Type: application/json" --data @/tmp/mongo_connector_configs.json http://ec2-54-111-22-333.us-west-1.compute.amazonaws.com:8083/connectors

curl: (7) Failed connect to ec2-54-153-75-126.us-west-1.compute.amazonaws.com:8083; Connection refused

Can you help?

thanks,

Robin

[#4]

Hi, I came back to distributed mode again. After ssh-keygen -t rsa and cat id_rsa.pub >> authorized_keys, this may cause the 'connection refused' error in my case.

It was able to create the connector, and create testCollection in mongodb, however, no data in the testCollection. can you give a tip on why I could not pass the data?

also, netstat -tnulp|grep 8083 output nothing. so it means the connector is not created, right?

Thank you so much.

Robin

[#5]

Hi @Robin Dong ,

if you try the standalone-mode, there is no configuration via REST at all, hence you do NOT need any curl command to provide the connector config to your worker. In standalone-mode you pass the connector config as a second commandline parameter to start your worker, see here for an example how to start the standalone stuff including the connector config.

Maybe it is worth providing both configurations, the standalone worker as well the distributed one. If you start the distributed worker, at the end of the commandline output there you will find the URL to the REST-Interface. Can you paste that terminal output as well ?

Do you execute the curl command from the same node where you started Connect Worker, or is it from a remote host and maybe the AWS Network/Security settings prevent you from talking to the REST Interface ?

Regards, Gerd

[#6]

I know this is an old post, but maybe it can still be usefull to someone.

To start the standalone you need to pass both the worker config (connect-standalone.properties) and the connectors configurations (mongo_connector_config.properties in Robin's case).

For someone with the same problem, bare in mind that for starting the standalone connector you need to pass ".properties" files. To update or add new connectors you would need to pass ".json" files.

You can see in the confluent documentation page here.

bin/connect-standalone worker.properties connector1.properties [connector2.properties connector3.properties ...]

Vicente