Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

[#1]

Fill in the details...

[#2 · Accepted solution]

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

[#3]

So I basically need to write a Java program which can pull the data from the wsdl which will act as a key for the producer ?

I couldn't get the thing where you are suggesting to serialize the package and the task of connecting to WSDL..

p.s.- I am completely new in KAFKA,haven't written a single producer/consumer...

[#4]

So you need to pull your data from the soap service. Like you normally would. Which will give you some data.

Lets assume a JSON String? There are different options to push it into Kafka, the Example I used directly takes a String, ( there is one taking a byte array as well )

After that you can simple connect a consumer to kafka and you get the message out as you put them in, your consumer then needs to parse the string/byte array to whatever format you want.

[#5]

The KeyedMessage by the way can be confusing if called KeyedMessage("topic", "value") your kafka topic does not have a key. The first string is the topic the second the value of the message. This is good for most situations.

IF you need a message key ( db primary key for example ) to do log compaction etc. you can do

KeyedMessage("topic","key","message")

But for you I would assume that you simply serialize the SOAP output into a String and push it into Kafka, then you need a consumer on the other side that can parse it again. ( for example in spark or storm )

[#6]

@Rohan kapadekar are you open to using additional products? Take a look at apache nifi. It has a visual command and control and a large seg of processors to use, including native Kafka.

[#7]

Can you please explain in brief about using Apache NiFi. Will it be able to pull data by using a WSDL?

[#8]

My knowledge is limited on wsdl let's ask experts @jwitt @mpayne @Aldrin Piri

[#9]

NiFi does not at present offer any generic SOAP interface support. You would need to built a custom processor to do that. NiFi you can think of as a great host for the Java process suggested in this thread. Once the data is pulled via the SOAP API you can then use NiFi to do any number of things such as delivery to Kafka all within a managed process. Then you get the benefits NiFi offers and address your core use case.

[#10]

+1 on adding WSDL processor 🙂