question Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward. in Archives of Support Questions (Read Only)

Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

rohan_kapadekar — Wed, 27 Jan 2016 17:34:40 GMT

Fill in the details...

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

bleonhardi — Wed, 27 Jan 2016 17:48:17 GMT

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

rohan_kapadekar — Wed, 27 Jan 2016 18:42:28 GMT

So I basically need to write a Java program which can pull the data from the wsdl which will act as a key for the producer ?

I couldn't get the thing where you are suggesting to serialize the package and the task of connecting to WSDL..

p.s.- I am completely new in KAFKA,haven't written a single producer/consumer...

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

aervits — Wed, 27 Jan 2016 19:09:35 GMT

@Rohan kapadekar are you open to using additional products? Take a look at apache nifi. It has a visual command and control and a large seg of processors to use, including native Kafka.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

rohan_kapadekar — Wed, 27 Jan 2016 19:45:33 GMT

Can you please explain in brief about using Apache NiFi. Will it be able to pull data by using a WSDL?

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

aervits — Wed, 27 Jan 2016 19:47:56 GMT

My knowledge is limited on wsdl let's ask experts @jwitt @mpayne @Aldrin Piri

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

bleonhardi — Wed, 27 Jan 2016 23:25:25 GMT

So you need to pull your data from the soap service. Like you normally would. Which will give you some data.

Lets assume a JSON String? There are different options to push it into Kafka, the Example I used directly takes a String, ( there is one taking a byte array as well )

After that you can simple connect a consumer to kafka and you get the message out as you put them in, your consumer then needs to parse the string/byte array to whatever format you want.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

bleonhardi — Wed, 27 Jan 2016 23:28:11 GMT

The KeyedMessage by the way can be confusing if called KeyedMessage("topic", "value") your kafka topic does not have a key. The first string is the topic the second the value of the message. This is good for most situations.

IF you need a message key ( db primary key for example ) to do log compaction etc. you can do

KeyedMessage("topic","key","message")

But for you I would assume that you simply serialize the SOAP output into a String and push it into Kafka, then you need a consumer on the other side that can parse it again. ( for example in spark or storm )

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

JoeWitt — Thu, 28 Jan 2016 10:04:59 GMT

NiFi does not at present offer any generic SOAP interface support. You would need to built a custom processor to do that. NiFi you can think of as a great host for the Java process suggested in this thread. Once the data is pulled via the SOAP API you can then use NiFi to do any number of things such as delivery to Kafka all within a managed process. Then you get the benefits NiFi offers and address your core use case.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

aervits — Thu, 28 Jan 2016 10:07:01 GMT

+1 on adding WSDL processor 🙂

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

mgrabenstein — Tue, 02 Feb 2016 04:06:35 GMT

+1 (another) for WSDL Processor.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

aervits — Tue, 02 Feb 2016 22:47:02 GMT

@Rohan kapadekar has this been resolved? Can you post your solution or accept best answer?

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

rohan_kapadekar — Fri, 05 Feb 2016 13:54:30 GMT

@Artem Ervits I used the method which @Benjamin Leonhardi suggested.

Wrote a Java code which would send a SOAP request,get the response and converted it to JSON string.Then I pushed it to the KAFKA producer.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Nitin1130 — Mon, 03 Jul 2023 09:51:25 GMT

Hi @rohan_kapadekar I want to store the request and response of Soap api via using jms or Kafka, Can you help me in this or suggest a solution by which I can implement this.