Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Solved Go to solution
Highlighted

Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

New Contributor

Fill in the details...

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

12 REPLIES 12

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

New Contributor

So I basically need to write a Java program which can pull the data from the wsdl which will act as a key for the producer ?

I couldn't get the thing where you are suggesting to serialize the package and the task of connecting to WSDL..

p.s.- I am completely new in KAFKA,haven't written a single producer/consumer...

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

So you need to pull your data from the soap service. Like you normally would. Which will give you some data.

Lets assume a JSON String? There are different options to push it into Kafka, the Example I used directly takes a String, ( there is one taking a byte array as well )

After that you can simple connect a consumer to kafka and you get the message out as you put them in, your consumer then needs to parse the string/byte array to whatever format you want.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

The KeyedMessage by the way can be confusing if called KeyedMessage("topic", "value") your kafka topic does not have a key. The first string is the topic the second the value of the message. This is good for most situations.

IF you need a message key ( db primary key for example ) to do log compaction etc. you can do

KeyedMessage("topic","key","message")

But for you I would assume that you simply serialize the SOAP output into a String and push it into Kafka, then you need a consumer on the other side that can parse it again. ( for example in spark or storm )

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

@Rohan kapadekar are you open to using additional products? Take a look at apache nifi. It has a visual command and control and a large seg of processors to use, including native Kafka.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

New Contributor

Can you please explain in brief about using Apache NiFi. Will it be able to pull data by using a WSDL?

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

My knowledge is limited on wsdl let's ask experts @jwitt @mpayne @Aldrin Piri

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

NiFi does not at present offer any generic SOAP interface support. You would need to built a custom processor to do that. NiFi you can think of as a great host for the Java process suggested in this thread. Once the data is pulled via the SOAP API you can then use NiFi to do any number of things such as delivery to Kafka all within a managed process. Then you get the benefits NiFi offers and address your core use case.

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

+1 on adding WSDL processor :)