Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Solved Go to solution

Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Fill in the details...

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

View solution in original post

12 REPLIES 12
Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Kafka producers are Java programs. Here is a very simple example of a KafkaProducer that uses Syslog4J I did a while back for example.

In your case you need to have a program that can pull the data from the webservice and then push it into the producer.

You need to serialize your package into a byte array for kafka but apart from that the main work will be connecting to WSDL.

https://github.com/benleon/SysLogProducer

View solution in original post

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

So I basically need to write a Java program which can pull the data from the wsdl which will act as a key for the producer ?

I couldn't get the thing where you are suggesting to serialize the package and the task of connecting to WSDL..

p.s.- I am completely new in KAFKA,haven't written a single producer/consumer...

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

So you need to pull your data from the soap service. Like you normally would. Which will give you some data.

Lets assume a JSON String? There are different options to push it into Kafka, the Example I used directly takes a String, ( there is one taking a byte array as well )

After that you can simple connect a consumer to kafka and you get the message out as you put them in, your consumer then needs to parse the string/byte array to whatever format you want.

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

The KeyedMessage by the way can be confusing if called KeyedMessage("topic", "value") your kafka topic does not have a key. The first string is the topic the second the value of the message. This is good for most situations.

IF you need a message key ( db primary key for example ) to do log compaction etc. you can do

KeyedMessage("topic","key","message")

But for you I would assume that you simply serialize the SOAP output into a String and push it into Kafka, then you need a consumer on the other side that can parse it again. ( for example in spark or storm )

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

@Rohan kapadekar are you open to using additional products? Take a look at apache nifi. It has a visual command and control and a large seg of processors to use, including native Kafka.

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Can you please explain in brief about using Apache NiFi. Will it be able to pull data by using a WSDL?

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

My knowledge is limited on wsdl let's ask experts @jwitt @mpayne @Aldrin Piri

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

NiFi does not at present offer any generic SOAP interface support. You would need to built a custom processor to do that. NiFi you can think of as a great host for the Java process suggested in this thread. Once the data is pulled via the SOAP API you can then use NiFi to do any number of things such as delivery to Kafka all within a managed process. Then you get the benefits NiFi offers and address your core use case.

Highlighted

Re: Hi , I need to integrate data coming from a WSDL soap webservice into KAFKA.Please guide me the way forward.

Mentor

+1 on adding WSDL processor :)

Don't have an account?
Coming from Hortonworks? Activate your account here