Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache NiFi/ How to put data from kafka consumer to cassandra table ?

Apache NiFi/ How to put data from kafka consumer to cassandra table ?

New Contributor

I am using ConsumerKafka processor to pull data from putkafka. Now i want to put same data into cassandra table. But i am getting errors. I am using PutCassandraQL processor to put data into cassandra table.

Is there any way to put data directly from ConsumerKafka to PutCassandraQL.

Response will be appreciated. Thanks.

5 REPLIES 5

Re: Apache NiFi/ How to put data from kafka consumer to cassandra table ?

There is currently no processor that can put the raw incoming data directly into Cassandra. For PutCassandraQL you would need to translate the data into CQL statements. This is often done by converting to JSON (if necessary), then using ConvertJSONToSQL (as of NiFi 1.5.0 via NIFI-4684 you can set the Prefix to "cql" vs the default "sql"), then PutCassandraQL.

I have written NIFI-5510 in Jira to cover the addition of a PutCassandraRecord processor, which should cover your use case of being able to go directly from ConsumeKafka to Cassandra.

Re: Apache NiFi/ How to put data from kafka consumer to cassandra table ?

New Contributor

Hi @Matt Burgess thank you for your quick response.
I tried as per your comment but problem still remains.
Can you go through below attached image.
I am not getting data in ConvertJSONToSQL processor.
My main objective is to put data into cassandra table through NiFI flow.

86450-nifi-flow.jpg

Re: Apache NiFi/ How to put data from kafka consumer to cassandra table ?

I forgot that ConvertJSONToSQL takes a DBCPConnectionPool as a required property, it is used to connect to the database to get table and column/type information in order to translate your JSON into SQL statements. You might be able to use the Cassandra JDBC driver and set up a DBCPConnectionPool using that driver.

An alternative is to use ConvertRecord with a JSONReader and a FreeFormTextRecordSetWriter, you might be able to generate the SQL statements yourself if you know the names and types of the fields. If you don't know the schema of the JSON, you can try InferAvroSchema first, but then I'm not sure what you'd be able to do with the FreeForm writer.

Highlighted

Re: Apache NiFi/ How to put data from kafka consumer to cassandra table ?

New Contributor

Hi @Matt Burgess thanks for your quick response.

I tried as per your suggestion but the error remains same. Is there any template related to this.

Re: Apache NiFi/ How to put data from kafka consumer to cassandra table ?

New Contributor

Thanks for this great community.

could you suggest any work flow to get data from GPS device to cassandra table with using Kafka.