Created 08-10-2018 12:41 PM
I am using ConsumerKafka processor to pull data from putkafka. Now i want to put same data into cassandra table. But i am getting errors. I am using PutCassandraQL processor to put data into cassandra table.
Is there any way to put data directly from ConsumerKafka to PutCassandraQL.
Response will be appreciated. Thanks.
Created 08-11-2018 11:53 PM
There is currently no processor that can put the raw incoming data directly into Cassandra. For PutCassandraQL you would need to translate the data into CQL statements. This is often done by converting to JSON (if necessary), then using ConvertJSONToSQL (as of NiFi 1.5.0 via NIFI-4684 you can set the Prefix to "cql" vs the default "sql"), then PutCassandraQL.
I have written NIFI-5510 in Jira to cover the addition of a PutCassandraRecord processor, which should cover your use case of being able to go directly from ConsumeKafka to Cassandra.
Created on 08-13-2018 11:26 AM - edited 08-17-2019 07:51 PM
Hi @Matt Burgess thank you for your quick response.
I tried as per your comment but problem still remains.
Can you go through below attached image.
I am not getting data in ConvertJSONToSQL processor.
My main objective is to put data into cassandra table through NiFI flow.
Created 08-13-2018 02:43 PM
I forgot that ConvertJSONToSQL takes a DBCPConnectionPool as a required property, it is used to connect to the database to get table and column/type information in order to translate your JSON into SQL statements. You might be able to use the Cassandra JDBC driver and set up a DBCPConnectionPool using that driver.
An alternative is to use ConvertRecord with a JSONReader and a FreeFormTextRecordSetWriter, you might be able to generate the SQL statements yourself if you know the names and types of the fields. If you don't know the schema of the JSON, you can try InferAvroSchema first, but then I'm not sure what you'd be able to do with the FreeForm writer.
Created 08-17-2018 06:04 AM
Hi @Matt Burgess thanks for your quick response.
I tried as per your suggestion but the error remains same. Is there any template related to this.
Created 08-17-2018 11:19 AM
Thanks for this great community.
could you suggest any work flow to get data from GPS device to cassandra table with using Kafka.