Support Questions
Find answers, ask questions, and share your expertise

NiFi 1.3: How to write a JDBC inherit record schema to a Confuent Schema registry

New Contributor

Hello,


I am retrieving a csv content from a JDBC connection using ExecuteSQLRecord processor and I would like to add to the attribute the schema information id.

I can easily set an attribute with the avro schema definition of the query but I would like that NiFi wrote the schema to the Confluent Registry, retrieved the schema Id and add it to the flow attribute.

 

To achieve this is I set the following properties in the ExecuteSQLRecord:

Database Connection Pooling Service

DBCPConnectionPool

SQL Pre-Query

No value set

SQL select query

SQLQuery

SQL Post-Query

No value set

Max Wait Time

0 seconds

Record Writer

CSVRecordSetWriter

Normalize Table/Column Names

FALSE

Use Avro Logical Types

TRUE

Max Rows Per Flow File

fetchSize

Output Batch Size

0

Fetch Size

fetchSize

 

In the CSVRecordSetWriter I have the following settings:

Schema Write StrategyConfluent Schema Registry Reference
Schema CacheNo value set
Schema Access StrategyInherit Record Schema
Schema RegistryConfluentSchemaRegistry
Schema Name\${schema.name}
Schema Version 
Schema Branch 
Schema Text\${avro.schema}

 

And in the ConfluentSchemaRegistry I have:

Schema Registry URLshttp://host.docker.internal:8081
SSL Context ServiceNo value set
Communications Timeout30 secs
Cache Size1000
Cache Expiration1 hour

 

But, when I try to run it I get the following error:

org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write Confluent Schema registry reference because the schema identifier is not known.

 

So I have two questions:

  1. Is it possible (in an easy way) in NiFi to write a new Schema to Confluent Schema registry?
  2.  Can I write a new schema using the Inherit Record Schema strategy and retrieve a schema ID?

 

Thanks in advance,

Alex

 

1 REPLY 1

Re: NiFi 1.3: How to write a JDBC inherit record schema to a Confuent Schema registry

Hi @Alex454 

 

While we welcome your question and a member of the Cloudera Community might well be able to answer it, I believe you would be much more likely to obtain a timely solution if you posted it to the Confluent Community.

 

 

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.