In this article, I am going to explain how you can work with the Schema Registry directly in your NiFi Data Flow.  In my previous article Using the Schema Registry API I talk about the work required to expose the API methods needed to Create a Schema Registry Entity and update that Entity with an Avro Schema.   In this article, I am going to take it one step further and complete both operations directly in my NiFi Data Flow. I will also include the Use Case I am working on. This flow accepts a CSV Parameter List which contains the columns and data types found within the fields object of the Avro Schema, processes the contents of the file, builds the required Avro Schema and saves it to the Schema Registry Entity I created.   At the time of writing this article I have not handled data types, so in these examples, the schema data types are all strings.  In future updates to my flow, I will be mapping different types to appropriate data types for hive queries.   For now, this article and the NiFi template will include two main parts: Create a Schema Registry Entity Via NiFi using InvokeHttp Processor Update the Created Schema Registry with an Avro Schema created from a Parameter List The NiFi Template https://github.com/steven-dfheinz/NiFi-Templates/raw/master/Schema_Registry_Demo.xml Create a Schema Registry Entity Via NiFi using InvokeHttp Processor GenerateFlowFile→UpdateAttribute→AttributesToJSON→InvokeHttp **Note: notice my use of an output port to route failures during testing. While working on my flows I route all relationships like this until a time I decide it is appropriate to auto-terminate or route to error handling process group.   GenerateFlowFile - Very simple here. This proc just starts the flow. I have the Run Schedule at 1 minute so I can start & stop to create only a single flow file through the flow for testing. UpdateAttribute - Manually create attributes required for Schema Creation: name type schemaGroup description evolve compatibility AttributesToJSON - Writes Attributes above to flow-file contents. InvokeHTTP - Executes API Post to Schema Registry URL  **Note: to see specific configuration, please download the template, add to your workspace, and inspect procs. Update the Created Schema Registry with an Avro Schema created from a Parameter List HandleHttpRequest→HandleHttpResponse→UpdateAttribute→ConvertCSVToAvro→ConvertAvroToJSON→ExtractText→AttributesToJSON→ReplaceText→UpdateAttribute→AttributesToJSON→InvokeHTTP **Note: notice my use of an output port to route failures during testing.  While working on my flows I route all relationships like this until a time I decide it is appropriate to auto-terminate or route to error handling process group. HandleHttpRequest - API Endpoint in NiFi Flow necessary to accept POST of Parameter List. HandleHttpResponse - Sends API Response 200 and closes API Connection. UpdateAttribute - Manually set some Attributes Required for Avro Schema type name ConvertCSVToAvro - Converts the CSV contents to Avro Schema ConvertAvroToJSON - Converts Avro to JSON ExtractText - Grabs content of flow-file into fields attribute AttributesToJSON - Writes Attributes above to flow-file contents. ReplaceText - Handles some JSON object formatting requirements  ${'$1':unescapeJson():replace('"[','['):replace(']"',']')} UpdateAttribute - Attributes required for Schema Update: schemaText description AttributesToJSON - Writes Attributes above to flow-file contents. InvokeHTTP - Executes API Post to Schema Registry URL  **Note: to see specific configuration, please download the template, add to your workspace, and inspect procs. As always, if you have any questions, or comments feel free to add them below or send me a message.  If you need more help getting this template to work, I would be more than happy to help out. ... View more

In this article I am going to explain how to use the Schema Registry API to Create and Delete a Schema Entity.     I am working on a Use Case for creating schemas with over 500 columns on the fly within a NiFi Data Flow.  In order to complete this task I needed to figure out how to create, and update the schemas from outside of the Schema Registry UI.   While testing I also needed to delete the test schemas since there is no way to delete them from within the UI.   First, to get the artifacts I need (API urls, and post formats), I use the Schema Registry UI in Chrome with my Developer Tools open to capture what happens when the UI buttons are clicked.  When I add a test schema, I notice 2 POST calls.   Creates the Schema Entity (without an actual Schema) Posts the Schema to the newly created Entity Using the information in Developer Tools, I then create 2 Postman api calls to duplicate, debug, and validate they will work externally.   ** Note: I am working in a single node sandbox (hdf.cloudera.com) hdf cluster without any authorization. If your cluster is secured you will need to satisfy access, authorization, and ssl requirements. Create Schema Entity POST: http://hdf.cloudera.com:7788/api/v1/schemaregistry/schemas  BODY: {"name":"test","type":"avro","schemaGroup":"test","description":"Testing creating a schema from API","evolve":true,"compatibility":"BACKWARD"} Add Schema To Entity POST: http://hdf.cloudera.com:7788/api/v1/schemaregistry/schemas/test/versions?branch=MASTER BODY: {"schemaText":"{ \"name\" : \"test\", \"type\" : \"record\", \"fields\" : [ { \"name\" : \"column0\", \"type\" : \"string\" }, { \"name\" : \"column1\", \"type\" : \"string\" }, { \"name\" : \"column2\", \"type\" : \"string\" } ]}","description":"Testing creating a schema from API"} ** Note: the string escaped format above is required. If your schema has return lines those should also be replaced with \n. Delete Schema DELETE: http://hdf.cloudera.com:7788/api/v1/schemaregistry/schemas/test/ ** thanks to @mvalleavila on [ This Thead ] for exposing the simple Delete.   The information above should be enough to get you started using the Schema Registry API via command line, curl, or any other method you prefer.  In my next article, I will go into deeper detail in how I fit these concepts into my NiFi Flow. ... View more