Big Data DevOps: Apache NiFi - HWX Schema Registry - Schema Backups
- Oct 5, 2026 9:12 AM UTC
- 0 replies
- 5209 views
- Translate All
Big Data DevOps: Part 2: Schemas, Schemas, Schemas. Know Your Records, Know Your DataTypes, Know Your Fields, Know Your Data.
Since we can do process records in Apache NiFi, Streaming Analytics Manager, Apache Kafka and any tool that can work with a schema, we have a real need to use a Schema Registry. I have mentioned them before. One thing that is important is to be able to automate the management of schemas. Today we will be listing and exporting them for backup and migration purposes. We will also cover how to upload new schemas and version of schemas.
The steps to backup schemas with Apache NiFi 1.5+ is easy.
Backup All Schemas
- GetHTTP: Get the List of Schemas for SR via GET
- SplitJson to turn list into individual records
- EvaluateJsonPath: get the schema name.
- InvokeHTTP: get the schema body
- EvaluateJsonPath: turn the schema text into a separate flow file
- Rename and save both the full JSON record from the registry and the schema only.
NiFi Flow

Initial Call to List All Schemas

Get The Schema Name

Example Schema with Text

An Example of JSON Schema Text

Build a New Flow File from The Schema Text JSON

Get the Latest Version of the Schema Text For this Schema By Name

The List Returned

Swagger Documentation for SR





Example Flow
Schema List JSON Formatting
"entities" : [ { "schemaMetadata" : { "type" : "avro", "schemaGroup" : "Kafka", "name" : "adsb", "description" : "adsb", "compatibility" : "BACKWARD", "validationLevel" : "ALL", "evolve" : true }, "id" : 3, "timestamp" : 1520460239420
Get Schema List REST URL (GET)
http://server:7788/api/v1/schemaregistry/schemas
Get Schema Body REST URL (GET)
http://server:7788/api/v1/schemaregistry/schemas/${schema}/versions/latest?branch=MASTER
If you wish you can use the Confluent style API against SR and against Confluent Schema Registry. it is slighty different, but easy to change our REST calls to process this.
Swagger Docs
http://YourHWXRegistry:7788/swagger#!/4._Confluent_Schema_Registry_compatible_API/getSubjects
Hortonworks Schema Registry from HDF 3.1
https://community.hortonworks.com/articles/171893/hdf-31-executing-apache-spark-via-executesparkinte-1.html

