You can install HDF on an HDP cluster, but be mindful of which services are running on which nodes so you don't get performance issues. To install HDF on a new HDP cluster, you can use these instructions; for an existing HDP cluster, use this. ... View more

If you're updating the record structure, you'll need to update the schema as well. It sounds like the data coming out of ExecuteSQL may have different schemas based on which table is specified by the user. If you know all the possible schemas, you could enumerate them in a Schema Registry, where the key is some attribute (such as table name, let's say it's called table.name), and in the reader(s) you can use ${table.name} as the Schema Name property, and "Use Schema Name" as the access strategy. Then you could add "copies" of the input schemas, with the additional fields and any updated types, as output schemas. Those would be the ones used by the readers/writers after all your scripts have run. If the schemas truly are dynamic and you just need to add fields, then if the schema is in the avro.schema attribute, you could update it manually using a scripted processor such as ExecuteScript. For example, here is a Groovy script to be used in ExecuteScript for updating the avro.schema attribute by adding 2 extra entries to the "fields" array: import groovy.json.* def flowFiles = session.get(1000) if(!flowFiles) return session.transfer(flowFiles.collect { flowFile -> def schema = flowFile.getAttribute('avro.schema') def json = new JsonSlurper().parseText(schema) json.fields += ['name': 'processGroupName', 'type': ['null', 'string']] json.fields += ['name': 'processGroupId', 'type': ['null', 'string']] session.putAttribute(flowFile, 'avro.schema', JsonOutput.toJson(json)) }, REL_SUCCESS) The script grabs up to 1000 flow files, then for each one, it reads in the schema from the avro.schema attribute, parses the JSON, then adds processGroupName and processGroupId (both optional strings) to the "fields", then converts the JSON back to a string and updates the avro.schema attribute. Once this is done for all flow files, they are transferred to success. ... View more

For algorithms that don't require secret information, I think it could be helpful to add such functions to NiFi Expression Language, although they don't exist right now (please feel free to file a Jira to add such a capability). With this capability you could use UpdateRecord to replace a field with its hashed value. In the meantime, if you are comfortable with a scripting language such as Groovy, Javascript, Jython, JRuby, Clojure, or Lua, you could write a ScriptedLookupService that can "lookup" a specified field by returning the hash of its value. Then you can use LookupRecord with that service to get the hashed values. Alternatively you could implement a ScriptedRecordSetWriter that is configured to hash the values of the specified field(s), and then ConvertRecord with that writer. ... View more