You can put an ExecuteScript processor before SelectHiveQL, and use that to check the time. If it must not execute then it can simply return without fetching any FlowFiles from the queue; otherwise you can simply fetch and transfer all available FlowFiles. ... View more

If I am reading your use case correctly, I think you're looking for what the ForkRecord processor does; it allows you to fork a (usually single) record into multiple records based on a Record Path (similar to JSONPath but different syntax and expressiveness), possibly keeping the "root" elements common to each outgoing record. ... View more

PutDatabaseRecord does not with work Hive (at least with the Apache Hive or HDP Hive JDBC drivers) as it uses methods of the JDBC API that are not supported by the driver. I'm not sure if the Simba driver works or not. A workaround for managed ACID tables is to use PutHiveStreaming instead, or if you have Hive 3, you can use PutHive3Streaming. The latter is a lot more flexible in terms of input, and relaxes some other requirements for the target table. For external tables, you could put the files directly into HDFS and create a table atop the target directory. If that "temp table" is to be inserted into a managed Hive table, you can then do that with PutHiveQL (after setting the flowfile content to the appropriate INSERT FROM SELECT HiveQL statement). Another (less-than-ideal) workaround is to convert individual records to SQL, update it to HiveQL as necessary (not always needed but sometimes), then send individual statements to PutHiveQL. This is very slow and ineffiencent ... View more

Can you share the configuration of your HortonworksSchemaRegistry controller service? ... View more

I don't think there's much you can do in NiFi, unless you can tune parameters in the JDBC URL, or use PartitionRecord to ensure each FlowFile has only one unique value for the partition column, but even then it looks like you'd need some special logic on the PostgreSQL side. I found this article about maybe this being more performant in later or upcoming versions of PostgreSQL, and this one about doing partitioning via child tables, not sure how your table(s) are set up but maybe some of this will help. ... View more

PutSQL actually doesn't want the semicolon at the end (that's for command-line and other stuff that allow multiple statements), do you get the same error if you leave the semicolon off? ... View more

The JsonPath Expression is meant to identify an array, then SplitJson will split each element into its own flow file. Try "$.data" as your JsonPath Expression, and use the "splits" relationship to send things downstream. The "original" relationship will contain the incoming flow file, which doesn't sound like what you want. ... View more

What does your query look like, what are the results, and what would you like to see in the single row/record? ... View more

The user-defined properties are available in your ProcessContext like all properties are, but they have a flag for being dynamic . Here's a snippet for iterating over the user-defined properties: context.getProperties().keySet().stream().filter(PropertyDescriptor::isDynamic).forEach() Fill in the lambda in the forEach() with your logic for handling the properties. (This is taken from my SO answer for the same question) ... View more

What does the generated SQL coming from ConvertJSONToSQL look like? Are the fields correctly uppercased? Does your database lowercase the column names? Did you try setting the "Translate Field Names" property to "true" in ConvertJSONToSQL? Does the case of the table name in the SQL match the case of the table name in the DB? If you're using "dbo.xxxx" as the Table Name property in ConvertJSONToSQL, instead try using just "xxxx" as the Table Name, and setting either Catalog Name or Schema Name (depending on your DB) to "dbo" (or DBO if necessary). ... View more

Have you tried PutDatabaseRecord? The reader provides the schema so there is no need to set the sql.args attributes. As of NiFi 1.9 the reader can also infer the datatypes, so you wouldn't have to specify the schema either. If you have an older NiFi, you can try ExecuteSQL -> ConvertRecord (Avro to JSON) -> InferAvroSchema -> PutDatabaseRecord, that's a heavy-handed way of getting the schema inferred. ... View more

You should be able to use SplitXml -> PutHDFS, using the Split Depth property to specify where to do the tag splitting. Each tag at that depth will be output as a separate flow file which you can send to HDFS via the PutHDFS processor. You may need to use UpdateAttribute to set the filename attribute, which is used by PutHDFS as the target filename. ... View more

You should be able to use ConvertRecord -> PutHDFS for this. ConvertRecord would use a XMLReader and CSVRecordSetWriter. ... View more

Did you mean ExtractText instead of ReplaceText? ... View more

You can use the SiteToSiteProvenanceReportingTask for this. Filter the reporting task to only emit events at the "certain point" you mention above. Each event has a "timestampMillis" and "lineageStart" field, you should be able to route on the difference of the two using QueryRecord, with something like: SELECT * FROM FLOWFILE WHERE timestampMillis - lineageStart > 60000 Which should emit a flowfile containing all events for which the associated entity (in this case, the flow file in the system) has been in the flow for over a minute. ... View more

You'll need to provide the CLOB as an attribute, meaning you've set attributes like sql.args.1.type to 2005 and sql.args.1.value to the CLOB value. Then your SQL statement would have a ? parameter, and the CLOB value will be inserted when the SQL statement is prepared. See NIFI-4352 for more information. ... View more

Once the value is in an attribute, it loses its type information as all attribute values are treated as strings. So when you use AttributesToJSON, you'll get a string value of the timestamp, not the long value. Instead, try UpdateRecord, you can set/generate the field using a ReplacementValueStrategy of "Literal Value", where you can use NiFi Expression Language such as ${now():toNumber()}. Just make sure that the field is in the output schema as a long. ... View more

You'll want to convert now() into a number for use by PutCassandraRecord, try ${now():toNumber()} instead. ... View more

Where in your flow do you have the information to populate the STATUS field? ... View more

It depends on how you are trying to "flatten" the nested XML into CSV fields, you can try UpdateRecord, JoltTransformRecord, etc. Each record-based processor does different transformations, but in the end they can all convert from XML to CSV, you just have to tell the right processor what to do. If you provide more information here (sample input and output, e.g.) I can try to help get you going. ... View more

The hard part here is that Hive returns STRUCT columns as JSON strings, so even if we can parse the JSON, we've lost the type information. It's possible we can retrieve it from the metadata and (if so) create a nested record from the results. Please feel free to file a Jira for this enhancement. ... View more