Member since
11-16-2015
889
Posts
647
Kudos Received
245
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
543 | 02-22-2024 12:38 PM | |
745 | 02-02-2023 07:07 AM | |
2037 | 12-07-2021 09:19 AM | |
3343 | 03-20-2020 12:34 PM | |
11322 | 01-27-2020 07:57 AM |
12-09-2017
02:21 AM
1 Kudo
What version of NiFi are you using? Check NIFI-3000 for a history of what's been done and what hasn't been done. Depending on your version, you will likely want to switch to record-aware processors such as ConvertRecord, as they support logical types as of NIFI-2624 where some other processors may not. You may also be able to leverage PartitionRecord to help with grouping the same values, or QueryRecord (with LIMIT 1 perhaps) to help with duplicate detection/elimination.
... View more
12-06-2017
06:05 PM
It is highly recommended that you don't place JARs in NiFi's lib/ folder; as you can see it can affect the behavior of the whole instance. Instead you can put all the Hive driver JAR(s) into some other folder, and add the path to that folder to the Database Driver Location(s) property of the DBCPConnectionPool. There is also a HiveConnectionPool controller service, which is similar to DBCPConnectionPool but it comes packaged with a Hive 1.2 driver. Depending on your Hive version, you should be able to use the built-in driver via HiveConnectionPool. If your Hive server is from HDP 2.4+, you will want to use the HDF version of NiFi, as that has the HDP Hive driver packaged within, versus the Apache Hive driver.
... View more
12-02-2017
03:50 PM
1 Kudo
What version of Hortonworks Data Flow NiFi are you using? As of Apache NiFi 1.1.0 (I'm not sure offhand which HDF NiFi release that corresponds to, possibly HDF 3.0), the Elasticsearch 5 processors are available. Having said that, the ElasticsearchHttp processors (PutElasticsearchHttp in your screenshot above, e.g.) use the REST API rather than the Transport Client, so those processors are much more robust in the sense of which Elasticsearch versions they support. I believe the ElasticsearchHttp processors support at least 2.X through at least 5.0.1 (but I think they might work for all of 5.X). They are not quite as performant as the transport client versions (PutElasticsearch and PutElasticsearch5), but for your version of NiFi PutElasticsearchHttp should allow you to connect to your ES5 cluster.
... View more
12-02-2017
03:42 PM
In this case, you have two different schemas, one for the input and one for the output. The one you list above is the output schema, but you will also need an input schema such as: {
"type": "record",
"name": "employeeInfo",
"fields": [
{ "name": "ID", "type": "long", "default": 0 },
{ "name": "Name", "type": "string", "default": "defaultName" },
{ "name": "Age", "type": ["null", "long"], "default": 0 },
{ "name": "JoinedOn", "type": "string", "default": "defaultDate" }
]
} Because your input and output schemas are not the same, ConvertRecord is not the right choice for this operation; the doc says "The Reader and Writer must be configured with "matching" schemas. By this, we mean the schemas must have the same field names". I believe you will need two UpdateRecords in a row: 1) Move the employee info from the top-level into the array (using a Replacement Value Strategy of "Record Path Value"), with an example user-defined property for the ID field having property name "/columns/ID" and value "/ID". 2) I'm not sure where you are getting your values for timestamp, database, and table_name, but you can probably set those with another UpdateRecord, this time with a Replacement Value Strategy of "Literal Value", possibly using Expression Language to get the values from flow file attributes ("/table_name" = "${table.name}" for example), or functions ("/timestamp" = "${now():toNumber()}" for example).
... View more
12-01-2017
03:05 PM
It is highly recommended to not put your JDBC driver JAR(s) in NiFi's lib/ directory, as they can disturb the behavior of the other components in the system. I recommend a separate path containing the driver JAR and all its dependencies in a flat directory. Also on Windows you may need to use the URL style "file://C/" or just "/" instead of "C:\" but I'm not sure about that part. Another caveat with Hive drivers is that some (including the official Apache Hive JDBC driver that comes with NiFi's Hive bundle) do not support all JDBC methods, such as setQueryTimeout(), or have different mechanisms for getting at table metadata (from the ResultSetMetaData rather than DatabaseMetaData or vice versa) than what is used by ExecuteSQL. Those reasons are why there are SelectHiveQL and PutHiveQL processors, so the Hive driver could be included and the processors can perform any driver-specific functions/workarounds as necessary. So you may find that the SQL processors do not work with your Hive driver, but I am not familiar with that driver so I can't say for sure. If you see errors such as "Method not supported", then this usually indicates the scenario I'm talking about.
... View more
11-30-2017
06:27 PM
After ExecuteStreamCommand, you'll want an UpdateAttribute processor to set "filename" to "${filename}.csv", or a slightly more complicated expression if you are trying to replace the .xml extension with .csv rather than just appending .csv
... View more
11-29-2017
06:20 PM
Yeah the JOLT DSL can be confusing at times. Here's a Chain Spec that does what you describe above, so you can replace your processors with JoltTransformJSON: [
{
"operation": "shift",
"spec": {
"Name": "metric",
"Timestamp": "&"
}
},
{
"operation": "default",
"spec": {
"tags": {
"c": "d",
"a": "b"
}
}
}
]
... View more
11-28-2017
06:53 PM
Looks like it might just have been a typo between "flowFile" and "flowfile"
... View more
11-28-2017
06:42 PM
What version of NiFi are you using? Is the "value" column in your database table a String or a Float/Double? What processor(s) are you using to read from the database? If using ExecuteSQL, could you do something like the following? SELECT metric, CAST(value AS DOUBLE) AS value, timestamp, tags from myTable Alternatively, as of NiFi 1.2.0 (HDF 3.0) you can use the JoltTransformJSON processor to do type conversion (see an example here). Also if you know what the schema is supposed to be, you could use ConvertRecord with a JsonRecordSetWriter which is associated with the "correct" schema. The reader can be an AvroReader which uses the Embedded Schema.
... View more
11-28-2017
06:37 PM
Is it possible to share your nifi-app.log on this question? Also, does this driver work from other utilities (Squirrel SQL, e.g.)?
... View more