Member since
06-08-2017
1048
Posts
467
Kudos Received
310
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 255 | 10-15-2019 08:12 PM | |
| 182 | 10-12-2019 08:29 PM | |
| 512 | 09-21-2019 10:04 AM | |
| 292 | 09-19-2019 07:11 AM | |
| 202 | 09-09-2019 09:07 PM |
03-28-2020
08:15 AM
I want single file in the output which having all the records from array
... View more
03-11-2020
09:50 PM
@Vj1989
As this thread was marked 'Solved' in Dec 2017 you would have a better chance of receiving a useful response by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.
... View more
02-20-2020
02:52 AM
We are using cloudera version CDH-5.12.0-1.cdh5.12.0.p0.29. And we are able to parse the query in hiveServer2.logs with ": Compiling command" or ": Executing command" e.g. 2020-02-20 09:37:23,024 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Handler-Pool: Thread-17501]: Executing command(queryId=${ACTUAL_HIVE_QUERY_ID}): ${ACTUAL_QUERY} 2020-02-20 09:37:32,590 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Handler-Pool: Thread-17332]: Compiling command(queryId=${ACTUAL_HIVE_QUERY_ID}):${ACTUAL_QUERY} But what you need to take care of is, you need to aggregate all following lines to above line untill you find out time stamp at starting of logs. So basically it may happen that query contain multiline and in hiveServer2.log also it printed in multiline. So If you want full query then you might need to aggregate lines of logs. We are actually parsing hiveServer2.log to parse all information about, query submitted by which user, what is query, what are the yarnjobs spawned to execute query, how much time each job took and Status of each yarnJob and query.
... View more
02-05-2020
01:55 PM
Instead of using the function: prepend (this just shifts your data), how do you do an outright replace for the header row? I tried using Literal search, but I end up having strange characters when the replace occurs
... View more
01-20-2020
03:55 AM
Dear Sir, i want to enter python calculated values which are saved in df4 should get stored a column of ''per" per is name of column in hive table . df2 = pd.read_sql( "SELECT * FROM ggg where clientcode='55'" , conn) this is how i am reading the data how to write df4 python calculated values in "per" column of hive table.
... View more
01-06-2020
04:40 PM
@J1TEN Please open a new case/question versus responding to old topic. Also, take a look at the articles section, I just posted how to use Schema Registry API and another example how to do it in NiFi
... View more
12-29-2019
08:42 PM
@ask_bill_brooks , Sorry, i am not seeing the accept as solution option in my screen. Thanks.
... View more
12-04-2019
12:24 PM
This is now possible with NiFi 1.10 using the new parquet reader and writer.
... View more
11-28-2019
04:11 PM
Hi, for the moment I've done it with two branches and a Notify/Wait. From the GetFile, a first Success branch go to the Execute SQL that have a Pre-Query that do the Truncate, and a Query that do a simple select query (here a select count() of the truncated table that should return 0 and could be used as check), then that go to a Notify to inform that the truncate is done. From the GetFile, a Second Success branch go to the Wait (for the truncate to be done), than move to the PutDatabaseRecord (with a UpdateAttibute before to set the schema.name). Also here https://stackoverflow.com/questions/59026724/nifi-1-10-putsql-error-query-must-be-like-insert/59069223#59069223 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.3">
<description>Truncate Load for dimensions (slow batch frequency) with archive.
No error management implemented</description>
<groupId>a0377c1a-016e-1000-167a-e6f1c899a66f</groupId>
<name>Clickhouse Truncate Load</name>
<snippet>
<connections>
<id>0e31c732-4e03-37c9-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>6200e074-aa50-36e9-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>b8fa507a-797c-3714-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>19a865e3-39c1-3fee-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>f4b41b47-34d9-3bb4-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>6200e074-aa50-36e9-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>2c1086e0-bbca-393a-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>d741151d-7167-3026-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>56a6347a-9259-3a63-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>329ecc10-4aff-35e8-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>53f7bf4b-d5d4-32c6-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>d741151d-7167-3026-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>872806d3-be25-34ec-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>d24ba22b-88ad-34ae-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>62e39b15-77da-3fa5-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>8d81818b-b883-35f3-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>56a6347a-9259-3a63-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<prioritizers>org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer</prioritizers>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>f1513fa5-c26a-3157-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>ead6ac6b-a621-3f65-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>62e39b15-77da-3fa5-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<prioritizers>org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer</prioritizers>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>f1513fa5-c26a-3157-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>fccc8d95-eaed-3a31-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>b8fa507a-797c-3714-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<prioritizers>org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer</prioritizers>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bbf1ef49-f963-37ab-0000-000000000000</groupId>
<id>f1513fa5-c26a-3157-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<controllerServices>
<id>517808ef-0266-3e67-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<bundle>
<artifact>nifi-dbcp-service-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<comments></comments>
<descriptors>
<entry>
<key>Database Connection URL</key>
<value>
<name>Database Connection URL</name>
</value>
</entry>
<entry>
<key>Database Driver Class Name</key>
<value>
<name>Database Driver Class Name</name>
</value>
</entry>
<entry>
<key>database-driver-locations</key>
<value>
<name>database-driver-locations</name>
</value>
</entry>
<entry>
<key>kerberos-credentials-service</key>
<value>
<identifiesControllerService>org.apache.nifi.kerberos.KerberosCredentialsService</identifiesControllerService>
<name>kerberos-credentials-service</name>
</value>
</entry>
<entry>
<key>Database User</key>
<value>
<name>Database User</name>
</value>
</entry>
<entry>
<key>Password</key>
<value>
<name>Password</name>
</value>
</entry>
<entry>
<key>Max Wait Time</key>
<value>
<name>Max Wait Time</name>
</value>
</entry>
<entry>
<key>Max Total Connections</key>
<value>
<name>Max Total Connections</name>
</value>
</entry>
<entry>
<key>Validation-query</key>
<value>
<name>Validation-query</name>
</value>
</entry>
<entry>
<key>dbcp-min-idle-conns</key>
<value>
<name>dbcp-min-idle-conns</name>
</value>
</entry>
<entry>
<key>dbcp-max-idle-conns</key>
<value>
<name>dbcp-max-idle-conns</name>
</value>
</entry>
<entry>
<key>dbcp-max-conn-lifetime</key>
<value>
<name>dbcp-max-conn-lifetime</name>
</value>
</entry>
<entry>
<key>dbcp-time-between-eviction-runs</key>
<value>
<name>dbcp-time-between-eviction-runs</name>
</value>
</entry>
<entry>
<key>dbcp-min-evictable-idle-time</key>
<value>
<name>dbcp-min-evictable-idle-time</name>
</value>
</entry>
<entry>
<key>dbcp-soft-min-evictable-idle-time</key>
<value>
<name>dbcp-soft-min-evictable-idle-time</name>
</value>
</entry>
</descriptors>
<name>DBCP_SBST</name>
<persistsState>false</persistsState>
<properties>
<entry>
<key>Database Connection URL</key>
<value>jdbc:clickhouse://192.168.1.185:8123/SBST</value>
</entry>
<entry>
<key>Database Driver Class Name</key>
<value>ru.yandex.clickhouse.ClickHouseDriver</value>
</entry>
<entry>
<key>database-driver-locations</key>
<value>/home/thales/downloads/clickhouse-jdbc/clickhouse-jdbc-release_0.2/target/clickhouse-jdbc-0.2-jar-with-dependencies.jar</value>
</entry>
<entry>
<key>kerberos-credentials-service</key>
</entry>
<entry>
<key>Database User</key>
</entry>
<entry>
<key>Password</key>
</entry>
<entry>
<key>Max Wait Time</key>
<value>500 millis</value>
</entry>
<entry>
<key>Max Total Connections</key>
<value>8</value>
</entry>
<entry>
<key>Validation-query</key>
</entry>
<entry>
<key>dbcp-min-idle-conns</key>
<value>0</value>
</entry>
<entry>
<key>dbcp-max-idle-conns</key>
<value>8</value>
</entry>
<entry>
<key>dbcp-max-conn-lifetime</key>
<value>-1</value>
</entry>
<entry>
<key>dbcp-time-between-eviction-runs</key>
<value>-1</value>
</entry>
<entry>
<key>dbcp-min-evictable-idle-time</key>
<value>30 mins</value>
</entry>
<entry>
<key>dbcp-soft-min-evictable-idle-time</key>
<value>-1</value>
</entry>
</properties>
<state>ENABLED</state>
<type>org.apache.nifi.dbcp.DBCPConnectionPool</type>
</controllerServices>
<controllerServices>
<id>8d6517b8-c698-3871-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<bundle>
<artifact>nifi-record-serialization-services-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<comments></comments>
<descriptors>
<entry>
<key>schema-access-strategy</key>
<value>
<name>schema-access-strategy</name>
</value>
</entry>
<entry>
<key>schema-registry</key>
<value>
<identifiesControllerService>org.apache.nifi.schemaregistry.services.SchemaRegistry</identifiesControllerService>
<name>schema-registry</name>
</value>
</entry>
<entry>
<key>schema-name</key>
<value>
<name>schema-name</name>
</value>
</entry>
<entry>
<key>schema-version</key>
<value>
<name>schema-version</name>
</value>
</entry>
<entry>
<key>schema-branch</key>
<value>
<name>schema-branch</name>
</value>
</entry>
<entry>
<key>schema-text</key>
<value>
<name>schema-text</name>
</value>
</entry>
<entry>
<key>csv-reader-csv-parser</key>
<value>
<name>csv-reader-csv-parser</name>
</value>
</entry>
<entry>
<key>Date Format</key>
<value>
<name>Date Format</name>
</value>
</entry>
<entry>
<key>Time Format</key>
<value>
<name>Time Format</name>
</value>
</entry>
<entry>
<key>Timestamp Format</key>
<value>
<name>Timestamp Format</name>
</value>
</entry>
<entry>
<key>CSV Format</key>
<value>
<name>CSV Format</name>
</value>
</entry>
<entry>
<key>Value Separator</key>
<value>
<name>Value Separator</name>
</value>
</entry>
<entry>
<key>Skip Header Line</key>
<value>
<name>Skip Header Line</name>
</value>
</entry>
<entry>
<key>ignore-csv-header</key>
<value>
<name>ignore-csv-header</name>
</value>
</entry>
<entry>
<key>Quote Character</key>
<value>
<name>Quote Character</name>
</value>
</entry>
<entry>
<key>Escape Character</key>
<value>
<name>Escape Character</name>
</value>
</entry>
<entry>
<key>Comment Marker</key>
<value>
<name>Comment Marker</name>
</value>
</entry>
<entry>
<key>Null String</key>
<value>
<name>Null String</name>
</value>
</entry>
<entry>
<key>Trim Fields</key>
<value>
<name>Trim Fields</name>
</value>
</entry>
<entry>
<key>csvutils-character-set</key>
<value>
<name>csvutils-character-set</name>
</value>
</entry>
</descriptors>
<name>CSVReaderTild</name>
<persistsState>false</persistsState>
<properties>
<entry>
<key>schema-access-strategy</key>
<value>schema-name</value>
</entry>
<entry>
<key>schema-registry</key>
<value>f6754d68-6957-3155-0000-000000000000</value>
</entry>
<entry>
<key>schema-name</key>
<value>${schema.name}</value>
</entry>
<entry>
<key>schema-version</key>
</entry>
<entry>
<key>schema-branch</key>
</entry>
<entry>
<key>schema-text</key>
<value>${avro.schema}</value>
</entry>
<entry>
<key>csv-reader-csv-parser</key>
<value>commons-csv</value>
</entry>
<entry>
<key>Date Format</key>
</entry>
<entry>
<key>Time Format</key>
</entry>
<entry>
<key>Timestamp Format</key>
</entry>
<entry>
<key>CSV Format</key>
<value>custom</value>
</entry>
<entry>
<key>Value Separator</key>
<value>~</value>
</entry>
<entry>
<key>Skip Header Line</key>
<value>false</value>
</entry>
<entry>
<key>ignore-csv-header</key>
<value>false</value>
</entry>
<entry>
<key>Quote Character</key>
<value>"</value>
</entry>
<entry>
<key>Escape Character</key>
<value>\</value>
</entry>
<entry>
<key>Comment Marker</key>
</entry>
<entry>
<key>Null String</key>
</entry>
<entry>
<key>Trim Fields</key>
<value>true</value>
</entry>
<entry>
<key>csvutils-character-set</key>
<value>UTF-8</value>
</entry>
</properties>
<state>ENABLED</state>
<type>org.apache.nifi.csv.CSVReader</type>
</controllerServices>
<controllerServices>
<id>ccdef25c-4ad3-37ac-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<bundle>
<artifact>nifi-distributed-cache-services-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<comments></comments>
<descriptors>
<entry>
<key>Server Hostname</key>
<value>
<name>Server Hostname</name>
</value>
</entry>
<entry>
<key>Server Port</key>
<value>
<name>Server Port</name>
</value>
</entry>
<entry>
<key>SSL Context Service</key>
<value>
<identifiesControllerService>org.apache.nifi.ssl.SSLContextService</identifiesControllerService>
<name>SSL Context Service</name>
</value>
</entry>
<entry>
<key>Communications Timeout</key>
<value>
<name>Communications Timeout</name>
</value>
</entry>
</descriptors>
<name>DistributedMapCacheClientService</name>
<persistsState>false</persistsState>
<properties>
<entry>
<key>Server Hostname</key>
<value>192.168.1.190</value>
</entry>
<entry>
<key>Server Port</key>
<value>4557</value>
</entry>
<entry>
<key>SSL Context Service</key>
</entry>
<entry>
<key>Communications Timeout</key>
<value>30 secs</value>
</entry>
</properties>
<state>ENABLED</state>
<type>org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService</type>
</controllerServices>
<controllerServices>
<id>f6754d68-6957-3155-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<bundle>
<artifact>nifi-registry-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<comments></comments>
<descriptors>
<entry>
<key>avro-reg-validated-field-names</key>
<value>
<name>avro-reg-validated-field-names</name>
</value>
</entry>
<entry>
<key>functionalcat</key>
<value>
<name>functionalcat</name>
</value>
</entry>
<entry>
<key>geographicalcat</key>
<value>
<name>geographicalcat</name>
</value>
</entry>
<entry>
<key>hisevent</key>
<value>
<name>hisevent</name>
</value>
</entry>
<entry>
<key>hiseventorigin</key>
<value>
<name>hiseventorigin</name>
</value>
</entry>
</descriptors>
<name>AvroSchemaRegistry</name>
<persistsState>false</persistsState>
<properties>
<entry>
<key>avro-reg-validated-field-names</key>
<value>true</value>
</entry>
<entry>
<key>functionalcat</key>
<value>{
"type": "record",
"name": "functionalcat",
"fields": [
{ "name": "FUNCTIONALCAT", "type": "int" },
{ "name": "FUNCTIONALDESC", "type": "string" }
]
}</value>
</entry>
<entry>
<key>geographicalcat</key>
<value>{
"type": "record",
"name": "geographicalcat",
"fields": [
{ "name": "GEOGRAPHICALCAT", "type": "int" },
{ "name": "GEOGRAPHICALCODE", "type": "string" },
{ "name": "GEOGRAPHICALDESC", "type": "string" }
]
}</value>
</entry>
<entry>
<key>hisevent</key>
<value>{
"type": "record",
"name": "hisevent",
"fields": [
{ "name": "ALARMID", "type": ["null", "long"] },
{ "name": "ENVIRONMENT", "type": ["null", "string"] },
{ "name": "VALUE", "type": "long" },
{ "name": "ACKREQUIRED", "type": "long" },
{ "name": "SEVERITY", "type": "long" },
{ "name": "EQUIPMENTCLASS", "type": ["null", "long"] },
{ "name": "FUNCTIONALCAT", "type": "long" },
{ "name": "GEOGRAPHICALCAT", "type": "long" },
{ "name": "DATEANDTIME", "type": "string" },
{ "name": "EQUIPMENTNAME", "type": "string" },
{ "name": "ASSETNAME", "type": "string" },
{ "name": "MESSAGE", "type": ["null", "string"] },
{ "name": "STATUS", "type": ["null", "string"] },
{ "name": "GROUP1", "type": ["null", "string"] },
{ "name": "GROUP2", "type": ["null", "string"] },
{ "name": "FORMAT", "type": ["null", "string"] },
{ "name": "DSSEVENTTYPE", "type": ["null", "string"] },
{ "name": "OPER", "type": ["null", "string"] }
]
}</value>
</entry>
<entry>
<key>hiseventorigin</key>
<value>{
"type": "record",
"name": "hisevent",
"fields": [
{ "name": "ALARMID", "type": ["null", "long"] },
{ "name": "ENVIRONMENT", "type": ["null", "string"] },
{ "name": "VALUE", "type": "long" },
{ "name": "ACKREQUIRED", "type": "long" },
{ "name": "SEVERITY", "type": "long" },
{ "name": "EQUIPMENTCLASS", "type": ["null", "long"] },
{ "name": "FUNCTIONALCAT", "type": "long" },
{ "name": "GEOGRAPHICALCAT", "type": "long" },
{ "name": "DATEANDTIME", "type": "string" },
{ "name": "EQUIPMENTNAME", "type": "string" },
{ "name": "ASSETNAME", "type": "string" },
{ "name": "MESSAGE", "type": ["null", "string"] },
{ "name": "STATUS", "type": ["null", "string"] },
{ "name": "GROUP1", "type": ["null", "string"] },
{ "name": "GROUP2", "type": ["null", "string"] },
{ "name": "FORMAT", "type": ["null", "string"] },
{ "name": "DSSEVENTTYPE", "type": ["null", "string"] },
{ "name": "OPER", "type": ["null", "string"] },
{ "name": "EMPTY", "type": ["null", "string"] }
]
}</value>
</entry>
</properties>
<state>ENABLED</state>
<type>org.apache.nifi.schemaregistry.services.AvroSchemaRegistry</type>
</controllerServices>
<labels>
<id>7bbcc1a5-298e-31f1-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>1040.0</x>
<y>616.0</y>
</position>
<height>184.0</height>
<label>Main Folder Level : Avro Registry, add Schema that describe the CSV
Variable RootFolder : File Location root
GetFile : input Directory based on RootFolder and File Name
ExecuteSQL : Truncate table and count queries
NotiFy and Wait : Name of the Identifier
Update Attribute : Schema Name
PutDatabaseRecord : Create a Record Reader controller CSVReader at this process group level,
just changing the CSV format, and using the registry of the root process group,
change the table name, the 2 Unmatched parmaters to Ignore</label>
<style>
<entry>
<key>font-size</key>
<value>14px</value>
</entry>
</style>
<width>688.0</width>
</labels>
<processors>
<id>53f7bf4b-d5d4-32c6-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>624.0</x>
<y>680.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>put-db-record-record-reader</key>
<value>
<identifiesControllerService>org.apache.nifi.serialization.RecordReaderFactory</identifiesControllerService>
<name>put-db-record-record-reader</name>
</value>
</entry>
<entry>
<key>put-db-record-statement-type</key>
<value>
<name>put-db-record-statement-type</name>
</value>
</entry>
<entry>
<key>put-db-record-dcbp-service</key>
<value>
<identifiesControllerService>org.apache.nifi.dbcp.DBCPService</identifiesControllerService>
<name>put-db-record-dcbp-service</name>
</value>
</entry>
<entry>
<key>put-db-record-catalog-name</key>
<value>
<name>put-db-record-catalog-name</name>
</value>
</entry>
<entry>
<key>put-db-record-schema-name</key>
<value>
<name>put-db-record-schema-name</name>
</value>
</entry>
<entry>
<key>put-db-record-table-name</key>
<value>
<name>put-db-record-table-name</name>
</value>
</entry>
<entry>
<key>put-db-record-translate-field-names</key>
<value>
<name>put-db-record-translate-field-names</name>
</value>
</entry>
<entry>
<key>put-db-record-unmatched-field-behavior</key>
<value>
<name>put-db-record-unmatched-field-behavior</name>
</value>
</entry>
<entry>
<key>put-db-record-unmatched-column-behavior</key>
<value>
<name>put-db-record-unmatched-column-behavior</name>
</value>
</entry>
<entry>
<key>put-db-record-update-keys</key>
<value>
<name>put-db-record-update-keys</name>
</value>
</entry>
<entry>
<key>put-db-record-field-containing-sql</key>
<value>
<name>put-db-record-field-containing-sql</name>
</value>
</entry>
<entry>
<key>put-db-record-allow-multiple-statements</key>
<value>
<name>put-db-record-allow-multiple-statements</name>
</value>
</entry>
<entry>
<key>put-db-record-quoted-identifiers</key>
<value>
<name>put-db-record-quoted-identifiers</name>
</value>
</entry>
<entry>
<key>put-db-record-quoted-table-identifiers</key>
<value>
<name>put-db-record-quoted-table-identifiers</name>
</value>
</entry>
<entry>
<key>put-db-record-query-timeout</key>
<value>
<name>put-db-record-query-timeout</name>
</value>
</entry>
<entry>
<key>rollback-on-failure</key>
<value>
<name>rollback-on-failure</name>
</value>
</entry>
<entry>
<key>table-schema-cache-size</key>
<value>
<name>table-schema-cache-size</name>
</value>
</entry>
<entry>
<key>put-db-record-max-batch-size</key>
<value>
<name>put-db-record-max-batch-size</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>put-db-record-record-reader</key>
<value>8d6517b8-c698-3871-0000-000000000000</value>
</entry>
<entry>
<key>put-db-record-statement-type</key>
<value>INSERT</value>
</entry>
<entry>
<key>put-db-record-dcbp-service</key>
<value>517808ef-0266-3e67-0000-000000000000</value>
</entry>
<entry>
<key>put-db-record-catalog-name</key>
</entry>
<entry>
<key>put-db-record-schema-name</key>
<value>SBST</value>
</entry>
<entry>
<key>put-db-record-table-name</key>
<value>geographicalcat</value>
</entry>
<entry>
<key>put-db-record-translate-field-names</key>
<value>true</value>
</entry>
<entry>
<key>put-db-record-unmatched-field-behavior</key>
<value>Ignore Unmatched Fields</value>
</entry>
<entry>
<key>put-db-record-unmatched-column-behavior</key>
<value>Ignore Unmatched Columns</value>
</entry>
<entry>
<key>put-db-record-update-keys</key>
</entry>
<entry>
<key>put-db-record-field-containing-sql</key>
</entry>
<entry>
<key>put-db-record-allow-multiple-statements</key>
<value>false</value>
</entry>
<entry>
<key>put-db-record-quoted-identifiers</key>
<value>false</value>
</entry>
<entry>
<key>put-db-record-quoted-table-identifiers</key>
<value>false</value>
</entry>
<entry>
<key>put-db-record-query-timeout</key>
<value>0 seconds</value>
</entry>
<entry>
<key>rollback-on-failure</key>
<value>false</value>
</entry>
<entry>
<key>table-schema-cache-size</key>
<value>100</value>
</entry>
<entry>
<key>put-db-record-max-batch-size</key>
<value>0</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_PutDatabaseRecord</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>retry</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.PutDatabaseRecord</type>
</processors>
<processors>
<id>56a6347a-9259-3a63-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>624.0</x>
<y>248.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>release-signal-id</key>
<value>
<name>release-signal-id</name>
</value>
</entry>
<entry>
<key>target-signal-count</key>
<value>
<name>target-signal-count</name>
</value>
</entry>
<entry>
<key>signal-counter-name</key>
<value>
<name>signal-counter-name</name>
</value>
</entry>
<entry>
<key>wait-buffer-count</key>
<value>
<name>wait-buffer-count</name>
</value>
</entry>
<entry>
<key>releasable-flowfile-count</key>
<value>
<name>releasable-flowfile-count</name>
</value>
</entry>
<entry>
<key>expiration-duration</key>
<value>
<name>expiration-duration</name>
</value>
</entry>
<entry>
<key>distributed-cache-service</key>
<value>
<identifiesControllerService>org.apache.nifi.distributed.cache.client.AtomicDistributedMapCacheClient</identifiesControllerService>
<name>distributed-cache-service</name>
</value>
</entry>
<entry>
<key>attribute-copy-mode</key>
<value>
<name>attribute-copy-mode</name>
</value>
</entry>
<entry>
<key>wait-mode</key>
<value>
<name>wait-mode</name>
</value>
</entry>
<entry>
<key>wait-penalty-duration</key>
<value>
<name>wait-penalty-duration</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>release-signal-id</key>
<value>NotifyGeographicalcatTruncated</value>
</entry>
<entry>
<key>target-signal-count</key>
<value>1</value>
</entry>
<entry>
<key>signal-counter-name</key>
</entry>
<entry>
<key>wait-buffer-count</key>
<value>1</value>
</entry>
<entry>
<key>releasable-flowfile-count</key>
<value>1</value>
</entry>
<entry>
<key>expiration-duration</key>
<value>10 min</value>
</entry>
<entry>
<key>distributed-cache-service</key>
<value>ccdef25c-4ad3-37ac-0000-000000000000</value>
</entry>
<entry>
<key>attribute-copy-mode</key>
<value>keeporiginal</value>
</entry>
<entry>
<key>wait-mode</key>
<value>keep</value>
</entry>
<entry>
<key>wait-penalty-duration</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_WaitTruncate</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>expired</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>wait</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.Wait</type>
</processors>
<processors>
<id>6200e074-aa50-36e9-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>208.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Mode</key>
<value>
<name>Mode</name>
</value>
</entry>
<entry>
<key>Compression Format</key>
<value>
<name>Compression Format</name>
</value>
</entry>
<entry>
<key>Compression Level</key>
<value>
<name>Compression Level</name>
</value>
</entry>
<entry>
<key>Update Filename</key>
<value>
<name>Update Filename</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Mode</key>
<value>compress</value>
</entry>
<entry>
<key>Compression Format</key>
<value>gzip</value>
</entry>
<entry>
<key>Compression Level</key>
<value>9</value>
</entry>
<entry>
<key>Update Filename</key>
<value>true</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>CompressContent</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.CompressContent</type>
</processors>
<processors>
<id>62e39b15-77da-3fa5-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>1264.0</x>
<y>40.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Database Connection Pooling Service</key>
<value>
<identifiesControllerService>org.apache.nifi.dbcp.DBCPService</identifiesControllerService>
<name>Database Connection Pooling Service</name>
</value>
</entry>
<entry>
<key>sql-pre-query</key>
<value>
<name>sql-pre-query</name>
</value>
</entry>
<entry>
<key>SQL select query</key>
<value>
<name>SQL select query</name>
</value>
</entry>
<entry>
<key>sql-post-query</key>
<value>
<name>sql-post-query</name>
</value>
</entry>
<entry>
<key>Max Wait Time</key>
<value>
<name>Max Wait Time</name>
</value>
</entry>
<entry>
<key>dbf-normalize</key>
<value>
<name>dbf-normalize</name>
</value>
</entry>
<entry>
<key>dbf-user-logical-types</key>
<value>
<name>dbf-user-logical-types</name>
</value>
</entry>
<entry>
<key>compression-format</key>
<value>
<name>compression-format</name>
</value>
</entry>
<entry>
<key>dbf-default-precision</key>
<value>
<name>dbf-default-precision</name>
</value>
</entry>
<entry>
<key>dbf-default-scale</key>
<value>
<name>dbf-default-scale</name>
</value>
</entry>
<entry>
<key>esql-max-rows</key>
<value>
<name>esql-max-rows</name>
</value>
</entry>
<entry>
<key>esql-output-batch-size</key>
<value>
<name>esql-output-batch-size</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Database Connection Pooling Service</key>
<value>517808ef-0266-3e67-0000-000000000000</value>
</entry>
<entry>
<key>sql-pre-query</key>
<value>truncate table SBST.geographicalcat</value>
</entry>
<entry>
<key>SQL select query</key>
<value>select count(*) as cnt from SBST.geographicalcat</value>
</entry>
<entry>
<key>sql-post-query</key>
</entry>
<entry>
<key>Max Wait Time</key>
<value>0 seconds</value>
</entry>
<entry>
<key>dbf-normalize</key>
<value>false</value>
</entry>
<entry>
<key>dbf-user-logical-types</key>
<value>false</value>
</entry>
<entry>
<key>compression-format</key>
<value>NONE</value>
</entry>
<entry>
<key>dbf-default-precision</key>
<value>10</value>
</entry>
<entry>
<key>dbf-default-scale</key>
<value>0</value>
</entry>
<entry>
<key>esql-max-rows</key>
<value>0</value>
</entry>
<entry>
<key>esql-output-batch-size</key>
<value>0</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_ExecuteSQLTruncate</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.ExecuteSQL</type>
</processors>
<processors>
<id>b8fa507a-797c-3714-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>8.0</x>
<y>0.0</y>
</position>
<bundle>
<artifact>nifi-update-attribute-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Delete Attributes Expression</key>
<value>
<name>Delete Attributes Expression</name>
</value>
</entry>
<entry>
<key>Store State</key>
<value>
<name>Store State</name>
</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
<value>
<name>Stateful Variables Initial Value</name>
</value>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>
<name>canonical-value-lookup-cache-size</name>
</value>
</entry>
<entry>
<key>filename</key>
<value>
<name>filename</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Delete Attributes Expression</key>
</entry>
<entry>
<key>Store State</key>
<value>Do not store state</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>100</value>
</entry>
<entry>
<key>filename</key>
<value>${filename:prepend("_"):prepend(${now():format("yyyy-MM-dd-HH-mm-ss")})}</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>UpdateAttribute</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.attributes.UpdateAttribute</type>
</processors>
<processors>
<id>d24ba22b-88ad-34ae-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>1264.0</x>
<y>264.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>release-signal-id</key>
<value>
<name>release-signal-id</name>
</value>
</entry>
<entry>
<key>signal-counter-name</key>
<value>
<name>signal-counter-name</name>
</value>
</entry>
<entry>
<key>signal-counter-delta</key>
<value>
<name>signal-counter-delta</name>
</value>
</entry>
<entry>
<key>signal-buffer-count</key>
<value>
<name>signal-buffer-count</name>
</value>
</entry>
<entry>
<key>distributed-cache-service</key>
<value>
<identifiesControllerService>org.apache.nifi.distributed.cache.client.AtomicDistributedMapCacheClient</identifiesControllerService>
<name>distributed-cache-service</name>
</value>
</entry>
<entry>
<key>attribute-cache-regex</key>
<value>
<name>attribute-cache-regex</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>release-signal-id</key>
<value>NotifyGeographicalcatTruncated</value>
</entry>
<entry>
<key>signal-counter-name</key>
<value>default</value>
</entry>
<entry>
<key>signal-counter-delta</key>
<value>1</value>
</entry>
<entry>
<key>signal-buffer-count</key>
<value>1</value>
</entry>
<entry>
<key>distributed-cache-service</key>
<value>ccdef25c-4ad3-37ac-0000-000000000000</value>
</entry>
<entry>
<key>attribute-cache-regex</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_NotifyTruncated</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.Notify</type>
</processors>
<processors>
<id>d741151d-7167-3026-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>624.0</x>
<y>464.0</y>
</position>
<bundle>
<artifact>nifi-update-attribute-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Delete Attributes Expression</key>
<value>
<name>Delete Attributes Expression</name>
</value>
</entry>
<entry>
<key>Store State</key>
<value>
<name>Store State</name>
</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
<value>
<name>Stateful Variables Initial Value</name>
</value>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>
<name>canonical-value-lookup-cache-size</name>
</value>
</entry>
<entry>
<key>schema.name</key>
<value>
<name>schema.name</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Delete Attributes Expression</key>
</entry>
<entry>
<key>Store State</key>
<value>Do not store state</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>100</value>
</entry>
<entry>
<key>schema.name</key>
<value>geographicalcat</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_UpdateAttributeSchemaName</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.attributes.UpdateAttribute</type>
</processors>
<processors>
<id>f1513fa5-c26a-3157-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>624.0</x>
<y>32.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Input Directory</key>
<value>
<name>Input Directory</name>
</value>
</entry>
<entry>
<key>File Filter</key>
<value>
<name>File Filter</name>
</value>
</entry>
<entry>
<key>Path Filter</key>
<value>
<name>Path Filter</name>
</value>
</entry>
<entry>
<key>Batch Size</key>
<value>
<name>Batch Size</name>
</value>
</entry>
<entry>
<key>Keep Source File</key>
<value>
<name>Keep Source File</name>
</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>
<name>Recurse Subdirectories</name>
</value>
</entry>
<entry>
<key>Polling Interval</key>
<value>
<name>Polling Interval</name>
</value>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>
<name>Ignore Hidden Files</name>
</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>
<name>Minimum File Age</name>
</value>
</entry>
<entry>
<key>Maximum File Age</key>
<value>
<name>Maximum File Age</name>
</value>
</entry>
<entry>
<key>Minimum File Size</key>
<value>
<name>Minimum File Size</name>
</value>
</entry>
<entry>
<key>Maximum File Size</key>
<value>
<name>Maximum File Size</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Input Directory</key>
<value>${FileSourceFolder}</value>
</entry>
<entry>
<key>File Filter</key>
<value>GeographicalCat.dat</value>
</entry>
<entry>
<key>Path Filter</key>
</entry>
<entry>
<key>Batch Size</key>
<value>10</value>
</entry>
<entry>
<key>Keep Source File</key>
<value>false</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>false</value>
</entry>
<entry>
<key>Polling Interval</key>
<value>0 sec</value>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>true</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>60 sec</value>
</entry>
<entry>
<key>Maximum File Age</key>
</entry>
<entry>
<key>Minimum File Size</key>
<value>0 B</value>
</entry>
<entry>
<key>Maximum File Size</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>10 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>geographicalcat_GetFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.GetFile</type>
</processors>
<processors>
<id>f4b41b47-34d9-3bb4-0000-000000000000</id>
<parentGroupId>bbf1ef49-f963-37ab-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>456.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Directory</key>
<value>
<name>Directory</name>
</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>
<name>Conflict Resolution Strategy</name>
</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>
<name>Create Missing Directories</name>
</value>
</entry>
<entry>
<key>Maximum File Count</key>
<value>
<name>Maximum File Count</name>
</value>
</entry>
<entry>
<key>Last Modified Time</key>
<value>
<name>Last Modified Time</name>
</value>
</entry>
<entry>
<key>Permissions</key>
<value>
<name>Permissions</name>
</value>
</entry>
<entry>
<key>Owner</key>
<value>
<name>Owner</name>
</value>
</entry>
<entry>
<key>Group</key>
<value>
<name>Group</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Directory</key>
<value>${FileArchiveFolder}/ref</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>fail</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>true</value>
</entry>
<entry>
<key>Maximum File Count</key>
</entry>
<entry>
<key>Last Modified Time</key>
</entry>
<entry>
<key>Permissions</key>
</entry>
<entry>
<key>Owner</key>
</entry>
<entry>
<key>Group</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>PutFile</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.PutFile</type>
</processors>
</snippet>
<timestamp>11/29/2019 08:00:19 HKT</timestamp>
</template>
... View more
11-24-2019
11:41 PM
Hi, What if I want to delete all attributes present in FlowFile except, id and CompanyName?
... View more
11-20-2019
02:56 AM
Hello @rohinimathursjc The best way to get the list of Hive external tables and Managed tables is using hive metastore. yes by applying some quires on hive metastore tables we can divide both Hive external tables and Managed tables . please follow the below link to get the queries https://askdoubts.com/question/list-of-hive-internal-tables-from-metadata/#comment-20
... View more
10-23-2019
05:28 AM
Hi @Gerva, You need to have below properties set to be able to use Hive ACID functionality: hive>set hive.support.concurrency = true;
hive>set hive.enforce.bucketing = true;
hive>set hive.exec.dynamic.partition.mode = nonstrict;
hive>set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
hive>set hive.compactor.initiator.on = true;
hive>set hive.compactor.worker.threads = a positive number on at least one instance of the Thrift metastore service;
... View more
10-22-2019
11:04 AM
Any help on this is a great help for us as we are unable to route the records based on substring value on the flow file content
... View more
10-18-2019
07:24 PM
We can store state to DistributeMapCache/HDFS/Hive/Hbase and pullled the stored state and use it in ExecuteSQL processor to incrementally pull the data. Try with this approach mentioned in this link. - In QueryDatabaseTable processor we need to configure max value columns so that nifi stores the state in to QueryDatabaseTable processor. In next run QueryDatabaseTable pulls only the latest changes from the table.
... View more
10-16-2019
05:07 AM
@Shu_ashu Thank you for the solution. I have got the issue resolved as it is working as expected.
... View more
10-14-2019
01:13 AM
@Shu_ashu Great!. With the instructions you have given me, the output file is created correctly. I also tried the option: CREATE TABLE AS scenariox_out AS SELECT select count (*) from scenariox; And the output file was created in: /user/hive/warehouse/scenariox.db/scenariox/scenariox_out/000000_0 Thank you. You have a good day
... View more
10-02-2019
06:03 PM
Hi All, I am doing a similar POC. We are trying to implement batch ingestion using NiFi. Problem statement: I am preparing a SQL query and put the query in the processor using REST API. But, I want to stop the processor group after the ingestion is done because for the next run I will put the query with update CDC column value. Can someone please help me with the logic of stopping the processor after the ingestion is done, because to modify the processor, it has to be stopped
... View more
09-25-2019
03:44 AM
Hi Vijay, Sorry, haven't able to nail down the cause yet, but can you collect EXPLAIN EXTENDED of the query and share the output as attachment to the post? EXPLAIN EXTENDED select * from tdb1.t1 where bs1_dt=2017-06-23; EXPLAIN EXTENDED select * from tdb1.t1; I would like to check how HS2 does the query plan and see if there is any clue. Cheers Eric
... View more
09-21-2019
10:04 AM
@budati For this case define your avro schema(with one field) to read incoming flowfile with some delimiter that doesn't exist in flowfile. So that whole row will be read as string then we can filter out the records by using not like (or) using regex operator in apache calicite. Select * from flowfile where col1 not like 'SKIP' Now output flowfile will not having any records that have SKIP in them and this solution will work dynamically for any number of columns.
... View more
09-19-2019
07:11 AM
1 Kudo
@budati i don't think there is a way to combine all 3 processors into one. We still need to use ExecuteSQL -> ConvertAvroToJson -> EvaluateJsonPath to extract the values from the flowfile. If the answer was helpful to resolve your issue, Accept the answer to close the thread 🙂
... View more
09-10-2019
12:27 PM
I need to zip 3 files from S3, so will the flow be like 3 FetchS3Object processors ( for 3 files ) → Merge Content → PutS3Object ?
... View more
09-10-2019
07:59 AM
1 Kudo
Found the solution! Just needed to apply the regex (?:"b":.\w+?)(\d+) thanks to @Shu_ashu
... View more
09-09-2019
09:37 PM
@Rohit1981 Try using ValidateRecord processor as described in this article with Strict Type Checking is set to "true" and to parse fixed width file refer to this link for detailed explanation for parsing.
... View more
09-09-2019
09:14 PM
@ANMAR Try with this regex in ExtractText processor. (?:"x":.\w+?)(\d+) This regex will extract only the digit in "x" key and adds that value for "y" key in ReplaceText processor.
... View more
09-09-2019
07:54 PM
@ANMAR Try with this regex in ExtractText processor. (?:"x":.\w+?)(\d+) This regex will extract only the digit in "x" key and adds that value for "y" key in ReplaceText processor.
... View more
09-05-2019
08:34 AM
Hi, Exit code 143 is related to Memory/GC issues. Your default Mapper/reducer memory setting may not be sufficient to run the large data set. Thus, try setting up higher AM, MAP and REDUCER memory when a large yarn job is invoked. For more please refer to this link. https://stackoverflow.com/questions/42972908/container-killed-by-the-applicationmaster-exit-code-is-143 Thanks AKR
... View more
09-04-2019
07:15 AM
I think that QueryDatabaseTable processor doesn't support fetching updated data, you can use CaptureDataChange Processor instead!
... View more
