Member since
08-04-2018
71
Posts
4
Kudos Received
0
Solutions
04-01-2020
09:31 AM
Hi guys,
I am using docker image for Spark: srcd/jupyter-spark (https://github.com/src-d/jupyter-spark-docker)
then I try to add packages:
--packages com.databricks:spark-avro_2.11:4.0.0
--packages com.crealytics:spark-excel_2.11:0.11.1
or copy --jar files
--jar hadoop-aws-2.6.0.jar
--jar spark-avro_2.11-4.0.0.jar
--jar spark-excel_2.11-0.8.3.jar
there is not any errors when I add, but I are a lot of error when I run script
like when I try to save spark dataframe to avro format
sdf.write.format("avro").mode("overwrite").save(avro_file_location)
Py4JJavaError : An error occurred while calling o264.save.
: org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;
or when I try to connect spark to S3
hadoopConf = sc._jsc.hadoopConfiguration()
hadoopConf.set('fs.s3.awsAccessKeyId', AWS_KEY) hadoopConf.set('fs.s3.awsSecretAccessKey', AWS_SECRET)
sdf = sql.read.format("csv").option("header", "true").load(s3_file)
Py4JJavaError : An error occurred while calling o231.load.
: java.io.IOException: No FileSystem for scheme: s3n
Do you have any suggestions?
Many thanks
... View more
Labels:
03-25-2020
02:34 PM
hi guys,
I done convert word to pdf files by python, but never try convert word to pdf by Apache Spark?
May you give a suggestion for it?
Many thanks,
... View more
- Tags:
- Spark
Labels:
03-10-2020
10:38 AM
kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
'JavaPackage' object is not callable Traceback (most recent call last): File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 78, in createStream helper = KafkaUtils._get_helper(ssc._sc) File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 217, in _get_helper return sc._jvm.org.apache.spark.streaming.kafka.KafkaUtilsPythonHelper() TypeError: 'JavaPackage' object is not callable
Any suggestions?
... View more
Labels:
03-05-2020
12:42 PM
Hi @GangWar , Thanks for your sharing, how ever I tried with image apache/nifi:latest https://hub.docker.com/r/apache/nifi/ My expectation is mount folder /opt/nifi/nifi-current/conf (inside nifi-container) to folder nifi (in docker-machine). However it doesnot work. Do you have any suggestions?
... View more
03-05-2020
11:16 AM
Hi guys,
Do you know How to set up Apache Nifi: Global variables?
so we can change the value of variables in each component like add running count for a record
Many thanks
... View more
- Tags:
- NiFi
Labels:
03-05-2020
11:13 AM
hi guys,
Any suggestions to install Install Impala on AWS EMR?
Many thanks
... View more
Labels:
02-27-2020
08:41 AM
hi all,
in general, it is very easy to mount folders from local/host to container
Any suggestions to mount folder inside container to local/host like
-v /path_folder_in_container/:/path_folder_in_local/
or
-v /path_folder_in_container/:/path_folder_in_host/
Many thanks
... View more
Labels:
02-20-2020
10:41 AM
hi all,
I have spark dataframe like
+------+--------------------+
|active| email|
+------+--------------------+
| 1|jared.ely@sakilac...|
| 1|mary.smith@sakila...|
| 1|patricia.johnson@...|
| 1|linda.williams@sa...|
| 1|barbara.jones@sak...|
| 1|elizabeth.brown@s...|
| 1|jennifer.davis@sa...|
| 1|maria.miller@saki...|
| 1|susan.wilson@saki...|
| 1|margaret.moore@sa...|
| 1|dorothy.taylor@sa...|
| 1|lisa.anderson@sak...|
| 1|nancy.thomas@saki...|
| 1|karen.jackson@sak...|
| 1|betty.white@sakil...|
| 1|helen.harris@saki...|
| 0|sandra.martin@sak...|
| 1|donna.thompson@sa...|
| 1|carol.garcia@saki...|
| 1|ruth.martinez@sak...|
+------+--------------------+
requirement: if active== 1 then the corresponding email will change to @gmail.com
for example row 1: active = 1 then email= jared.ely@gmail.com
What is bet best way for transformation this case, using spark transformation?
Any suggestions will appreciate
... View more
Labels:
02-17-2020
05:27 PM
thanks @MattWho
... View more
02-14-2020
12:01 PM
Nifi 1.8.0 ConvertJSONToAvro Please check again, many thanks
... View more
02-14-2020
11:41 AM
Cannot import data to postgresSQL (SAME issue with PutDatabaseRecord and PutSQL )
due to Batch entry 0 INSERT INTO customer (customer_id, store_id, first_name, last_name, email, address_id, activebool, create_date, last_update, active) VALUES (1,1,'Mary','Smith','mary.smith@sakilacustomer.org',5,'TRUE','2006-02-14','2013-05-26 14:49:45',1) was aborted: ERROR: column "create_date" is of type date but expression is of type character varying
Any suggestions? (xml file below)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.3"> <description></description> <groupId>a3d73b93-5247-1def-509e-14cd1ac69742</groupId> <name>CSV to Postgresql</name> <snippet> <connections> <id>4fc6d8b8-bd61-3313-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>4fc6d8b8-bd61-3313-98fc-59d5256a421e</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>7cc91796-c792-3b07-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>7cc91796-c792-3b07-ae1d-8fc97ad5b74a</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>ddd0dff0-76dd-3427-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>ddd0dff0-76dd-3427-852f-d4fa13e9ec46</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>6c54eacd-143c-33fa-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>6c54eacd-143c-33fa-aebc-f9e61ce024f2</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>4632754b-e43b-3b79-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>031921be-9955-326b-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>031921be-9955-326b-b891-5e1c9d5536f4</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>834c6890-31d2-3aed-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>834c6890-31d2-3aed-b6bc-c740e496ce94</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>031921be-9955-326b-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>031921be-9955-326b-b891-5e1c9d5536f4</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>98af5b16-4035-379a-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>98af5b16-4035-379a-ab1c-c5e5d95aabdb</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>8f478363-3718-3893-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>8f478363-3718-3893-9c04-2986d3d46720</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>ddd0dff0-76dd-3427-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>ddd0dff0-76dd-3427-852f-d4fa13e9ec46</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>4632754b-e43b-3b79-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>aa3bba4a-8b32-3ca5-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>aa3bba4a-8b32-3ca5-888b-d616b2a53d7c</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>7cc91796-c792-3b07-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>7cc91796-c792-3b07-ae1d-8fc97ad5b74a</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>4632754b-e43b-3b79-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>cf43fc03-eac8-3164-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>cf43fc03-eac8-3164-8079-e434ae2839bc</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>9f35d6a6-94ae-3849-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>9f35d6a6-94ae-3849-85b7-6115a9fc87fe</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>ddd0dff0-76dd-3427-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>ddd0dff0-76dd-3427-852f-d4fa13e9ec46</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>eaba1dbc-b2a8-30be-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>eaba1dbc-b2a8-30be-afb1-440bdbfba52b</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>470.94140625</x> <y>357.442138671875</y> </bends> <bends> <x>526.94140625</x> <y>429.442138671875</y> </bends> <destination> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>4632754b-e43b-3b79-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>retry</selectedRelationships> <source> <groupId>bcdb9e55-19ce-3ec3-0000-000000000000</groupId> <id>4632754b-e43b-3b79-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <controllerServices> <id>1390d92e-bc8d-3ccd-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>1390d92e-bc8d-3ccd-86ff-93bb5fd79fc1</versionedComponentId> <bundle> <artifact>nifi-registry-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>avro-reg-validated-field-names</key> <value> <name>avro-reg-validated-field-names</name> </value> </entry> <entry> <key>customer</key> <value> <name>customer</name> </value> </entry> </descriptors> <name>potgresCustomerWithoutheader -- AvroSchemaRegistry</name> <persistsState>false</persistsState> <properties> <entry> <key>avro-reg-validated-field-names</key> </entry> <entry> <key>customer</key> <value>{ "type":"record", "name":"customer", "namespace":"nifi", "fields":[ {"name":"customer_id","type":["null","int"]}, {"name":"store_id","type":["null","int"]}, {"name":"first_name","type":["null","string"]}, {"name":"last_name","type":["null","string"]}, {"name":"email","type":["null","string"]}, {"name":"address_id","type":["null","int"]}, {"name":"activebool","type":["null","boolean"]}, {"name":"create_date","type":"string"}, {"name":"last_update","type":["null","string"]}, {"name":"active","type":["null","int"]} ]}</value> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.schemaregistry.services.AvroSchemaRegistry</type> </controllerServices> <controllerServices> <id>868992b5-01dc-3f11-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <versionedComponentId>868992b5-01dc-3f11-bb43-23b61ed20a90</versionedComponentId> <bundle> <artifact>nifi-dbcp-service-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>Database Connection URL</key> <value> <name>Database Connection URL</name> </value> </entry> <entry> <key>Database Driver Class Name</key> <value> <name>Database Driver Class Name</name> </value> </entry> <entry> <key>database-driver-locations</key> <value> <name>database-driver-locations</name> </value> </entry> <entry> <key>kerberos-credentials-service</key> <value> <identifiesControllerService>org.apache.nifi.kerberos.KerberosCredentialsService</identifiesControllerService> <name>kerberos-credentials-service</name> </value> </entry> <entry> <key>Database User</key> <value> <name>Database User</name> </value> </entry> <entry> <key>Password</key> <value> <name>Password</name> </value> </entry> <entry> <key>Max Wait Time</key> <value> <name>Max Wait Time</name> </value> </entry> <entry> <key>Max Total Connections</key> <value> <name>Max Total Connections</name> </value> </entry> <entry> <key>Validation-query</key> <value> <name>Validation-query</name> </value> </entry> <entry> <key>dbcp-min-idle-conns</key> <value> <name>dbcp-min-idle-conns</name> </value> </entry> <entry> <key>dbcp-max-idle-conns</key> <value> <name>dbcp-max-idle-conns</name> </value> </entry> <entry> <key>dbcp-max-conn-lifetime</key> <value> <name>dbcp-max-conn-lifetime</name> </value> </entry> <entry> <key>dbcp-time-between-eviction-runs</key> <value> <name>dbcp-time-between-eviction-runs</name> </value> </entry> <entry> <key>dbcp-min-evictable-idle-time</key> <value> <name>dbcp-min-evictable-idle-time</name> </value> </entry> <entry> <key>dbcp-soft-min-evictable-idle-time</key> <value> <name>dbcp-soft-min-evictable-idle-time</name> </value> </entry> </descriptors> <name>postgres -- DBCPConnectionPool</name> <persistsState>false</persistsState> <properties> <entry> <key>Database Connection URL</key> <value>jdbc:postgresql://postgresdb.ckdsaqpaaule.us-east-1.rds.amazonaws.com:5432/thuyle_dvdrental</value> </entry> <entry> <key>Database Driver Class Name</key> <value>org.postgresql.Driver</value> </entry> <entry> <key>database-driver-locations</key> <value>/home/nifi/jdbc-drivers/postgresql-42.2.6.jar</value> </entry> <entry> <key>kerberos-credentials-service</key> </entry> <entry> <key>Database User</key> <value>postgresadmin</value> </entry> <entry> <key>Password</key> </entry> <entry> <key>Max Wait Time</key> </entry> <entry> <key>Max Total Connections</key> </entry> <entry> <key>Validation-query</key> </entry> <entry> <key>dbcp-min-idle-conns</key> </entry> <entry> <key>dbcp-max-idle-conns</key> </entry> <entry> <key>dbcp-max-conn-lifetime</key> </entry> <entry> <key>dbcp-time-between-eviction-runs</key> </entry> <entry> <key>dbcp-min-evictable-idle-time</key> </entry> <entry> <key>dbcp-soft-min-evictable-idle-time</key> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.dbcp.DBCPConnectionPool</type> </controllerServices> <controllerServices> <id>e6a5d1b7-12c8-3106-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <bundle> <artifact>nifi-record-serialization-services-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>schema-access-strategy</key> <value> <name>schema-access-strategy</name> </value> </entry> <entry> <key>schema-registry</key> <value> <identifiesControllerService>org.apache.nifi.schemaregistry.services.SchemaRegistry</identifiesControllerService> <name>schema-registry</name> </value> </entry> <entry> <key>schema-name</key> <value> <name>schema-name</name> </value> </entry> <entry> <key>schema-version</key> <value> <name>schema-version</name> </value> </entry> <entry> <key>schema-branch</key> <value> <name>schema-branch</name> </value> </entry> <entry> <key>schema-text</key> <value> <name>schema-text</name> </value> </entry> <entry> <key>csv-reader-csv-parser</key> <value> <name>csv-reader-csv-parser</name> </value> </entry> <entry> <key>Date Format</key> <value> <name>Date Format</name> </value> </entry> <entry> <key>Time Format</key> <value> <name>Time Format</name> </value> </entry> <entry> <key>Timestamp Format</key> <value> <name>Timestamp Format</name> </value> </entry> <entry> <key>CSV Format</key> <value> <name>CSV Format</name> </value> </entry> <entry> <key>Value Separator</key> <value> <name>Value Separator</name> </value> </entry> <entry> <key>Skip Header Line</key> <value> <name>Skip Header Line</name> </value> </entry> <entry> <key>ignore-csv-header</key> <value> <name>ignore-csv-header</name> </value> </entry> <entry> <key>Quote Character</key> <value> <name>Quote Character</name> </value> </entry> <entry> <key>Escape Character</key> <value> <name>Escape Character</name> </value> </entry> <entry> <key>Comment Marker</key> <value> <name>Comment Marker</name> </value> </entry> <entry> <key>Null String</key> <value> <name>Null String</name> </value> </entry> <entry> <key>Trim Fields</key> <value> <name>Trim Fields</name> </value> </entry> <entry> <key>csvutils-character-set</key> <value> <name>csvutils-character-set</name> </value> </entry> </descriptors> <name>potgresCustomerWithoutheader -- CSVReader</name> <persistsState>false</persistsState> <properties> <entry> <key>schema-access-strategy</key> <value>schema-name</value> </entry> <entry> <key>schema-registry</key> <value>1390d92e-bc8d-3ccd-0000-000000000000</value> </entry> <entry> <key>schema-name</key> </entry> <entry> <key>schema-version</key> </entry> <entry> <key>schema-branch</key> </entry> <entry> <key>schema-text</key> </entry> <entry> <key>csv-reader-csv-parser</key> </entry> <entry> <key>Date Format</key> <value>yyyy-MM-dd</value> </entry> <entry> <key>Time Format</key> <value>HH:mm:ss</value> </entry> <entry> <key>Timestamp Format</key> <value>yyyy-MM-dd HH:mm:ss</value> </entry> <entry> <key>CSV Format</key> </entry> <entry> <key>Value Separator</key> </entry> <entry> <key>Skip Header Line</key> </entry> <entry> <key>ignore-csv-header</key> </entry> <entry> <key>Quote Character</key> </entry> <entry> <key>Escape Character</key> </entry> <entry> <key>Comment Marker</key> </entry> <entry> <key>Null String</key> </entry> <entry> <key>Trim Fields</key> </entry> <entry> <key>csvutils-character-set</key> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.csv.CSVReader</type> </controllerServices> <processors> <id>031921be-9955-326b-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>8.0</x> <y>224.0</y> </position> <versionedComponentId>031921be-9955-326b-b891-5e1c9d5536f4</versionedComponentId> <bundle> <artifact>nifi-update-attribute-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Delete Attributes Expression</key> <value> <name>Delete Attributes Expression</name> </value> </entry> <entry> <key>Store State</key> <value> <name>Store State</name> </value> </entry> <entry> <key>Stateful Variables Initial Value</key> <value> <name>Stateful Variables Initial Value</name> </value> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value> <name>canonical-value-lookup-cache-size</name> </value> </entry> <entry> <key>error_file</key> <value> <name>error_file</name> </value> </entry> <entry> <key>received_date</key> <value> <name>received_date</name> </value> </entry> <entry> <key>running_count</key> <value> <name>running_count</name> </value> </entry> <entry> <key>schema.name</key> <value> <name>schema.name</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Delete Attributes Expression</key> </entry> <entry> <key>Store State</key> <value>Do not store state</value> </entry> <entry> <key>Stateful Variables Initial Value</key> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value>100</value> </entry> <entry> <key>error_file</key> <value>/home/nifi/error_customer.csv</value> </entry> <entry> <key>received_date</key> <value>${now():format("yyyy-MM-dd HH:mm:ss.SSSS")}</value> </entry> <entry> <key>running_count</key> <value>0</value> </entry> <entry> <key>schema.name</key> <value>customer</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>UpdateAttribute</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>RUNNING</state> <style/> <type>org.apache.nifi.processors.attributes.UpdateAttribute</type> </processors> <processors> <id>4632754b-e43b-3b79-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>448.0</y> </position> <versionedComponentId>4632754b-e43b-3b79-b472-df88a152ce2e</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>put-db-record-record-reader</key> <value> <identifiesControllerService>org.apache.nifi.serialization.RecordReaderFactory</identifiesControllerService> <name>put-db-record-record-reader</name> </value> </entry> <entry> <key>put-db-record-statement-type</key> <value> <name>put-db-record-statement-type</name> </value> </entry> <entry> <key>put-db-record-dcbp-service</key> <value> <identifiesControllerService>org.apache.nifi.dbcp.DBCPService</identifiesControllerService> <name>put-db-record-dcbp-service</name> </value> </entry> <entry> <key>put-db-record-catalog-name</key> <value> <name>put-db-record-catalog-name</name> </value> </entry> <entry> <key>put-db-record-schema-name</key> <value> <name>put-db-record-schema-name</name> </value> </entry> <entry> <key>put-db-record-table-name</key> <value> <name>put-db-record-table-name</name> </value> </entry> <entry> <key>put-db-record-translate-field-names</key> <value> <name>put-db-record-translate-field-names</name> </value> </entry> <entry> <key>put-db-record-unmatched-field-behavior</key> <value> <name>put-db-record-unmatched-field-behavior</name> </value> </entry> <entry> <key>put-db-record-unmatched-column-behavior</key> <value> <name>put-db-record-unmatched-column-behavior</name> </value> </entry> <entry> <key>put-db-record-update-keys</key> <value> <name>put-db-record-update-keys</name> </value> </entry> <entry> <key>put-db-record-field-containing-sql</key> <value> <name>put-db-record-field-containing-sql</name> </value> </entry> <entry> <key>put-db-record-allow-multiple-statements</key> <value> <name>put-db-record-allow-multiple-statements</name> </value> </entry> <entry> <key>put-db-record-quoted-identifiers</key> <value> <name>put-db-record-quoted-identifiers</name> </value> </entry> <entry> <key>put-db-record-quoted-table-identifiers</key> <value> <name>put-db-record-quoted-table-identifiers</name> </value> </entry> <entry> <key>put-db-record-query-timeout</key> <value> <name>put-db-record-query-timeout</name> </value> </entry> <entry> <key>rollback-on-failure</key> <value> <name>rollback-on-failure</name> </value> </entry> <entry> <key>table-schema-cache-size</key> <value> <name>table-schema-cache-size</name> </value> </entry> <entry> <key>put-db-record-max-batch-size</key> <value> <name>put-db-record-max-batch-size</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>put-db-record-record-reader</key> <value>e6a5d1b7-12c8-3106-0000-000000000000</value> </entry> <entry> <key>put-db-record-statement-type</key> <value>INSERT</value> </entry> <entry> <key>put-db-record-dcbp-service</key> <value>868992b5-01dc-3f11-0000-000000000000</value> </entry> <entry> <key>put-db-record-catalog-name</key> </entry> <entry> <key>put-db-record-schema-name</key> </entry> <entry> <key>put-db-record-table-name</key> <value>customer</value> </entry> <entry> <key>put-db-record-translate-field-names</key> <value>true</value> </entry> <entry> <key>put-db-record-unmatched-field-behavior</key> <value>Ignore Unmatched Fields</value> </entry> <entry> <key>put-db-record-unmatched-column-behavior</key> <value>Fail on Unmatched Columns</value> </entry> <entry> <key>put-db-record-update-keys</key> </entry> <entry> <key>put-db-record-field-containing-sql</key> </entry> <entry> <key>put-db-record-allow-multiple-statements</key> <value>false</value> </entry> <entry> <key>put-db-record-quoted-identifiers</key> <value>false</value> </entry> <entry> <key>put-db-record-quoted-table-identifiers</key> <value>false</value> </entry> <entry> <key>put-db-record-query-timeout</key> <value>0 seconds</value> </entry> <entry> <key>rollback-on-failure</key> <value>false</value> </entry> <entry> <key>table-schema-cache-size</key> <value>100</value> </entry> <entry> <key>put-db-record-max-batch-size</key> <value>0</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Postgres-- PutDatabaseRecord</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>retry</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>RUNNING</state> <style/> <type>org.apache.nifi.processors.standard.PutDatabaseRecord</type> </processors> <processors> <id>7cc91796-c792-3b07-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>672.0</y> </position> <versionedComponentId>7cc91796-c792-3b07-ae1d-8fc97ad5b74a</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Success-Log</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> <processors> <id>98af5b16-4035-379a-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>8.0</x> <y>0.0</y> </position> <versionedComponentId>98af5b16-4035-379a-ab1c-c5e5d95aabdb</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>File Size</key> <value> <name>File Size</name> </value> </entry> <entry> <key>Batch Size</key> <value> <name>Batch Size</name> </value> </entry> <entry> <key>Data Format</key> <value> <name>Data Format</name> </value> </entry> <entry> <key>Unique FlowFiles</key> <value> <name>Unique FlowFiles</name> </value> </entry> <entry> <key>generate-ff-custom-text</key> <value> <name>generate-ff-custom-text</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>File Size</key> <value>0B</value> </entry> <entry> <key>Batch Size</key> <value>1</value> </entry> <entry> <key>Data Format</key> <value>Text</value> </entry> <entry> <key>Unique FlowFiles</key> <value>false</value> </entry> <entry> <key>generate-ff-custom-text</key> <value>1,1,Mary,Smith,mary.smith@sakilacustomer.org,5,true,2006-02-14,2013-05-26 14:49:45,1 2,1,Patricia,Johnson,patricia.johnson@sakilacustomer.org,6,true,2006-02-14,2013-05-26 14:49:45,1 3,1,Linda,Williams,linda.williams@sakilacustomer.org,7,true,2006-02-14,2013-05-26 14:49:45,1 4,2,Barbara,Jones,barbara.jones@sakilacustomer.org,8,true,2006-02-14,2013-05-26 14:49:45,1 5,1,Elizabeth,Brown,elizabeth.brown@sakilacustomer.org,9,true,2006-02-14,2013-05-26 14:49:45,1 6,2,Jennifer,Davis,jennifer.davis@sakilacustomer.org,10,true,2006-02-14,2013-05-26 14:49:45,1 7,1,Maria,Miller,maria.miller@sakilacustomer.org,11,true,2006-02-14,2013-05-26 14:49:45,1 8,2,Susan,Wilson,susan.wilson@sakilacustomer.org,12,true,2006-02-14,2013-05-26 14:49:45,1 9,2,Margaret,Moore,margaret.moore@sakilacustomer.org,13,true,2006-02-14,2013-05-26 14:49:45,1 10,1,Dorothy,Taylor,dorothy.taylor@sakilacustomer.org,14,true,2006-02-14,2013-05-26 14:49:45,1 11,2,Lisa,Anderson,lisa.anderson@sakilacustomer.org,15,true,2006-02-14,2013-05-26 14:49:45,1 12,1,Nancy,Thomas,nancy.thomas@sakilacustomer.org,16,true,2006-02-14,2013-05-26 14:49:45,1 13,2,Karen,Jackson,karen.jackson@sakilacustomer.org,17,true,2006-02-14,2013-05-26 14:49:45,1 14,2,Betty,White,betty.white@sakilacustomer.org,18,true,2006-02-14,2013-05-26 14:49:45,1 15,1,Helen,Harris,helen.harris@sakilacustomer.org,19,true,2006-02-14,2013-05-26 14:49:45,1 16,2,Sandra,Martin,sandra.martin@sakilacustomer.org,20,true,2006-02-14,2013-05-26 14:49:45,0 17,1,Donna,Thompson,donna.thompson@sakilacustomer.org,21,true,2006-02-14,2013-05-26 14:49:45,1 18,2,Carol,Garcia,carol.garcia@sakilacustomer.org,22,true,2006-02-14,2013-05-26 14:49:45,1 19,1,Ruth,Martinez,ruth.martinez@sakilacustomer.org,23,true,2006-02-14,2013-05-26 14:49:45,1 20,2,Sharon,Robinson,sharon.robinson@sakilacustomer.org,24,true,2006-02-14,2013-05-26 14:49:45,1 21,1,Michelle,Clark,michelle.clark@sakilacustomer.org,25,true,2006-02-14,2013-05-26 14:49:45,1 22,1,Laura,Rodriguez,laura.rodriguez@sakilacustomer.org,26,true,2006-02-14,2013-05-26 14:49:45,1</value> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>10000000000 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>without header -- GenerateFlowFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>RUNNING</state> <style/> <type>org.apache.nifi.processors.standard.GenerateFlowFile</type> </processors> <processors> <id>9f35d6a6-94ae-3849-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>616.0</x> <y>224.0</y> </position> <versionedComponentId>9f35d6a6-94ae-3849-85b7-6115a9fc87fe</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Failure-Log</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> <processors> <id>ddd0dff0-76dd-3427-0000-000000000000</id> <parentGroupId>bcdb9e55-19ce-3ec3-0000-000000000000</parentGroupId> <position> <x>616.0</x> <y>456.0</y> </position> <versionedComponentId>ddd0dff0-76dd-3427-852f-d4fa13e9ec46</versionedComponentId> <bundle> <artifact>nifi-scripting-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Script Engine</key> <value> <name>Script Engine</name> </value> </entry> <entry> <key>Script File</key> <value> <name>Script File</name> </value> </entry> <entry> <key>Script Body</key> <value> <name>Script Body</name> </value> </entry> <entry> <key>Module Directory</key> <value> <name>Module Directory</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Script Engine</key> <value>python</value> </entry> <entry> <key>Script File</key> </entry> <entry> <key>Script Body</key> <value>from org.apache.nifi.processors.script import ExecuteScript import subprocess from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import InputStreamCallback import os.path
# Define a subclass of InputStreamCallback for use in session.read() class PyInputStreamCallback(InputStreamCallback): def __init__(self, error_file): self.error_file = error_file pass def process(self, inputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) # Do something with text here if(text != None): fd = open(self.error_file,'a') fd.write(text) fd.write("\n") fd.close() # end class
# implicit return at the end flowFile = session.get() if flowFile is not None: error_file = flowFile.getAttribute('error_file') session.read(flowFile, PyInputStreamCallback(error_file)) session.transfer(flowFile,ExecuteScript.REL_SUCCESS)
</value> </entry> <entry> <key>Module Directory</key> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Write error to log file -- ExecuteScript</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.script.ExecuteScript</type> </processors> </snippet> <timestamp>02/14/2020 19:34:57 UTC</timestamp> </template>
... View more
- Tags:
- NiFi
- PostgreSQL
Labels:
02-13-2020
08:14 AM
PutDatabaseRecord cannot be converted to a timestamp
due to The value of the sql.args.9.value is '2013-05-26 14:49:45', which cannot be converted to a timestamp; routing to failure: java.sql.SQLDataException: The value of the sql.args.9.value is ' 2013-05-26 14:49:45 ', which cannot be converted to a timestamp
similar with https://community.cloudera.com/t5/Support-Questions/PUTSQL-which-cannot-be-converted-to-a-timestamp/m-p/289632#M214382 but used PutDatabaseRecord processor
... View more
Labels:
02-13-2020
08:09 AM
ExecuteScript (Scrip Engine) does not running correct
input { "store_id": 1, "address_id": 86, "last_name": "username_last", "active": 1, "activebool": true, "received_date": "2020-02-13 15:28:52.374000", "last_update": "2013-05-26 14:49:45", "running_count": 1, "customer_id": 82, "create_date": "2006-02-14", "first_name": "username_first", "email": "username_first.username_last@example.org" }
#### ExecuteScript
import json import datetime import java.io from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import StreamCallback from org.apache.nifi.processor.io import InputStreamCallback class PyStreamCallback(StreamCallback): def __init__(self): self.insert_cmd = "" pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = json.loads(text)
table_name = "customer" self.insert_cmd += 'INSERT INTO {} '.format(table_name) self.insert_cmd += '({}) '.format(', '.join(obj.keys())) self.insert_cmd += "VALUES ({});".format(', '.join("'{0}'".format(value) if type(value) == str else str(value) for value in obj.values())) outputStream.write(bytearray(json.dumps(obj, indent=4).encode('utf-8'))) flowFile = session.get() if (flowFile != None): pyStreamCallback = PyStreamCallback() flowFile = session.write(flowFile, pyStreamCallback) flowFile = session.putAttribute(flowFile, 'insert_cmd', str(pyStreamCallback.insert_cmd))
session.transfer(flowFile, REL_SUCCESS) session.commit()
out put
insert_cmd
INSERT INTO customer (store_id, address_id, last_name, active, activebool, received_date, last_update, running_count, customer_id, create_date, first_name, email) VALUES (1, 86, Coleman, 1, true, '2020-02-13 15:28:52.374000', '2013-05-26 14:49:45', 1, 82, '2006-02-14', username_first,username_first.username_last@example.org);
Why nifi dont make quote for boolean value????
... View more
Labels:
02-13-2020
07:25 AM
hi all,
Nifi 1.11.1 cannot find Inferring schema processor, any suggestions?
... View more
- Tags:
- Nifi 1.11.1
Labels:
02-13-2020
07:24 AM
It seems more complicated than using JSONToAttributes or ConvertJSONToAvro processor (NiFi version lower 1.7 are supported)
... View more
02-13-2020
07:15 AM
Any suggestions?
... View more
02-12-2020
12:55 PM
Nifi 1.11.1 cannot find JSONToAttributes and ConvertJSONToAvro,
Any suggestions???
... View more
- Tags:
- custom-processor
- NiFi
Labels:
02-12-2020
07:54 AM
it is the same issue due to The value of the sql.args.9.value is '2013-05-26 14:49:45', which cannot be converted to a timestamp; routing to failure: java.sql.SQLDataException: The value of the sql.args.9.value is '2013-05-26 14:49:45', which cannot be converted to a timestamp <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.3"> <description></description> <groupId>360e8d37-0170-1000-2124-f7af50537a66</groupId> <name>test_PutDatabseRecord</name> <snippet> <connections> <id>4628c95c-b1dc-3fdd-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>7c677559-6b04-31fb-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>7c677559-6b04-31fb-b3f0-f177624fd063</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>29beb3d5-b88e-32df-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>532025c1-8aa9-3e5e-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>532025c1-8aa9-3e5e-b287-a646c6c3b0e1</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>2b305558-f05a-3254-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>2b305558-f05a-3254-a069-72ac003eac04</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>05136c6f-c4b4-3e66-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>05136c6f-c4b4-3e66-b29c-c4da38c83eed</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>5874ecf1-cb6b-3019-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>5874ecf1-cb6b-3019-bb91-3406bdc548ca</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>7c677559-6b04-31fb-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>7c677559-6b04-31fb-b3f0-f177624fd063</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>0</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>05136c6f-c4b4-3e66-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>05136c6f-c4b4-3e66-b29c-c4da38c83eed</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>9a089cff-61b8-33f8-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>9a089cff-61b8-33f8-a5f9-67b717032071</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>05136c6f-c4b4-3e66-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>05136c6f-c4b4-3e66-b29c-c4da38c83eed</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>c9cb38c4-95dd-35a1-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>c9cb38c4-95dd-35a1-8f17-4779ac3bb0cf</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>b223e632-2be6-3b55-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>b223e632-2be6-3b55-8e70-0da7e8a2cf28</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>29beb3d5-b88e-32df-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>e6e49310-06e6-3642-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>e6e49310-06e6-3642-87e1-78f5d613f631</versionedComponentId> </source> <zIndex>0</zIndex> </connections> <connections> <id>e9bda431-226f-3f7f-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>c9cb38c4-95dd-35a1-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>c9cb38c4-95dd-35a1-8f17-4779ac3bb0cf</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>splits</selectedRelationships> <source> <groupId>7ba66444-6b1d-3473-0000-000000000000</groupId> <id>29beb3d5-b88e-32df-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <controllerServices> <id>0b91e500-a5a4-3561-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>bb40b6e1-30fb-3327-972f-507afbc081ce</versionedComponentId> <bundle> <artifact>nifi-registry-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>avro-reg-validated-field-names</key> <value> <name>avro-reg-validated-field-names</name> </value> </entry> <entry> <key>customer</key> <value> <name>customer</name> </value> </entry> </descriptors> <name>customer-AvroSchemaRegistry</name> <persistsState>false</persistsState> <properties> <entry> <key>avro-reg-validated-field-names</key> <value>true</value> </entry> <entry> <key>customer</key> <value>{ "type":"record", "name":"customer", "namespace":"nifi", "fields":[ {"name":"customer_id","type":["null","int"]}, {"name":"store_id","type":["null","int"]}, {"name":"first_name","type":["null","string"]}, {"name":"last_name","type":["null","string"]}, {"name":"email","type":["null","string"]}, {"name":"address_id","type":["null","int"]}, {"name":"activebool","type":["null","boolean"]}, {"name":"create_date","type":["null","string"]}, {"name":"last_update","type":["null","string"], "logicalType":"timestamp-millis"}, {"name":"active","type":["null","int"]}]}</value> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.schemaregistry.services.AvroSchemaRegistry</type> </controllerServices> <controllerServices> <id>23b93123-18d3-3c4b-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>a17a8d5b-3caa-324e-95d4-a2f5a0981073</versionedComponentId> <bundle> <artifact>nifi-record-serialization-services-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>schema-access-strategy</key> <value> <name>schema-access-strategy</name> </value> </entry> <entry> <key>schema-registry</key> <value> <identifiesControllerService>org.apache.nifi.schemaregistry.services.SchemaRegistry</identifiesControllerService> <name>schema-registry</name> </value> </entry> <entry> <key>schema-name</key> <value> <name>schema-name</name> </value> </entry> <entry> <key>schema-version</key> <value> <name>schema-version</name> </value> </entry> <entry> <key>schema-branch</key> <value> <name>schema-branch</name> </value> </entry> <entry> <key>schema-text</key> <value> <name>schema-text</name> </value> </entry> <entry> <key>csv-reader-csv-parser</key> <value> <name>csv-reader-csv-parser</name> </value> </entry> <entry> <key>Date Format</key> <value> <name>Date Format</name> </value> </entry> <entry> <key>Time Format</key> <value> <name>Time Format</name> </value> </entry> <entry> <key>Timestamp Format</key> <value> <name>Timestamp Format</name> </value> </entry> <entry> <key>CSV Format</key> <value> <name>CSV Format</name> </value> </entry> <entry> <key>Value Separator</key> <value> <name>Value Separator</name> </value> </entry> <entry> <key>Skip Header Line</key> <value> <name>Skip Header Line</name> </value> </entry> <entry> <key>ignore-csv-header</key> <value> <name>ignore-csv-header</name> </value> </entry> <entry> <key>Quote Character</key> <value> <name>Quote Character</name> </value> </entry> <entry> <key>Escape Character</key> <value> <name>Escape Character</name> </value> </entry> <entry> <key>Comment Marker</key> <value> <name>Comment Marker</name> </value> </entry> <entry> <key>Null String</key> <value> <name>Null String</name> </value> </entry> <entry> <key>Trim Fields</key> <value> <name>Trim Fields</name> </value> </entry> <entry> <key>csvutils-character-set</key> <value> <name>csvutils-character-set</name> </value> </entry> </descriptors> <name>customer-CSVReader</name> <persistsState>false</persistsState> <properties> <entry> <key>schema-access-strategy</key> <value>schema-name</value> </entry> <entry> <key>schema-registry</key> <value>0b91e500-a5a4-3561-0000-000000000000</value> </entry> <entry> <key>schema-name</key> <value>${schema.name}</value> </entry> <entry> <key>schema-version</key> </entry> <entry> <key>schema-branch</key> </entry> <entry> <key>schema-text</key> <value>${avro.schema}</value> </entry> <entry> <key>csv-reader-csv-parser</key> <value>commons-csv</value> </entry> <entry> <key>Date Format</key> </entry> <entry> <key>Time Format</key> </entry> <entry> <key>Timestamp Format</key> </entry> <entry> <key>CSV Format</key> <value>custom</value> </entry> <entry> <key>Value Separator</key> <value>,</value> </entry> <entry> <key>Skip Header Line</key> <value>true</value> </entry> <entry> <key>ignore-csv-header</key> <value>false</value> </entry> <entry> <key>Quote Character</key> <value>"</value> </entry> <entry> <key>Escape Character</key> <value>\</value> </entry> <entry> <key>Comment Marker</key> </entry> <entry> <key>Null String</key> </entry> <entry> <key>Trim Fields</key> <value>true</value> </entry> <entry> <key>csvutils-character-set</key> <value>UTF-8</value> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.csv.CSVReader</type> </controllerServices> <controllerServices> <id>bb04676f-6936-3eaa-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <versionedComponentId>56dc4974-9651-3216-ab2a-e676e24dd703</versionedComponentId> <bundle> <artifact>nifi-dbcp-service-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <comments></comments> <descriptors> <entry> <key>Database Connection URL</key> <value> <name>Database Connection URL</name> </value> </entry> <entry> <key>Database Driver Class Name</key> <value> <name>Database Driver Class Name</name> </value> </entry> <entry> <key>database-driver-locations</key> <value> <name>database-driver-locations</name> </value> </entry> <entry> <key>kerberos-credentials-service</key> <value> <identifiesControllerService>org.apache.nifi.kerberos.KerberosCredentialsService</identifiesControllerService> <name>kerberos-credentials-service</name> </value> </entry> <entry> <key>Database User</key> <value> <name>Database User</name> </value> </entry> <entry> <key>Password</key> <value> <name>Password</name> </value> </entry> <entry> <key>Max Wait Time</key> <value> <name>Max Wait Time</name> </value> </entry> <entry> <key>Max Total Connections</key> <value> <name>Max Total Connections</name> </value> </entry> <entry> <key>Validation-query</key> <value> <name>Validation-query</name> </value> </entry> <entry> <key>dbcp-min-idle-conns</key> <value> <name>dbcp-min-idle-conns</name> </value> </entry> <entry> <key>dbcp-max-idle-conns</key> <value> <name>dbcp-max-idle-conns</name> </value> </entry> <entry> <key>dbcp-max-conn-lifetime</key> <value> <name>dbcp-max-conn-lifetime</name> </value> </entry> <entry> <key>dbcp-time-between-eviction-runs</key> <value> <name>dbcp-time-between-eviction-runs</name> </value> </entry> <entry> <key>dbcp-min-evictable-idle-time</key> <value> <name>dbcp-min-evictable-idle-time</name> </value> </entry> <entry> <key>dbcp-soft-min-evictable-idle-time</key> <value> <name>dbcp-soft-min-evictable-idle-time</name> </value> </entry> </descriptors> <name>thuyledb-postgres-DBCPConnectionPool</name> <persistsState>false</persistsState> <properties> <entry> <key>Database Connection URL</key> <value>jdbc:postgresql://postgresdb.ckdsaqpaaule.us-east-1.rds.amazonaws.com:5432/thuyle_dvdrental</value> </entry> <entry> <key>Database Driver Class Name</key> <value>org.postgresql.Driver</value> </entry> <entry> <key>database-driver-locations</key> <value>/home/nifi/jdbc-drivers/postgresql-42.2.6.jar</value> </entry> <entry> <key>kerberos-credentials-service</key> </entry> <entry> <key>Database User</key> <value>postgresadmin</value> </entry> <entry> <key>Password</key> </entry> <entry> <key>Max Wait Time</key> <value>500 millis</value> </entry> <entry> <key>Max Total Connections</key> <value>8</value> </entry> <entry> <key>Validation-query</key> </entry> <entry> <key>dbcp-min-idle-conns</key> <value>0</value> </entry> <entry> <key>dbcp-max-idle-conns</key> <value>8</value> </entry> <entry> <key>dbcp-max-conn-lifetime</key> <value>-1</value> </entry> <entry> <key>dbcp-time-between-eviction-runs</key> <value>-1</value> </entry> <entry> <key>dbcp-min-evictable-idle-time</key> <value>30 mins</value> </entry> <entry> <key>dbcp-soft-min-evictable-idle-time</key> <value>-1</value> </entry> </properties> <state>ENABLED</state> <type>org.apache.nifi.dbcp.DBCPConnectionPool</type> </controllerServices> <processors> <id>05136c6f-c4b4-3e66-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>8.0</x> <y>584.0</y> </position> <versionedComponentId>05136c6f-c4b4-3e66-b29c-c4da38c83eed</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>put-db-record-record-reader</key> <value> <identifiesControllerService>org.apache.nifi.serialization.RecordReaderFactory</identifiesControllerService> <name>put-db-record-record-reader</name> </value> </entry> <entry> <key>put-db-record-statement-type</key> <value> <name>put-db-record-statement-type</name> </value> </entry> <entry> <key>put-db-record-dcbp-service</key> <value> <identifiesControllerService>org.apache.nifi.dbcp.DBCPService</identifiesControllerService> <name>put-db-record-dcbp-service</name> </value> </entry> <entry> <key>put-db-record-catalog-name</key> <value> <name>put-db-record-catalog-name</name> </value> </entry> <entry> <key>put-db-record-schema-name</key> <value> <name>put-db-record-schema-name</name> </value> </entry> <entry> <key>put-db-record-table-name</key> <value> <name>put-db-record-table-name</name> </value> </entry> <entry> <key>put-db-record-translate-field-names</key> <value> <name>put-db-record-translate-field-names</name> </value> </entry> <entry> <key>put-db-record-unmatched-field-behavior</key> <value> <name>put-db-record-unmatched-field-behavior</name> </value> </entry> <entry> <key>put-db-record-unmatched-column-behavior</key> <value> <name>put-db-record-unmatched-column-behavior</name> </value> </entry> <entry> <key>put-db-record-update-keys</key> <value> <name>put-db-record-update-keys</name> </value> </entry> <entry> <key>put-db-record-field-containing-sql</key> <value> <name>put-db-record-field-containing-sql</name> </value> </entry> <entry> <key>put-db-record-allow-multiple-statements</key> <value> <name>put-db-record-allow-multiple-statements</name> </value> </entry> <entry> <key>put-db-record-quoted-identifiers</key> <value> <name>put-db-record-quoted-identifiers</name> </value> </entry> <entry> <key>put-db-record-quoted-table-identifiers</key> <value> <name>put-db-record-quoted-table-identifiers</name> </value> </entry> <entry> <key>put-db-record-query-timeout</key> <value> <name>put-db-record-query-timeout</name> </value> </entry> <entry> <key>rollback-on-failure</key> <value> <name>rollback-on-failure</name> </value> </entry> <entry> <key>table-schema-cache-size</key> <value> <name>table-schema-cache-size</name> </value> </entry> <entry> <key>put-db-record-max-batch-size</key> <value> <name>put-db-record-max-batch-size</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>put-db-record-record-reader</key> <value>23b93123-18d3-3c4b-0000-000000000000</value> </entry> <entry> <key>put-db-record-statement-type</key> <value>INSERT</value> </entry> <entry> <key>put-db-record-dcbp-service</key> <value>bb04676f-6936-3eaa-0000-000000000000</value> </entry> <entry> <key>put-db-record-catalog-name</key> </entry> <entry> <key>put-db-record-schema-name</key> </entry> <entry> <key>put-db-record-table-name</key> <value>customer</value> </entry> <entry> <key>put-db-record-translate-field-names</key> <value>true</value> </entry> <entry> <key>put-db-record-unmatched-field-behavior</key> <value>Ignore Unmatched Fields</value> </entry> <entry> <key>put-db-record-unmatched-column-behavior</key> <value>Fail on Unmatched Columns</value> </entry> <entry> <key>put-db-record-update-keys</key> </entry> <entry> <key>put-db-record-field-containing-sql</key> </entry> <entry> <key>put-db-record-allow-multiple-statements</key> <value>false</value> </entry> <entry> <key>put-db-record-quoted-identifiers</key> <value>false</value> </entry> <entry> <key>put-db-record-quoted-table-identifiers</key> <value>false</value> </entry> <entry> <key>put-db-record-query-timeout</key> <value>0 seconds</value> </entry> <entry> <key>rollback-on-failure</key> <value>false</value> </entry> <entry> <key>table-schema-cache-size</key> <value>100</value> </entry> <entry> <key>put-db-record-max-batch-size</key> <value>0</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>PutDatabaseRecord</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>true</autoTerminate> <name>retry</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.PutDatabaseRecord</type> </processors> <processors> <id>29beb3d5-b88e-32df-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>192.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Line Split Count</key> <value> <name>Line Split Count</name> </value> </entry> <entry> <key>Maximum Fragment Size</key> <value> <name>Maximum Fragment Size</name> </value> </entry> <entry> <key>Header Line Count</key> <value> <name>Header Line Count</name> </value> </entry> <entry> <key>Header Line Marker Characters</key> <value> <name>Header Line Marker Characters</name> </value> </entry> <entry> <key>Remove Trailing Newlines</key> <value> <name>Remove Trailing Newlines</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Line Split Count</key> <value>1</value> </entry> <entry> <key>Maximum Fragment Size</key> </entry> <entry> <key>Header Line Count</key> <value>1</value> </entry> <entry> <key>Header Line Marker Characters</key> </entry> <entry> <key>Remove Trailing Newlines</key> <value>true</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>SplitText</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>true</autoTerminate> <name>original</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>splits</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.SplitText</type> </processors> <processors> <id>2b305558-f05a-3254-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>8.0</x> <y>792.0</y> </position> <versionedComponentId>2b305558-f05a-3254-a069-72ac003eac04</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Success-Log</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> <processors> <id>7c677559-6b04-31fb-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>608.0</x> <y>584.0</y> </position> <versionedComponentId>7c677559-6b04-31fb-b3f0-f177624fd063</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Failure-Log</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> <processors> <id>c9cb38c4-95dd-35a1-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>8.0</x> <y>376.0</y> </position> <versionedComponentId>c9cb38c4-95dd-35a1-8f17-4779ac3bb0cf</versionedComponentId> <bundle> <artifact>nifi-update-attribute-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Delete Attributes Expression</key> <value> <name>Delete Attributes Expression</name> </value> </entry> <entry> <key>Store State</key> <value> <name>Store State</name> </value> </entry> <entry> <key>Stateful Variables Initial Value</key> <value> <name>Stateful Variables Initial Value</name> </value> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value> <name>canonical-value-lookup-cache-size</name> </value> </entry> <entry> <key>schema.name</key> <value> <name>schema.name</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Delete Attributes Expression</key> </entry> <entry> <key>Store State</key> <value>Do not store state</value> </entry> <entry> <key>Stateful Variables Initial Value</key> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value>100</value> </entry> <entry> <key>schema.name</key> <value>customer</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>UpdateAttribute</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.attributes.UpdateAttribute</type> </processors> <processors> <id>d7a85def-1b91-3a09-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>1160.0</x> <y>856.0</y> </position> <versionedComponentId>d7a85def-1b91-3a09-9416-ccd9d7fdbcc6</versionedComponentId> <bundle> <artifact>nifi-scripting-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Script Engine</key> <value> <name>Script Engine</name> </value> </entry> <entry> <key>Script File</key> <value> <name>Script File</name> </value> </entry> <entry> <key>Script Body</key> <value> <name>Script Body</name> </value> </entry> <entry> <key>Module Directory</key> <value> <name>Module Directory</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Script Engine</key> <value>python</value> </entry> <entry> <key>Script File</key> </entry> <entry> <key>Script Body</key> <value>import json import datetime import java.io from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import StreamCallback class ModJSON(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = json.loads(text) rows = len(obj["columns"]) newcolumns = {} for row in range(0, rows): if (obj["columns"][row]["column_type"]==93): obj["columns"][row]["value"] = datetime.datetime.strptime(obj["columns"][row]["value"], "%a %b %d %H:%M:%S %Z %Y").strftime("%Y-%m-%d %H:%M:%S.%f") newcolumns[obj["columns"][row]["name"]] = obj["columns"][row]["value"] outputStream.write(bytearray(json.dumps(newcolumns, indent=4).encode('utf-8'))) flowFile = session.get() if (flowFile != None): flowFile = session.write(flowFile, ModJSON()) session.transfer(flowFile, REL_SUCCESS) session.commit()</value> </entry> <entry> <key>Module Directory</key> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>Convert datetime -- ExecuteScript</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.script.ExecuteScript</type> </processors> <processors> <id>e6e49310-06e6-3642-0000-000000000000</id> <parentGroupId>7ba66444-6b1d-3473-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>0.0</y> </position> <versionedComponentId>e6e49310-06e6-3642-87e1-78f5d613f631</versionedComponentId> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.1</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>File Size</key> <value> <name>File Size</name> </value> </entry> <entry> <key>Batch Size</key> <value> <name>Batch Size</name> </value> </entry> <entry> <key>Data Format</key> <value> <name>Data Format</name> </value> </entry> <entry> <key>Unique FlowFiles</key> <value> <name>Unique FlowFiles</name> </value> </entry> <entry> <key>generate-ff-custom-text</key> <value> <name>generate-ff-custom-text</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>File Size</key> <value>0B</value> </entry> <entry> <key>Batch Size</key> <value>1</value> </entry> <entry> <key>Data Format</key> <value>Text</value> </entry> <entry> <key>Unique FlowFiles</key> <value>false</value> </entry> <entry> <key>generate-ff-custom-text</key> <value>"customer_id","store_id","first_name","last_name","email","address_id","activebool","create_date","last_update","active" 1,1,Mary,Smith,mary.smith@sakilacustomer.org,5,true,2006-02-14,2013-05-26 14:49:45,1 2,1,Patricia,Johnson,patricia.johnson@sakilacustomer.org,6,true,2006-02-14,2013-05-26 14:49:45,1 3,1,Linda,Williams,linda.williams@sakilacustomer.org,7,true,2006-02-14,2013-05-26 14:49:45,1 4,2,Barbara,Jones,barbara.jones@sakilacustomer.org,8,true,2006-02-14,2013-05-26 14:49:45,1 5,1,Elizabeth,Brown,elizabeth.brown@sakilacustomer.org,9,true,2006-02-14,2013-05-26 14:49:45,1 6,2,Jennifer,Davis,jennifer.davis@sakilacustomer.org,10,true,2006-02-14,2013-05-26 14:49:45,1 7,1,Maria,Miller,maria.miller@sakilacustomer.org,11,true,2006-02-14,2013-05-26 14:49:45,1 8,2,Susan,Wilson,susan.wilson@sakilacustomer.org,12,true,2006-02-14,2013-05-26 14:49:45,1 9,2,Margaret,Moore,margaret.moore@sakilacustomer.org,13,true,2006-02-14,2013-05-26 14:49:45,1 10,1,Dorothy,Taylor,dorothy.taylor@sakilacustomer.org,14,true,2006-02-14,2013-05-26 14:49:45,1 11,2,Lisa,Anderson,lisa.anderson@sakilacustomer.org,15,true,2006-02-14,2013-05-26 14:49:45,1 12,1,Nancy,Thomas,nancy.thomas@sakilacustomer.org,16,true,2006-02-14,2013-05-26 14:49:45,1 13,2,Karen,Jackson,karen.jackson@sakilacustomer.org,17,true,2006-02-14,2013-05-26 14:49:45,1 14,2,Betty,White,betty.white@sakilacustomer.org,18,true,2006-02-14,2013-05-26 14:49:45,1 15,1,Helen,Harris,helen.harris@sakilacustomer.org,19,true,2006-02-14,2013-05-26 14:49:45,1 16,2,Sandra,Martin,sandra.martin@sakilacustomer.org,20,true,2006-02-14,2013-05-26 14:49:45,0 17,1,Donna,Thompson,donna.thompson@sakilacustomer.org,21,true,2006-02-14,2013-05-26 14:49:45,1 18,2,Carol,Garcia,carol.garcia@sakilacustomer.org,22,true,2006-02-14,2013-05-26 14:49:45,1 19,1,Ruth,Martinez,ruth.martinez@sakilacustomer.org,23,true,2006-02-14,2013-05-26 14:49:45,1 20,2,Sharon,Robinson,sharon.robinson@sakilacustomer.org,24,true,2006-02-14,2013-05-26 14:49:45,1 21,1,Michelle,Clark,michelle.clark@sakilacustomer.org,25,true,2006-02-14,2013-05-26 14:49:45,1 22,1,Laura,Rodriguez,laura.rodriguez@sakilacustomer.org,26,true,2006-02-14,2013-05-26 14:49:45,1</value> </entry> <entry> <key>character-set</key> <value>UTF-8</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>100000 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>customer -- GenerateFlowFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.GenerateFlowFile</type> </processors> </snippet> <timestamp>02/12/2020 17:26:41 UTC</timestamp> </template>
... View more
02-11-2020
02:15 PM
PutDatabaseRecord[id=0f1a88fc-bd24-36fa-e244-e50ac953449d] Failed to process StandardFlowFileRecord[uuid=d40fe26f-06c0-44af-8f93-7850b0f6cffe,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1581454928503-52, container=default, section=52], offset=19212, length=59617],offset=0,name=d40fe26f-06c0-44af-8f93-7850b0f6cffe,size=59617] due to (line 1) invalid char between encapsulated token and delimiter
... View more
Labels:
02-11-2020
02:14 PM
The value of the sql.args.9.value is '2013-05-26 14:49:045', which cannot be converted to a timestamp
... View more
Labels:
08-10-2019
10:43 PM
How Apache Nifi delete record from BigQuery? Can we use ExceuteScript processor? if yes, how to import library? any sample script?
... View more
Labels:
08-09-2019
01:49 PM
How Apche Nifi Insert/Update/delete data from BigQuery
... View more
Labels:
07-25-2019
12:56 PM
Sorry, but It dose not work in my side. I guess that you didnot test it $.[2].length() In EvaluateJsonPath is always unmatched
... View more
07-23-2019
08:20 PM
Apache nifi version 1.9.2 issue with PutBigQueryBatch
... View more
Labels:
07-23-2019
07:19 PM
Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations) With Delete the do correct with Update: They automatic change Primarykey from product_key to PRODUCTKEY (I tested with PutSQL and Putdatabaserecord it have the same issue) Any helps?
... View more
Labels:
07-23-2019
05:14 PM
In this case, you are changing variable manually, my expectation is changing it through any processors (as UpdateAttribute, ExcecuteScript, ... )
... View more
07-22-2019
07:54 PM
thanks @Shu, It doesnot work with when empty json and also nested json
... View more
07-22-2019
12:25 PM
yes, I created variable after now I want to update it with new value, how I can do it?,
... View more
07-21-2019
04:57 PM
Hi all, I am using Apache NIFI, how I can change/modify/replace value of global/environment variables?
... View more
Labels:
07-21-2019
02:50 PM
Apache nifi - Split json error when an array has only one record or empty,
... View more
