Member since
09-24-2017
28
Posts
0
Kudos Received
0
Solutions
09-27-2020
09:20 PM
I am running a NiFi flow where I have InvokeHTTP Processor calling a python script thru flask API. When I am processing many files (in 100's) then some files like (1-2%) fails with java.net.Socket.Exception:Broken Pipe (Write Failed) error in InvokeHTTP Processor. I have observed this happening more when I am running two separate flows accessing same flask API but different python script. What do you think I should change to avoid it. It only happens once in a while for only few files (1-2%) of total flow file.
... View more
Labels:
- Labels:
-
Apache NiFi
08-15-2020
09:21 PM
I have a incoming csv file which I need to route to different nifi flows based on the values in one of the column. Each csv has a column called "Country" which has same country name in all of its rows. What I want is File 1 - 50 rows (All rows will only one value in country column which will be any of the below) England, england, Britain, UK, United kingdom, USA, United States So the file which contain values like England, england, Britain, UK, United kingdom in country column should be routed to one flow and if file contain USA or United States then it should be routed to another flow. Note - One file will only contain one value in all rows. So routing based on first row value of country column for entire file is fine. How to achieve this?
... View more
Labels:
- Labels:
-
Apache NiFi
06-04-2020
01:59 PM
I had been using ExecuteStreamCommand processor to run python scripts and successfully giving flowfile as input to python scripts using sys.stdin and flowfile output as sys.stdout. Now all of these python scripts to be converted to Pyspark. How can have flow file as input/output in ExecuteStreamCommand Processor. Sys.stdin and sys.stdout does not seems to be working with pyspark. Is there any other way?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Spark
05-28-2020
05:35 PM
I have a csv flowfile with single record. I need to create its file name based on couple of column values in the csv file. Can you please let me know how we can do it by using the column name only not the position of the column as column position may change. Example CSV File Name , City, State, Country, Gender John, Dallas, Texas, USA, M File name should be John_USA.csv
... View more
Labels:
- Labels:
-
Apache NiFi
05-24-2020
09:02 PM
I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code - Table Definition - CREATE TABLE `local_db`.`table_x` ( `Unique_code` varchar(50) NOT NULL, `city` varchar(200) DEFAULT NULL, `state` varchar(50) DEFAULT NULL, `population` bigint(20) DEFAULT NULL, `Govt` varchar(50) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; ``` input_file = "input_file" csv_data = csv.reader(open(input_file)) try: connection = mysql.connector.connect(host='localhost', database='local_db', user='root', password='root', port = '3306') cursor = connection.cursor() for row in csv_data: cursor.execute(""" INSERT INTO table_x(Unique_code,city,state,population,Govt) VALUES("%s", "%s", "%s","%s", "%s") ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \ population = VALUES(population),Govt = VALUES(Govt)""") connection.commit() print(cursor.rowcount, "Record inserted successfully into table_x") cursor.close() except mysql.connector.Error as error: print("Failed to insert record into table_x table {}".format(error)) finally: if (connection.is_connected()): connection.close() print("MySQL connection is closed") ``` But I am getting below error - Failed to insert record into table_x table 1366 (22007): Incorrect integer value: '%s' for column `local_db`.`table_x`.`population` at row 1 MySQL connection is closed In other thread it was suggested to change SET sql_mode = "" But its not an option for me since I would be running it on organization server which I can not change for this only. Please suggest what code changes I can do here to handle this situation.
... View more
Labels:
- Labels:
-
Apache NiFi
05-20-2020
10:11 AM
Hi Steven, Thanks a lot for responding to my query. I helps a lot. I have downloaded your flow. I have few clarification questions. 1. In Updateattribute processor in your template - It is not adding any flow file attribute. So we need to add Code and Continent there right? 2. So we need to add anything in the properites of invokehttp processor? We will get response from invokeHTTP and that response needs to replace the values of longitude and latitude in original flow file. How to do that? 3. If original flow file is a csv file instead of json. Will the same flow work or do we need to convert csv to json first? csv file is like this Company_name, City, Country, Latitude, Longitude Abc Corp, Newyork, USA, 98.765, 67.5656
... View more
05-19-2020
11:35 PM
Hi, I have a flowfile json request which looks like this. Actually flowfile is a csv file which I am converting to json first so that another json request can be created for invokehttp processor. But if this csv to json conversion is not required for it then I would prefer to keep it csv only. { "type": "record", "name": "Location", "fields": [{ "name": "Company_name", "type": ["null", "string"] }, { "name": "City", "type": ["null", "string"] }, { "name": "Country", "type": ["null", "string"] }, { "name": "Latitude", "type": ["null", "string"] }, { "name": "Longitude", "type": ["null", "string"] } ] } I need to send a request to invokehttp processor which look like below - { "Code":"A-5678", "Latitude":"78.495603", "Longitude":"5.969756", "Continent":"" } Latitude and longitude is to be taken from flowfile fields while Code and Continent (To be sent as blank string) are not available in flowfile and has to be created. How can I create this custom request. I tried evaluatejsonpath processor but unable to add Continent as it does not take blank value. Once the request is received I need to replace the Latitude and Logitude values in flow file with the response received. Please let me know how to achieve it. The final output has to be csv file. @Shu_ashu @mburgess Please help
... View more
Labels:
- Labels:
-
Apache NiFi
04-24-2020
08:47 PM
Can someone please tell me how to do UPSERT using PutDatabaseRecord processors. I am using MariaDB
... View more
Labels:
- Labels:
-
Apache NiFi
03-05-2020
09:48 AM
Hi All, This is what I am trying to achieve. all of them are csv files Flowfile (csv) - col1_raw, col2_raw, col3_raw, col4_raw 1 , 2 , 3 , 4 1 , 2 , 3 , 4 Look up file1 (csv) col_name_raw , col_name_business, class col1_raw , col1_business , Secret col2_raw , col2_business , Public col3_raw , col3_business , Secret col4_raw , col4_business , Public Look up file2 (csv) class, action public, copy secret, nocopy output (csv) of flowfile should be col2_business, col4_business 2 , 4 2 , 4 So basically I am filtering the columns of flowfile based on the action listed lookup file2. Also changing the raw column name to business column name as per lookup file1. In SQL it is very easy but in Nifi how can I achieve this using existing processors. Please help.
... View more
- Tags:
- NiFi
Labels:
- Labels:
-
Apache NiFi
02-25-2020
03:43 PM
Hi,
I have created a NiFi flow which take multiple input csv files (Each one have 5 sections) and then pass to executestreamcommand processor. Which eventually trigger a python script and python create output of 5 csv files (Which are basically merging of specific section from all the files). Now challenge is how to pass these 5 csv files back to NiFi flow which takes these files to an another executestreamcommand processor which will again trigger a python script to convert each of the 5 files to xml files. Here is the sequence
1. List File
2. Fetch File
3. ExecuteStreamCommand - Create 5 csv file from all the input files passed.
4. ExecuteStreamCommand - Convert 5 csv files to xml
5. PutFile - Put all 5 xml files to a folder
I am unable to write the 5 csv files from step 3 back to NiFi flow so that next processor can process them. Please help.
... View more
Labels:
- Labels:
-
Apache NiFi
02-17-2020
10:28 PM
Hi Alim, Your solution works fine for me except the xml file content is coming in one single line instead of blocks. Any idea why?
... View more
02-16-2020
01:11 PM
Hi Matt, I am able to trigger python script now but stuck on how to pass the output file to STDout so that it can come back to flowfile. Here is what I am trying. STDIN seems to be working fine but STDOUT is not working f = sys.stdin reader = csv.reader(f) *********** ##Python code for creating the output file and passing it to STDOUT xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile
... View more
02-16-2020
01:08 PM
Hi,
Can some one please give an example of STDIN and STDOUT in python script to take input from Nifi flowfile and paas the output file to NiFi flowfile again. I am using below script for STDIN and STDOUT. STDIN seems to be working fine but STDOUT is not working. Please help
f = sys.stdin reader = csv.reader(f)
***********
##Python code for creating the output file and passing it to STDOUT
xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile
... View more
Labels:
- Labels:
-
Apache NiFi
02-15-2020
01:09 AM
Hi Matt, I am able to execute python script from command line. Executescript processor is not an option here because it is an experimental process. I am looking for something more robust which can handle heavy data and also I should be able to put the flow in production. Can you please see the settings in my flow and see if I am doing anything wrong here?
... View more
02-15-2020
12:05 AM
My NiFi was working fine until today (After my laptop hanged and I restarted it). Now NiFi is unable to launch and gives below error -
As of now I tried below -
1. Deleted content, database, flowfile and provenance repository and restart NiFi - Did not help
2. Restart my machine - Did not help
3. Uninstalled jre and installed jdk - Did not help ( It was working fine when I had jre until today)
My current jdk location is C:\Program Files\Java\jdk-13.0.2
I am using nifi-1.11.0
********************************************************************
2020-02-15 01:21:04,851 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2020-02-15 01:21:04,852 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0
2020-02-15 01:21:04,852 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\Progra~1\Java\jdk-13.0.2\bin\java.exe -classpath C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\conf;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\javax.servlet-api-3.1.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jcl-over-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jetty-schemas-3.1.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jul-to-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\log4j-over-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\logback-classic-1.2.3.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\logback-core-1.2.3.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-api-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-framework-api-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-nar-utils-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-properties-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-runtime-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\slf4j-api-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\javax.activation-api-1.2.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\javax.annotation-api-1.3.2.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-api-2.3.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-core-2.3.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-impl-2.3.0.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\conf\nifi.properties -Dnifi.bootstrap.listen.port=65326 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\\logs org.apache.nifi.NiFi
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.nifi.bootstrap.util.OSUtils (file:/C:/Users/503067805/Desktop/IDP_Project/nifi-1.11.0/lib/bootstrap/nifi-bootstrap-1.11.0.jar) to method java.lang.ProcessImpl.pid()
WARNING: Please consider reporting this to the maintainers of org.apache.nifi.bootstrap.util.OSUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-02-15 01:21:04,879 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\run\nifi.pid; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file
2020-02-15 01:21:04,888 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read status file C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\run\nifi.status; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file
2020-02-15 01:21:04,894 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 8552
... View more
Labels:
- Labels:
-
Apache NiFi
02-13-2020
12:17 PM
Hi,
I am running a small test where I want to merge (union) two CSV files of same format. my Mergecontent Processor is merge the file but it keep sending multiple merged file to next processor. I am sending the file to Mergecontent thru Listfile and Fetchfile and I can see they send only two files to Mergecontent but input to mergecontent show hundreds of files. Not sure why. Please help. Attaching the screenshots
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.3">
<description></description>
<groupId>3ed4e195-0170-1000-7be4-197989ceef99</groupId>
<name>Merge CSV Files</name>
<snippet>
<connections>
<id>2ca736d6-69cf-39d1-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<bends>
<x>473.8800048828125</x>
<y>869.5999755859375</y>
</bends>
<bends>
<x>473.8800048828125</x>
<y>919.5999755859375</y>
</bends>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>6ad9f863-3108-3ed3-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>failure</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>6ad9f863-3108-3ed3-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>5662111e-1e71-3ffd-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>1cf0fdfc-41f0-38e0-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>merged</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>65d7ef96-2d72-396c-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>6b194994-5bdb-3a7a-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>33c56676-058e-35fa-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>74ae6286-74b3-3a30-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>7463a2e7-4e73-3da3-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>6ad9f863-3108-3ed3-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>1cf0fdfc-41f0-38e0-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>addb7fb8-03da-37bd-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>7f36d436-85ae-332b-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>6ad9f863-3108-3ed3-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>d4b3f000-3ea4-3514-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<bends>
<x>468.20001220703125</x>
<y>236.1199951171875</y>
</bends>
<bends>
<x>468.20001220703125</x>
<y>286.1199951171875</y>
</bends>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>33c56676-058e-35fa-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>failure</selectedRelationships>
<selectedRelationships>not.found</selectedRelationships>
<selectedRelationships>permission.denied</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>33c56676-058e-35fa-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>dc3e7cfe-efa4-3344-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>65d7ef96-2d72-396c-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>33c56676-058e-35fa-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>eef0c5c9-9123-3437-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<bends>
<x>469.0</x>
<y>462.0</y>
</bends>
<bends>
<x>469.0</x>
<y>512.0</y>
</bends>
<destination>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>65d7ef96-2d72-396c-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>failure</selectedRelationships>
<selectedRelationships>original</selectedRelationships>
<source>
<groupId>bdc9724d-9179-3f54-0000-000000000000</groupId>
<id>65d7ef96-2d72-396c-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<processors>
<id>1cf0fdfc-41f0-38e0-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>624.0</y>
</position>
<bundle>
<artifact>nifi-update-attribute-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Delete Attributes Expression</key>
<value>
<name>Delete Attributes Expression</name>
</value>
</entry>
<entry>
<key>Store State</key>
<value>
<name>Store State</name>
</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
<value>
<name>Stateful Variables Initial Value</name>
</value>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>
<name>canonical-value-lookup-cache-size</name>
</value>
</entry>
<entry>
<key>filename</key>
<value>
<name>filename</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Delete Attributes Expression</key>
</entry>
<entry>
<key>Store State</key>
<value>Do not store state</value>
</entry>
<entry>
<key>Stateful Variables Initial Value</key>
</entry>
<entry>
<key>canonical-value-lookup-cache-size</key>
<value>100</value>
</entry>
<entry>
<key>filename</key>
<value>${filename}.csv</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>UpdateAttribute</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.attributes.UpdateAttribute</type>
</processors>
<processors>
<id>33c56676-058e-35fa-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>200.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>File to Fetch</key>
<value>
<name>File to Fetch</name>
</value>
</entry>
<entry>
<key>Completion Strategy</key>
<value>
<name>Completion Strategy</name>
</value>
</entry>
<entry>
<key>Move Destination Directory</key>
<value>
<name>Move Destination Directory</name>
</value>
</entry>
<entry>
<key>Move Conflict Strategy</key>
<value>
<name>Move Conflict Strategy</name>
</value>
</entry>
<entry>
<key>Log level when file not found</key>
<value>
<name>Log level when file not found</name>
</value>
</entry>
<entry>
<key>Log level when permission denied</key>
<value>
<name>Log level when permission denied</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>File to Fetch</key>
<value>${absolute.path}/${filename}</value>
</entry>
<entry>
<key>Completion Strategy</key>
<value>None</value>
</entry>
<entry>
<key>Move Destination Directory</key>
</entry>
<entry>
<key>Move Conflict Strategy</key>
<value>Rename</value>
</entry>
<entry>
<key>Log level when file not found</key>
<value>ERROR</value>
</entry>
<entry>
<key>Log level when permission denied</key>
<value>ERROR</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>FetchFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>not.found</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>permission.denied</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.FetchFile</type>
</processors>
<processors>
<id>65d7ef96-2d72-396c-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>424.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Merge Strategy</key>
<value>
<name>Merge Strategy</name>
</value>
</entry>
<entry>
<key>Merge Format</key>
<value>
<name>Merge Format</name>
</value>
</entry>
<entry>
<key>Attribute Strategy</key>
<value>
<name>Attribute Strategy</name>
</value>
</entry>
<entry>
<key>Correlation Attribute Name</key>
<value>
<name>Correlation Attribute Name</name>
</value>
</entry>
<entry>
<key>mergecontent-metadata-strategy</key>
<value>
<name>mergecontent-metadata-strategy</name>
</value>
</entry>
<entry>
<key>Minimum Number of Entries</key>
<value>
<name>Minimum Number of Entries</name>
</value>
</entry>
<entry>
<key>Maximum Number of Entries</key>
<value>
<name>Maximum Number of Entries</name>
</value>
</entry>
<entry>
<key>Minimum Group Size</key>
<value>
<name>Minimum Group Size</name>
</value>
</entry>
<entry>
<key>Maximum Group Size</key>
<value>
<name>Maximum Group Size</name>
</value>
</entry>
<entry>
<key>Max Bin Age</key>
<value>
<name>Max Bin Age</name>
</value>
</entry>
<entry>
<key>Maximum number of Bins</key>
<value>
<name>Maximum number of Bins</name>
</value>
</entry>
<entry>
<key>Delimiter Strategy</key>
<value>
<name>Delimiter Strategy</name>
</value>
</entry>
<entry>
<key>Header File</key>
<value>
<name>Header File</name>
</value>
</entry>
<entry>
<key>Footer File</key>
<value>
<name>Footer File</name>
</value>
</entry>
<entry>
<key>Demarcator File</key>
<value>
<name>Demarcator File</name>
</value>
</entry>
<entry>
<key>Compression Level</key>
<value>
<name>Compression Level</name>
</value>
</entry>
<entry>
<key>Keep Path</key>
<value>
<name>Keep Path</name>
</value>
</entry>
<entry>
<key>Tar Modified Time</key>
<value>
<name>Tar Modified Time</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Merge Strategy</key>
<value>Bin-Packing Algorithm</value>
</entry>
<entry>
<key>Merge Format</key>
<value>Binary Concatenation</value>
</entry>
<entry>
<key>Attribute Strategy</key>
<value>Keep Only Common Attributes</value>
</entry>
<entry>
<key>Correlation Attribute Name</key>
</entry>
<entry>
<key>mergecontent-metadata-strategy</key>
<value>Do Not Merge Uncommon Metadata</value>
</entry>
<entry>
<key>Minimum Number of Entries</key>
<value>2</value>
</entry>
<entry>
<key>Maximum Number of Entries</key>
<value>1000</value>
</entry>
<entry>
<key>Minimum Group Size</key>
<value>5 kb</value>
</entry>
<entry>
<key>Maximum Group Size</key>
</entry>
<entry>
<key>Max Bin Age</key>
<value>5 s</value>
</entry>
<entry>
<key>Maximum number of Bins</key>
<value>1</value>
</entry>
<entry>
<key>Delimiter Strategy</key>
<value>Filename</value>
</entry>
<entry>
<key>Header File</key>
</entry>
<entry>
<key>Footer File</key>
</entry>
<entry>
<key>Demarcator File</key>
</entry>
<entry>
<key>Compression Level</key>
<value>1</value>
</entry>
<entry>
<key>Keep Path</key>
<value>false</value>
</entry>
<entry>
<key>Tar Modified Time</key>
<value>${file.lastModifiedTime}</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>MergeContent</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>merged</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>original</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.MergeContent</type>
</processors>
<processors>
<id>6ad9f863-3108-3ed3-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>832.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Directory</key>
<value>
<name>Directory</name>
</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>
<name>Conflict Resolution Strategy</name>
</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>
<name>Create Missing Directories</name>
</value>
</entry>
<entry>
<key>Maximum File Count</key>
<value>
<name>Maximum File Count</name>
</value>
</entry>
<entry>
<key>Last Modified Time</key>
<value>
<name>Last Modified Time</name>
</value>
</entry>
<entry>
<key>Permissions</key>
<value>
<name>Permissions</name>
</value>
</entry>
<entry>
<key>Owner</key>
<value>
<name>Owner</name>
</value>
</entry>
<entry>
<key>Group</key>
<value>
<name>Group</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Directory</key>
<value>C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Merge_Files\Output</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>fail</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>true</value>
</entry>
<entry>
<key>Maximum File Count</key>
</entry>
<entry>
<key>Last Modified Time</key>
</entry>
<entry>
<key>Permissions</key>
</entry>
<entry>
<key>Owner</key>
</entry>
<entry>
<key>Group</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>PutFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.PutFile</type>
</processors>
<processors>
<id>74ae6286-74b3-3a30-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>0.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Input Directory</key>
<value>
<name>Input Directory</name>
</value>
</entry>
<entry>
<key>listing-strategy</key>
<value>
<name>listing-strategy</name>
</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>
<name>Recurse Subdirectories</name>
</value>
</entry>
<entry>
<key>Input Directory Location</key>
<value>
<name>Input Directory Location</name>
</value>
</entry>
<entry>
<key>File Filter</key>
<value>
<name>File Filter</name>
</value>
</entry>
<entry>
<key>Path Filter</key>
<value>
<name>Path Filter</name>
</value>
</entry>
<entry>
<key>Include File Attributes</key>
<value>
<name>Include File Attributes</name>
</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>
<name>Minimum File Age</name>
</value>
</entry>
<entry>
<key>Maximum File Age</key>
<value>
<name>Maximum File Age</name>
</value>
</entry>
<entry>
<key>Minimum File Size</key>
<value>
<name>Minimum File Size</name>
</value>
</entry>
<entry>
<key>Maximum File Size</key>
<value>
<name>Maximum File Size</name>
</value>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>
<name>Ignore Hidden Files</name>
</value>
</entry>
<entry>
<key>target-system-timestamp-precision</key>
<value>
<name>target-system-timestamp-precision</name>
</value>
</entry>
<entry>
<key>et-state-cache</key>
<value>
<identifiesControllerService>org.apache.nifi.distributed.cache.client.DistributedMapCacheClient</identifiesControllerService>
<name>et-state-cache</name>
</value>
</entry>
<entry>
<key>et-time-window</key>
<value>
<name>et-time-window</name>
</value>
</entry>
<entry>
<key>et-initial-listing-target</key>
<value>
<name>et-initial-listing-target</name>
</value>
</entry>
<entry>
<key>et-node-identifier</key>
<value>
<name>et-node-identifier</name>
</value>
</entry>
<entry>
<key>track-performance</key>
<value>
<name>track-performance</name>
</value>
</entry>
<entry>
<key>max-performance-metrics</key>
<value>
<name>max-performance-metrics</name>
</value>
</entry>
<entry>
<key>max-operation-time</key>
<value>
<name>max-operation-time</name>
</value>
</entry>
<entry>
<key>max-listing-time</key>
<value>
<name>max-listing-time</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Input Directory</key>
<value>C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Merge_Files\Input</value>
</entry>
<entry>
<key>listing-strategy</key>
<value>timestamps</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>true</value>
</entry>
<entry>
<key>Input Directory Location</key>
<value>Local</value>
</entry>
<entry>
<key>File Filter</key>
<value>[^\.].*</value>
</entry>
<entry>
<key>Path Filter</key>
</entry>
<entry>
<key>Include File Attributes</key>
<value>true</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>0 sec</value>
</entry>
<entry>
<key>Maximum File Age</key>
</entry>
<entry>
<key>Minimum File Size</key>
<value>0 B</value>
</entry>
<entry>
<key>Maximum File Size</key>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>true</value>
</entry>
<entry>
<key>target-system-timestamp-precision</key>
<value>auto-detect</value>
</entry>
<entry>
<key>et-state-cache</key>
</entry>
<entry>
<key>et-time-window</key>
<value>3 hours</value>
</entry>
<entry>
<key>et-initial-listing-target</key>
<value>all</value>
</entry>
<entry>
<key>et-node-identifier</key>
<value>${hostname()}</value>
</entry>
<entry>
<key>track-performance</key>
<value>false</value>
</entry>
<entry>
<key>max-performance-metrics</key>
<value>100000</value>
</entry>
<entry>
<key>max-operation-time</key>
<value>10 secs</value>
</entry>
<entry>
<key>max-listing-time</key>
<value>3 mins</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>ListFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.ListFile</type>
</processors>
<processors>
<id>7f36d436-85ae-332b-0000-000000000000</id>
<parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>1032.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Log Level</key>
<value>
<name>Log Level</name>
</value>
</entry>
<entry>
<key>Log Payload</key>
<value>
<name>Log Payload</name>
</value>
</entry>
<entry>
<key>Attributes to Log</key>
<value>
<name>Attributes to Log</name>
</value>
</entry>
<entry>
<key>attributes-to-log-regex</key>
<value>
<name>attributes-to-log-regex</name>
</value>
</entry>
<entry>
<key>Attributes to Ignore</key>
<value>
<name>Attributes to Ignore</name>
</value>
</entry>
<entry>
<key>attributes-to-ignore-regex</key>
<value>
<name>attributes-to-ignore-regex</name>
</value>
</entry>
<entry>
<key>Log prefix</key>
<value>
<name>Log prefix</name>
</value>
</entry>
<entry>
<key>character-set</key>
<value>
<name>character-set</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Log Level</key>
<value>info</value>
</entry>
<entry>
<key>Log Payload</key>
<value>false</value>
</entry>
<entry>
<key>Attributes to Log</key>
</entry>
<entry>
<key>attributes-to-log-regex</key>
<value>.*</value>
</entry>
<entry>
<key>Attributes to Ignore</key>
</entry>
<entry>
<key>attributes-to-ignore-regex</key>
</entry>
<entry>
<key>Log prefix</key>
</entry>
<entry>
<key>character-set</key>
<value>windows-1252</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>LogAttribute</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.LogAttribute</type>
</processors>
</snippet>
<timestamp>02/13/2020 14:10:59 CST</timestamp>
</template>
and the flow
... View more
Labels:
- Labels:
-
Apache NiFi
02-13-2020
12:07 PM
Hi Mat I can execute the script from spyder (Anaconda) and it is successful. From executestreamcommand processor its not even triggering the script it seems. Its a .py file in the folder. Can this be an issue?
... View more
02-13-2020
12:05 PM
Thank you very much
... View more
- Tags:
- hank you
02-12-2020
09:23 PM
I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help
Below is the error in bulletin board
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
... View more
Labels:
- Labels:
-
Apache NiFi
02-12-2020
10:35 AM
Hi,
I am looking for a sample NiFi template which can combine multiple csv files coming from single source into one csv file and then convert it into a xml file. Can some one please help in getting the template for this.
Thanks
Note - I have tried mergecontent processor to combine the csv files but it keeps generating 100s of non csv files not sure what is happening. Attaching the template I created for your reference.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.3">
<description></description>
<groupId>3a0529f4-0170-1000-7792-3205dea567f8</groupId>
<name>Merge CSV Files using MergeContent Processor</name>
<snippet>
<connections>
<id>026df4be-1012-30bc-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<bends>
<x>469.0</x>
<y>455.0</y>
</bends>
<bends>
<x>469.0</x>
<y>505.0</y>
</bends>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c300524c-bcd5-3925-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>failure</selectedRelationships>
<selectedRelationships>original</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c300524c-bcd5-3925-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>884b9ed5-2ad3-3919-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<bends>
<x>469.0</x>
<y>623.0</y>
</bends>
<bends>
<x>469.0</x>
<y>673.0</y>
</bends>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c3c7aecc-217a-397d-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>failure</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c3c7aecc-217a-397d-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>9810e9ce-09a4-37e5-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>083d8a61-3cc9-33c7-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c3c7aecc-217a-397d-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>b8698473-5613-35e5-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c3c7aecc-217a-397d-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>merged</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c300524c-bcd5-3925-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>cbc26047-48ce-39ab-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>67f0a86e-9bea-3891-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>968a0059-8b8b-31fc-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<connections>
<id>e39e8309-ada9-308a-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>c300524c-bcd5-3925-0000-000000000000</id>
<type>PROCESSOR</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId>
<id>67f0a86e-9bea-3891-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<processors>
<id>083d8a61-3cc9-33c7-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>792.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Log Level</key>
<value>
<name>Log Level</name>
</value>
</entry>
<entry>
<key>Log Payload</key>
<value>
<name>Log Payload</name>
</value>
</entry>
<entry>
<key>Attributes to Log</key>
<value>
<name>Attributes to Log</name>
</value>
</entry>
<entry>
<key>attributes-to-log-regex</key>
<value>
<name>attributes-to-log-regex</name>
</value>
</entry>
<entry>
<key>Attributes to Ignore</key>
<value>
<name>Attributes to Ignore</name>
</value>
</entry>
<entry>
<key>attributes-to-ignore-regex</key>
<value>
<name>attributes-to-ignore-regex</name>
</value>
</entry>
<entry>
<key>Log prefix</key>
<value>
<name>Log prefix</name>
</value>
</entry>
<entry>
<key>character-set</key>
<value>
<name>character-set</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Log Level</key>
<value>info</value>
</entry>
<entry>
<key>Log Payload</key>
<value>false</value>
</entry>
<entry>
<key>Attributes to Log</key>
</entry>
<entry>
<key>attributes-to-log-regex</key>
<value>.*</value>
</entry>
<entry>
<key>Attributes to Ignore</key>
</entry>
<entry>
<key>attributes-to-ignore-regex</key>
</entry>
<entry>
<key>Log prefix</key>
</entry>
<entry>
<key>character-set</key>
<value>windows-1252</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>LogAttribute</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.LogAttribute</type>
</processors>
<processors>
<id>67f0a86e-9bea-3891-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>192.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>File to Fetch</key>
<value>
<name>File to Fetch</name>
</value>
</entry>
<entry>
<key>Completion Strategy</key>
<value>
<name>Completion Strategy</name>
</value>
</entry>
<entry>
<key>Move Destination Directory</key>
<value>
<name>Move Destination Directory</name>
</value>
</entry>
<entry>
<key>Move Conflict Strategy</key>
<value>
<name>Move Conflict Strategy</name>
</value>
</entry>
<entry>
<key>Log level when file not found</key>
<value>
<name>Log level when file not found</name>
</value>
</entry>
<entry>
<key>Log level when permission denied</key>
<value>
<name>Log level when permission denied</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>File to Fetch</key>
<value>${absolute.path}/${filename}</value>
</entry>
<entry>
<key>Completion Strategy</key>
<value>None</value>
</entry>
<entry>
<key>Move Destination Directory</key>
</entry>
<entry>
<key>Move Conflict Strategy</key>
<value>Rename</value>
</entry>
<entry>
<key>Log level when file not found</key>
<value>ERROR</value>
</entry>
<entry>
<key>Log level when permission denied</key>
<value>ERROR</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>FetchFile</name>
<relationships>
<autoTerminate>true</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>not.found</name>
</relationships>
<relationships>
<autoTerminate>true</autoTerminate>
<name>permission.denied</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.FetchFile</type>
</processors>
<processors>
<id>968a0059-8b8b-31fc-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>0.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Input Directory</key>
<value>
<name>Input Directory</name>
</value>
</entry>
<entry>
<key>listing-strategy</key>
<value>
<name>listing-strategy</name>
</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>
<name>Recurse Subdirectories</name>
</value>
</entry>
<entry>
<key>Input Directory Location</key>
<value>
<name>Input Directory Location</name>
</value>
</entry>
<entry>
<key>File Filter</key>
<value>
<name>File Filter</name>
</value>
</entry>
<entry>
<key>Path Filter</key>
<value>
<name>Path Filter</name>
</value>
</entry>
<entry>
<key>Include File Attributes</key>
<value>
<name>Include File Attributes</name>
</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>
<name>Minimum File Age</name>
</value>
</entry>
<entry>
<key>Maximum File Age</key>
<value>
<name>Maximum File Age</name>
</value>
</entry>
<entry>
<key>Minimum File Size</key>
<value>
<name>Minimum File Size</name>
</value>
</entry>
<entry>
<key>Maximum File Size</key>
<value>
<name>Maximum File Size</name>
</value>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>
<name>Ignore Hidden Files</name>
</value>
</entry>
<entry>
<key>target-system-timestamp-precision</key>
<value>
<name>target-system-timestamp-precision</name>
</value>
</entry>
<entry>
<key>et-state-cache</key>
<value>
<identifiesControllerService>org.apache.nifi.distributed.cache.client.DistributedMapCacheClient</identifiesControllerService>
<name>et-state-cache</name>
</value>
</entry>
<entry>
<key>et-time-window</key>
<value>
<name>et-time-window</name>
</value>
</entry>
<entry>
<key>et-initial-listing-target</key>
<value>
<name>et-initial-listing-target</name>
</value>
</entry>
<entry>
<key>et-node-identifier</key>
<value>
<name>et-node-identifier</name>
</value>
</entry>
<entry>
<key>track-performance</key>
<value>
<name>track-performance</name>
</value>
</entry>
<entry>
<key>max-performance-metrics</key>
<value>
<name>max-performance-metrics</name>
</value>
</entry>
<entry>
<key>max-operation-time</key>
<value>
<name>max-operation-time</name>
</value>
</entry>
<entry>
<key>max-listing-time</key>
<value>
<name>max-listing-time</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Input Directory</key>
<value>C:\Users\503067805\Desktop\IDP Project\POC - Well Path\Input</value>
</entry>
<entry>
<key>listing-strategy</key>
<value>timestamps</value>
</entry>
<entry>
<key>Recurse Subdirectories</key>
<value>true</value>
</entry>
<entry>
<key>Input Directory Location</key>
<value>Local</value>
</entry>
<entry>
<key>File Filter</key>
<value>[^\.].*</value>
</entry>
<entry>
<key>Path Filter</key>
</entry>
<entry>
<key>Include File Attributes</key>
<value>true</value>
</entry>
<entry>
<key>Minimum File Age</key>
<value>0 sec</value>
</entry>
<entry>
<key>Maximum File Age</key>
</entry>
<entry>
<key>Minimum File Size</key>
<value>0 B</value>
</entry>
<entry>
<key>Maximum File Size</key>
</entry>
<entry>
<key>Ignore Hidden Files</key>
<value>true</value>
</entry>
<entry>
<key>target-system-timestamp-precision</key>
<value>auto-detect</value>
</entry>
<entry>
<key>et-state-cache</key>
</entry>
<entry>
<key>et-time-window</key>
<value>3 hours</value>
</entry>
<entry>
<key>et-initial-listing-target</key>
<value>all</value>
</entry>
<entry>
<key>et-node-identifier</key>
<value>${hostname()}</value>
</entry>
<entry>
<key>track-performance</key>
<value>false</value>
</entry>
<entry>
<key>max-performance-metrics</key>
<value>100000</value>
</entry>
<entry>
<key>max-operation-time</key>
<value>10 secs</value>
</entry>
<entry>
<key>max-listing-time</key>
<value>3 mins</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>ListFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.ListFile</type>
</processors>
<processors>
<id>c300524c-bcd5-3925-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>416.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Merge Strategy</key>
<value>
<name>Merge Strategy</name>
</value>
</entry>
<entry>
<key>Merge Format</key>
<value>
<name>Merge Format</name>
</value>
</entry>
<entry>
<key>Attribute Strategy</key>
<value>
<name>Attribute Strategy</name>
</value>
</entry>
<entry>
<key>Correlation Attribute Name</key>
<value>
<name>Correlation Attribute Name</name>
</value>
</entry>
<entry>
<key>mergecontent-metadata-strategy</key>
<value>
<name>mergecontent-metadata-strategy</name>
</value>
</entry>
<entry>
<key>Minimum Number of Entries</key>
<value>
<name>Minimum Number of Entries</name>
</value>
</entry>
<entry>
<key>Maximum Number of Entries</key>
<value>
<name>Maximum Number of Entries</name>
</value>
</entry>
<entry>
<key>Minimum Group Size</key>
<value>
<name>Minimum Group Size</name>
</value>
</entry>
<entry>
<key>Maximum Group Size</key>
<value>
<name>Maximum Group Size</name>
</value>
</entry>
<entry>
<key>Max Bin Age</key>
<value>
<name>Max Bin Age</name>
</value>
</entry>
<entry>
<key>Maximum number of Bins</key>
<value>
<name>Maximum number of Bins</name>
</value>
</entry>
<entry>
<key>Delimiter Strategy</key>
<value>
<name>Delimiter Strategy</name>
</value>
</entry>
<entry>
<key>Header File</key>
<value>
<name>Header File</name>
</value>
</entry>
<entry>
<key>Footer File</key>
<value>
<name>Footer File</name>
</value>
</entry>
<entry>
<key>Demarcator File</key>
<value>
<name>Demarcator File</name>
</value>
</entry>
<entry>
<key>Compression Level</key>
<value>
<name>Compression Level</name>
</value>
</entry>
<entry>
<key>Keep Path</key>
<value>
<name>Keep Path</name>
</value>
</entry>
<entry>
<key>Tar Modified Time</key>
<value>
<name>Tar Modified Time</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Merge Strategy</key>
<value>Bin-Packing Algorithm</value>
</entry>
<entry>
<key>Merge Format</key>
<value>Binary Concatenation</value>
</entry>
<entry>
<key>Attribute Strategy</key>
<value>Keep Only Common Attributes</value>
</entry>
<entry>
<key>Correlation Attribute Name</key>
</entry>
<entry>
<key>mergecontent-metadata-strategy</key>
<value>Do Not Merge Uncommon Metadata</value>
</entry>
<entry>
<key>Minimum Number of Entries</key>
<value>50</value>
</entry>
<entry>
<key>Maximum Number of Entries</key>
<value>1000</value>
</entry>
<entry>
<key>Minimum Group Size</key>
<value>50 MB</value>
</entry>
<entry>
<key>Maximum Group Size</key>
</entry>
<entry>
<key>Max Bin Age</key>
<value>5 s</value>
</entry>
<entry>
<key>Maximum number of Bins</key>
<value>1</value>
</entry>
<entry>
<key>Delimiter Strategy</key>
<value>Filename</value>
</entry>
<entry>
<key>Header File</key>
</entry>
<entry>
<key>Footer File</key>
</entry>
<entry>
<key>Demarcator File</key>
</entry>
<entry>
<key>Compression Level</key>
<value>1</value>
</entry>
<entry>
<key>Keep Path</key>
<value>false</value>
</entry>
<entry>
<key>Tar Modified Time</key>
<value>${file.lastModifiedTime}</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>MergeContent</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>merged</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>original</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.MergeContent</type>
</processors>
<processors>
<id>c3c7aecc-217a-397d-0000-000000000000</id>
<parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>584.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.11.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Directory</key>
<value>
<name>Directory</name>
</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>
<name>Conflict Resolution Strategy</name>
</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>
<name>Create Missing Directories</name>
</value>
</entry>
<entry>
<key>Maximum File Count</key>
<value>
<name>Maximum File Count</name>
</value>
</entry>
<entry>
<key>Last Modified Time</key>
<value>
<name>Last Modified Time</name>
</value>
</entry>
<entry>
<key>Permissions</key>
<value>
<name>Permissions</name>
</value>
</entry>
<entry>
<key>Owner</key>
<value>
<name>Owner</name>
</value>
</entry>
<entry>
<key>Group</key>
<value>
<name>Group</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>Directory</key>
<value>C:\Users\503067805\Desktop\IDP Project\POC - Well Path\Output</value>
</entry>
<entry>
<key>Conflict Resolution Strategy</key>
<value>fail</value>
</entry>
<entry>
<key>Create Missing Directories</key>
<value>true</value>
</entry>
<entry>
<key>Maximum File Count</key>
</entry>
<entry>
<key>Last Modified Time</key>
</entry>
<entry>
<key>Permissions</key>
</entry>
<entry>
<key>Owner</key>
</entry>
<entry>
<key>Group</key>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>PutFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>failure</name>
</relationships>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.PutFile</type>
</processors>
</snippet>
<timestamp>02/12/2020 12:31:46 CST</timestamp>
</template>
... View more
Labels:
- Labels:
-
Apache NiFi
02-11-2020
09:01 PM
Hi,
I have csv files and I want to move the content of files along with its meta data (File name, source (To be hard coded), control number (Part of file name - to be extracted from file name itself) thru NiFi. So here is the sample File name and layout -
File name - 12345_user_data.csv (control_number_user_data.csv)
source - Newyork
CSV File Content/columns -
Fields - abc1, abc2, abc3, abc4
values - 1,2,3,4
Postgres Database table layout
Table name - User_Education
fields name -
control_number, file_name, source, abc1, abc2, abc3, abc4
Values -
12345, 12345_user_data.csv, Newyork, 1,2,3,4
I am planning to use below processors -
ListFile
FetchFile
UpdateAttributes
PutDatabaseRecords
LogAttributes
But I am not sure how to combine the actual content with the meta data to load into one single table. Please help
... View more
Labels:
- Labels:
-
Apache NiFi
02-11-2020
10:41 AM
Hi,
I am getting below error when I run run-nifi-registry.bat registry from windows 10
Error: Could not find or load main class Project\nifi-registry-0.4.0\bin\..\conf;C:\Users\503067805\Desktop\IDP
Do you know how to solve it?
... View more
Labels:
- Labels:
-
NiFi Registry
09-24-2017
03:34 AM
Hi Jay, Appreciate your help. I tried to do ssh using putty but got error. See the screenshots. Please suggest if I am doing anything wrong. Main issue is I am not able to open 127.0.0.1:8888 or 127.0.0.1:8080 on the browser. Please help
... View more
09-24-2017
02:43 AM
Thanks Jay for responding. I tried it but it says connection refused. Please see the screenshot. kindly help
... View more
09-24-2017
01:55 AM
Rajendra. Is your issue resolved? I am facing the same
... View more
09-24-2017
01:55 AM
I also have same issue. Struggling for last few days. Can some one help. I have tried on mac and windows both. Facing same issue.
... View more
09-24-2017
01:40 AM
Hi, I have installed virtual box (5.1.28) and imported hortonwork sandbox (HDP_2.6_1_docker). It start ok giving 3 options (Only last one works). I can then login to sandbox with id - root and pwd - hadoop. after that none of the command work (See the screenshot). even ifconfig -a or ipconfig does not work. Also unable to access 127.0.0.1:8888 or 127.0.0.1:8080 using browser. Tried all the solutions found online like - 1. Check port setting. It looks fine. 2. Changing the network adapter to "bridged adapter" and "Host Only Adapter". But none helped. 3. Shut of wirewall setting. It did not help either. I am dead in the water for last few days. Please help
... View more
Labels:
- Labels:
-
Apache Ambari
