I had been using ExecuteStreamCommand processor to run python scripts and successfully giving flowfile as input to python scripts using sys.stdin and flowfile output as sys.stdout. Now all of these python scripts to be converted to Pyspark. How can have flow file as input/output in ExecuteStreamCommand Processor. Sys.stdin and sys.stdout does not seems to be working with pyspark. Is there any other way? ... View more

I have a csv flowfile with single record. I need to create its file name based on couple of column values in the csv file. Can you please let me know how we can do it by using the column name only not the position of the column as column position may change. Example  CSV File Name , City, State, Country, Gender John, Dallas, Texas, USA, M   File name should be John_USA.csv   ... View more

I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code - Table Definition - CREATE TABLE `local_db`.`table_x` ( `Unique_code` varchar(50) NOT NULL, `city` varchar(200) DEFAULT NULL, `state` varchar(50) DEFAULT NULL, `population` bigint(20) DEFAULT NULL, `Govt` varchar(50) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; ``` input_file = "input_file" csv_data = csv.reader(open(input_file)) try: connection = mysql.connector.connect(host='localhost', database='local_db', user='root', password='root', port = '3306') cursor = connection.cursor() for row in csv_data: cursor.execute(""" INSERT INTO table_x(Unique_code,city,state,population,Govt) VALUES("%s", "%s", "%s","%s", "%s") ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \ population = VALUES(population),Govt = VALUES(Govt)""") connection.commit() print(cursor.rowcount, "Record inserted successfully into table_x") cursor.close() except mysql.connector.Error as error: print("Failed to insert record into table_x table {}".format(error)) finally: if (connection.is_connected()): connection.close() print("MySQL connection is closed") ``` But I am getting below error - Failed to insert record into table_x table 1366 (22007): Incorrect integer value: '%s' for column `local_db`.`table_x`.`population` at row 1 MySQL connection is closed In other thread it was suggested to change SET sql_mode = "" But its not an option for me since I would be running it on organization server which I can not change for this only. Please suggest what code changes I can do here to handle this situation. ... View more

Can someone please tell me how to do UPSERT using PutDatabaseRecord processors. I am using MariaDB ... View more

Hi All, This is what I am trying to achieve. all of them are csv files Flowfile (csv) - col1_raw, col2_raw, col3_raw, col4_raw 1 , 2 , 3 , 4 1 , 2 , 3 , 4   Look up file1 (csv) col_name_raw , col_name_business, class col1_raw , col1_business , Secret col2_raw , col2_business , Public col3_raw , col3_business , Secret col4_raw , col4_business , Public   Look up file2 (csv) class, action public, copy secret, nocopy   output (csv) of flowfile should be col2_business, col4_business 2 , 4 2 , 4   So basically I am filtering the columns of flowfile based on the action listed lookup file2. Also changing the raw column name to business column name as per lookup file1. In SQL it is very easy but in Nifi how can I achieve this using existing processors. Please help. ... View more

Hi, I have created a NiFi flow which take multiple input csv files (Each one have 5 sections) and then pass to executestreamcommand processor. Which eventually trigger a python script and python create output of 5 csv files (Which are basically merging of specific section from all the files). Now challenge is how to pass these 5 csv files back to NiFi flow which takes these files to an another executestreamcommand processor which will again trigger a python script to convert each of the 5 files to xml files. Here is the sequence 1. List File 2. Fetch File 3. ExecuteStreamCommand - Create 5 csv file from all the input files passed. 4. ExecuteStreamCommand - Convert 5 csv files to xml 5. PutFile - Put all 5 xml files to a folder I am unable to write the 5 csv files from step 3 back to NiFi flow so that next processor can process them. Please help.       ... View more

Hi, Can some one please give an example of STDIN and STDOUT in python script to take input from Nifi flowfile and paas the output file to NiFi flowfile again. I am using below script for STDIN and STDOUT. STDIN seems to be working fine but STDOUT is not working. Please help   f = sys.stdin reader = csv.reader(f) *********** ##Python code for creating the output file and passing it to STDOUT xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile   ... View more

My NiFi was working fine until today (After my laptop hanged and I restarted it). Now NiFi is unable to launch and gives below error - As of now I tried below - 1. Deleted  content, database, flowfile and provenance  repository and restart NiFi - Did not help 2. Restart my machine - Did not help 3. Uninstalled jre and installed jdk - Did not help ( It was working fine when I had jre until today) My current jdk location is C:\Program Files\Java\jdk-13.0.2 I am using nifi-1.11.0     ******************************************************************** 2020-02-15 01:21:04,851 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-02-15 01:21:04,852 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0 2020-02-15 01:21:04,852 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\Progra~1\Java\jdk-13.0.2\bin\java.exe -classpath C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\conf;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\javax.servlet-api-3.1.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jcl-over-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jetty-schemas-3.1.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\jul-to-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\log4j-over-slf4j-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\logback-classic-1.2.3.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\logback-core-1.2.3.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-api-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-framework-api-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-nar-utils-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-properties-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\nifi-runtime-1.11.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\slf4j-api-1.7.26.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\javax.activation-api-1.2.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\javax.annotation-api-1.3.2.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-api-2.3.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-core-2.3.0.jar;C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\lib\java11\jaxb-impl-2.3.0.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom   -Dsun.net.http.allowRestrictedHeaders=true   -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\.\conf\nifi.properties -Dnifi.bootstrap.listen.port=65326 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\\logs org.apache.nifi.NiFi WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.nifi.bootstrap.util.OSUtils (file:/C:/Users/503067805/Desktop/IDP_Project/nifi-1.11.0/lib/bootstrap/nifi-bootstrap-1.11.0.jar) to method   java.lang.ProcessImpl.pid() WARNING: Please consider reporting this to the maintainers of org.apache.nifi.bootstrap.util.OSUtils WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2020-02-15 01:21:04,879 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\run\nifi.pid; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-15 01:21:04,888 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read status file C:\Users\503067~1\Desktop\IDP_PR~1\NIFI-1~1.0\bin\..\run\nifi.status; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file 2020-02-15 01:21:04,894 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 8552   ... View more

Hi, I am running a small test where I want to merge (union) two CSV files of same format. my Mergecontent Processor is merge the file but it keep sending multiple merged file to next processor. I am sending the file to Mergecontent thru Listfile and Fetchfile and I can see they send only two files to Mergecontent but input to mergecontent show hundreds of files. Not sure why. Please help. Attaching the screenshots   <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.3"> <description></description> <groupId>3ed4e195-0170-1000-7be4-197989ceef99</groupId> <name>Merge CSV Files</name> <snippet> <connections> <id>2ca736d6-69cf-39d1-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>473.8800048828125</x> <y>869.5999755859375</y> </bends> <bends> <x>473.8800048828125</x> <y>919.5999755859375</y> </bends> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>6ad9f863-3108-3ed3-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>6ad9f863-3108-3ed3-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>5662111e-1e71-3ffd-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>1cf0fdfc-41f0-38e0-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>merged</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>65d7ef96-2d72-396c-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>6b194994-5bdb-3a7a-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>33c56676-058e-35fa-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>74ae6286-74b3-3a30-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>7463a2e7-4e73-3da3-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>6ad9f863-3108-3ed3-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>1cf0fdfc-41f0-38e0-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>addb7fb8-03da-37bd-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>7f36d436-85ae-332b-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>6ad9f863-3108-3ed3-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>d4b3f000-3ea4-3514-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>468.20001220703125</x> <y>236.1199951171875</y> </bends> <bends> <x>468.20001220703125</x> <y>286.1199951171875</y> </bends> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>33c56676-058e-35fa-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <selectedRelationships>not.found</selectedRelationships> <selectedRelationships>permission.denied</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>33c56676-058e-35fa-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>dc3e7cfe-efa4-3344-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>65d7ef96-2d72-396c-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>33c56676-058e-35fa-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>eef0c5c9-9123-3437-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>469.0</x> <y>462.0</y> </bends> <bends> <x>469.0</x> <y>512.0</y> </bends> <destination> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>65d7ef96-2d72-396c-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <selectedRelationships>original</selectedRelationships> <source> <groupId>bdc9724d-9179-3f54-0000-000000000000</groupId> <id>65d7ef96-2d72-396c-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <processors> <id>1cf0fdfc-41f0-38e0-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>624.0</y> </position> <bundle> <artifact>nifi-update-attribute-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Delete Attributes Expression</key> <value> <name>Delete Attributes Expression</name> </value> </entry> <entry> <key>Store State</key> <value> <name>Store State</name> </value> </entry> <entry> <key>Stateful Variables Initial Value</key> <value> <name>Stateful Variables Initial Value</name> </value> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value> <name>canonical-value-lookup-cache-size</name> </value> </entry> <entry> <key>filename</key> <value> <name>filename</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Delete Attributes Expression</key> </entry> <entry> <key>Store State</key> <value>Do not store state</value> </entry> <entry> <key>Stateful Variables Initial Value</key> </entry> <entry> <key>canonical-value-lookup-cache-size</key> <value>100</value> </entry> <entry> <key>filename</key> <value>${filename}.csv</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>UpdateAttribute</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.attributes.UpdateAttribute</type> </processors> <processors> <id>33c56676-058e-35fa-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>200.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>File to Fetch</key> <value> <name>File to Fetch</name> </value> </entry> <entry> <key>Completion Strategy</key> <value> <name>Completion Strategy</name> </value> </entry> <entry> <key>Move Destination Directory</key> <value> <name>Move Destination Directory</name> </value> </entry> <entry> <key>Move Conflict Strategy</key> <value> <name>Move Conflict Strategy</name> </value> </entry> <entry> <key>Log level when file not found</key> <value> <name>Log level when file not found</name> </value> </entry> <entry> <key>Log level when permission denied</key> <value> <name>Log level when permission denied</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>File to Fetch</key> <value>${absolute.path}/${filename}</value> </entry> <entry> <key>Completion Strategy</key> <value>None</value> </entry> <entry> <key>Move Destination Directory</key> </entry> <entry> <key>Move Conflict Strategy</key> <value>Rename</value> </entry> <entry> <key>Log level when file not found</key> <value>ERROR</value> </entry> <entry> <key>Log level when permission denied</key> <value>ERROR</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>FetchFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>not.found</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>permission.denied</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.FetchFile</type> </processors> <processors> <id>65d7ef96-2d72-396c-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>424.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Merge Strategy</key> <value> <name>Merge Strategy</name> </value> </entry> <entry> <key>Merge Format</key> <value> <name>Merge Format</name> </value> </entry> <entry> <key>Attribute Strategy</key> <value> <name>Attribute Strategy</name> </value> </entry> <entry> <key>Correlation Attribute Name</key> <value> <name>Correlation Attribute Name</name> </value> </entry> <entry> <key>mergecontent-metadata-strategy</key> <value> <name>mergecontent-metadata-strategy</name> </value> </entry> <entry> <key>Minimum Number of Entries</key> <value> <name>Minimum Number of Entries</name> </value> </entry> <entry> <key>Maximum Number of Entries</key> <value> <name>Maximum Number of Entries</name> </value> </entry> <entry> <key>Minimum Group Size</key> <value> <name>Minimum Group Size</name> </value> </entry> <entry> <key>Maximum Group Size</key> <value> <name>Maximum Group Size</name> </value> </entry> <entry> <key>Max Bin Age</key> <value> <name>Max Bin Age</name> </value> </entry> <entry> <key>Maximum number of Bins</key> <value> <name>Maximum number of Bins</name> </value> </entry> <entry> <key>Delimiter Strategy</key> <value> <name>Delimiter Strategy</name> </value> </entry> <entry> <key>Header File</key> <value> <name>Header File</name> </value> </entry> <entry> <key>Footer File</key> <value> <name>Footer File</name> </value> </entry> <entry> <key>Demarcator File</key> <value> <name>Demarcator File</name> </value> </entry> <entry> <key>Compression Level</key> <value> <name>Compression Level</name> </value> </entry> <entry> <key>Keep Path</key> <value> <name>Keep Path</name> </value> </entry> <entry> <key>Tar Modified Time</key> <value> <name>Tar Modified Time</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Merge Strategy</key> <value>Bin-Packing Algorithm</value> </entry> <entry> <key>Merge Format</key> <value>Binary Concatenation</value> </entry> <entry> <key>Attribute Strategy</key> <value>Keep Only Common Attributes</value> </entry> <entry> <key>Correlation Attribute Name</key> </entry> <entry> <key>mergecontent-metadata-strategy</key> <value>Do Not Merge Uncommon Metadata</value> </entry> <entry> <key>Minimum Number of Entries</key> <value>2</value> </entry> <entry> <key>Maximum Number of Entries</key> <value>1000</value> </entry> <entry> <key>Minimum Group Size</key> <value>5 kb</value> </entry> <entry> <key>Maximum Group Size</key> </entry> <entry> <key>Max Bin Age</key> <value>5 s</value> </entry> <entry> <key>Maximum number of Bins</key> <value>1</value> </entry> <entry> <key>Delimiter Strategy</key> <value>Filename</value> </entry> <entry> <key>Header File</key> </entry> <entry> <key>Footer File</key> </entry> <entry> <key>Demarcator File</key> </entry> <entry> <key>Compression Level</key> <value>1</value> </entry> <entry> <key>Keep Path</key> <value>false</value> </entry> <entry> <key>Tar Modified Time</key> <value>${file.lastModifiedTime}</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>MergeContent</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>merged</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>original</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.MergeContent</type> </processors> <processors> <id>6ad9f863-3108-3ed3-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>832.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Directory</key> <value> <name>Directory</name> </value> </entry> <entry> <key>Conflict Resolution Strategy</key> <value> <name>Conflict Resolution Strategy</name> </value> </entry> <entry> <key>Create Missing Directories</key> <value> <name>Create Missing Directories</name> </value> </entry> <entry> <key>Maximum File Count</key> <value> <name>Maximum File Count</name> </value> </entry> <entry> <key>Last Modified Time</key> <value> <name>Last Modified Time</name> </value> </entry> <entry> <key>Permissions</key> <value> <name>Permissions</name> </value> </entry> <entry> <key>Owner</key> <value> <name>Owner</name> </value> </entry> <entry> <key>Group</key> <value> <name>Group</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Directory</key> <value>C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Merge_Files\Output</value> </entry> <entry> <key>Conflict Resolution Strategy</key> <value>fail</value> </entry> <entry> <key>Create Missing Directories</key> <value>true</value> </entry> <entry> <key>Maximum File Count</key> </entry> <entry> <key>Last Modified Time</key> </entry> <entry> <key>Permissions</key> </entry> <entry> <key>Owner</key> </entry> <entry> <key>Group</key> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>PutFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.PutFile</type> </processors> <processors> <id>74ae6286-74b3-3a30-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>0.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Input Directory</key> <value> <name>Input Directory</name> </value> </entry> <entry> <key>listing-strategy</key> <value> <name>listing-strategy</name> </value> </entry> <entry> <key>Recurse Subdirectories</key> <value> <name>Recurse Subdirectories</name> </value> </entry> <entry> <key>Input Directory Location</key> <value> <name>Input Directory Location</name> </value> </entry> <entry> <key>File Filter</key> <value> <name>File Filter</name> </value> </entry> <entry> <key>Path Filter</key> <value> <name>Path Filter</name> </value> </entry> <entry> <key>Include File Attributes</key> <value> <name>Include File Attributes</name> </value> </entry> <entry> <key>Minimum File Age</key> <value> <name>Minimum File Age</name> </value> </entry> <entry> <key>Maximum File Age</key> <value> <name>Maximum File Age</name> </value> </entry> <entry> <key>Minimum File Size</key> <value> <name>Minimum File Size</name> </value> </entry> <entry> <key>Maximum File Size</key> <value> <name>Maximum File Size</name> </value> </entry> <entry> <key>Ignore Hidden Files</key> <value> <name>Ignore Hidden Files</name> </value> </entry> <entry> <key>target-system-timestamp-precision</key> <value> <name>target-system-timestamp-precision</name> </value> </entry> <entry> <key>et-state-cache</key> <value> <identifiesControllerService>org.apache.nifi.distributed.cache.client.DistributedMapCacheClient</identifiesControllerService> <name>et-state-cache</name> </value> </entry> <entry> <key>et-time-window</key> <value> <name>et-time-window</name> </value> </entry> <entry> <key>et-initial-listing-target</key> <value> <name>et-initial-listing-target</name> </value> </entry> <entry> <key>et-node-identifier</key> <value> <name>et-node-identifier</name> </value> </entry> <entry> <key>track-performance</key> <value> <name>track-performance</name> </value> </entry> <entry> <key>max-performance-metrics</key> <value> <name>max-performance-metrics</name> </value> </entry> <entry> <key>max-operation-time</key> <value> <name>max-operation-time</name> </value> </entry> <entry> <key>max-listing-time</key> <value> <name>max-listing-time</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Input Directory</key> <value>C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Merge_Files\Input</value> </entry> <entry> <key>listing-strategy</key> <value>timestamps</value> </entry> <entry> <key>Recurse Subdirectories</key> <value>true</value> </entry> <entry> <key>Input Directory Location</key> <value>Local</value> </entry> <entry> <key>File Filter</key> <value>[^\.].*</value> </entry> <entry> <key>Path Filter</key> </entry> <entry> <key>Include File Attributes</key> <value>true</value> </entry> <entry> <key>Minimum File Age</key> <value>0 sec</value> </entry> <entry> <key>Maximum File Age</key> </entry> <entry> <key>Minimum File Size</key> <value>0 B</value> </entry> <entry> <key>Maximum File Size</key> </entry> <entry> <key>Ignore Hidden Files</key> <value>true</value> </entry> <entry> <key>target-system-timestamp-precision</key> <value>auto-detect</value> </entry> <entry> <key>et-state-cache</key> </entry> <entry> <key>et-time-window</key> <value>3 hours</value> </entry> <entry> <key>et-initial-listing-target</key> <value>all</value> </entry> <entry> <key>et-node-identifier</key> <value>${hostname()}</value> </entry> <entry> <key>track-performance</key> <value>false</value> </entry> <entry> <key>max-performance-metrics</key> <value>100000</value> </entry> <entry> <key>max-operation-time</key> <value>10 secs</value> </entry> <entry> <key>max-listing-time</key> <value>3 mins</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>ListFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.ListFile</type> </processors> <processors> <id>7f36d436-85ae-332b-0000-000000000000</id> <parentGroupId>bdc9724d-9179-3f54-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>1032.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>windows-1252</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>LogAttribute</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> </snippet> <timestamp>02/13/2020 14:10:59 CST</timestamp> </template>   and the flow ... View more

Hi, I am looking for a sample NiFi template which can combine multiple csv files coming from single source into one csv file and then convert it into a xml file. Can some one please help in getting the template for this. Thanks Note - I have tried mergecontent processor to combine the csv files but it keeps generating 100s of non csv files not sure what is happening. Attaching the template I created for your reference.   <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.3"> <description></description> <groupId>3a0529f4-0170-1000-7792-3205dea567f8</groupId> <name>Merge CSV Files using MergeContent Processor</name> <snippet> <connections> <id>026df4be-1012-30bc-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>469.0</x> <y>455.0</y> </bends> <bends> <x>469.0</x> <y>505.0</y> </bends> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c300524c-bcd5-3925-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <selectedRelationships>original</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c300524c-bcd5-3925-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>884b9ed5-2ad3-3919-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <bends> <x>469.0</x> <y>623.0</y> </bends> <bends> <x>469.0</x> <y>673.0</y> </bends> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c3c7aecc-217a-397d-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>failure</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c3c7aecc-217a-397d-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>9810e9ce-09a4-37e5-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>083d8a61-3cc9-33c7-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c3c7aecc-217a-397d-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>b8698473-5613-35e5-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c3c7aecc-217a-397d-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>merged</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c300524c-bcd5-3925-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>cbc26047-48ce-39ab-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>67f0a86e-9bea-3891-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>968a0059-8b8b-31fc-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <connections> <id>e39e8309-ada9-308a-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>c300524c-bcd5-3925-0000-000000000000</id> <type>PROCESSOR</type> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression> <loadBalancePartitionAttribute></loadBalancePartitionAttribute> <loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus> <loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>ee4a5604-53c3-33b8-0000-000000000000</groupId> <id>67f0a86e-9bea-3891-0000-000000000000</id> <type>PROCESSOR</type> </source> <zIndex>0</zIndex> </connections> <processors> <id>083d8a61-3cc9-33c7-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>792.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Log Level</key> <value> <name>Log Level</name> </value> </entry> <entry> <key>Log Payload</key> <value> <name>Log Payload</name> </value> </entry> <entry> <key>Attributes to Log</key> <value> <name>Attributes to Log</name> </value> </entry> <entry> <key>attributes-to-log-regex</key> <value> <name>attributes-to-log-regex</name> </value> </entry> <entry> <key>Attributes to Ignore</key> <value> <name>Attributes to Ignore</name> </value> </entry> <entry> <key>attributes-to-ignore-regex</key> <value> <name>attributes-to-ignore-regex</name> </value> </entry> <entry> <key>Log prefix</key> <value> <name>Log prefix</name> </value> </entry> <entry> <key>character-set</key> <value> <name>character-set</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Log Level</key> <value>info</value> </entry> <entry> <key>Log Payload</key> <value>false</value> </entry> <entry> <key>Attributes to Log</key> </entry> <entry> <key>attributes-to-log-regex</key> <value>.*</value> </entry> <entry> <key>Attributes to Ignore</key> </entry> <entry> <key>attributes-to-ignore-regex</key> </entry> <entry> <key>Log prefix</key> </entry> <entry> <key>character-set</key> <value>windows-1252</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>LogAttribute</name> <relationships> <autoTerminate>true</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.LogAttribute</type> </processors> <processors> <id>67f0a86e-9bea-3891-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>192.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>File to Fetch</key> <value> <name>File to Fetch</name> </value> </entry> <entry> <key>Completion Strategy</key> <value> <name>Completion Strategy</name> </value> </entry> <entry> <key>Move Destination Directory</key> <value> <name>Move Destination Directory</name> </value> </entry> <entry> <key>Move Conflict Strategy</key> <value> <name>Move Conflict Strategy</name> </value> </entry> <entry> <key>Log level when file not found</key> <value> <name>Log level when file not found</name> </value> </entry> <entry> <key>Log level when permission denied</key> <value> <name>Log level when permission denied</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>File to Fetch</key> <value>${absolute.path}/${filename}</value> </entry> <entry> <key>Completion Strategy</key> <value>None</value> </entry> <entry> <key>Move Destination Directory</key> </entry> <entry> <key>Move Conflict Strategy</key> <value>Rename</value> </entry> <entry> <key>Log level when file not found</key> <value>ERROR</value> </entry> <entry> <key>Log level when permission denied</key> <value>ERROR</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>FetchFile</name> <relationships> <autoTerminate>true</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>true</autoTerminate> <name>not.found</name> </relationships> <relationships> <autoTerminate>true</autoTerminate> <name>permission.denied</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.FetchFile</type> </processors> <processors> <id>968a0059-8b8b-31fc-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>0.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Input Directory</key> <value> <name>Input Directory</name> </value> </entry> <entry> <key>listing-strategy</key> <value> <name>listing-strategy</name> </value> </entry> <entry> <key>Recurse Subdirectories</key> <value> <name>Recurse Subdirectories</name> </value> </entry> <entry> <key>Input Directory Location</key> <value> <name>Input Directory Location</name> </value> </entry> <entry> <key>File Filter</key> <value> <name>File Filter</name> </value> </entry> <entry> <key>Path Filter</key> <value> <name>Path Filter</name> </value> </entry> <entry> <key>Include File Attributes</key> <value> <name>Include File Attributes</name> </value> </entry> <entry> <key>Minimum File Age</key> <value> <name>Minimum File Age</name> </value> </entry> <entry> <key>Maximum File Age</key> <value> <name>Maximum File Age</name> </value> </entry> <entry> <key>Minimum File Size</key> <value> <name>Minimum File Size</name> </value> </entry> <entry> <key>Maximum File Size</key> <value> <name>Maximum File Size</name> </value> </entry> <entry> <key>Ignore Hidden Files</key> <value> <name>Ignore Hidden Files</name> </value> </entry> <entry> <key>target-system-timestamp-precision</key> <value> <name>target-system-timestamp-precision</name> </value> </entry> <entry> <key>et-state-cache</key> <value> <identifiesControllerService>org.apache.nifi.distributed.cache.client.DistributedMapCacheClient</identifiesControllerService> <name>et-state-cache</name> </value> </entry> <entry> <key>et-time-window</key> <value> <name>et-time-window</name> </value> </entry> <entry> <key>et-initial-listing-target</key> <value> <name>et-initial-listing-target</name> </value> </entry> <entry> <key>et-node-identifier</key> <value> <name>et-node-identifier</name> </value> </entry> <entry> <key>track-performance</key> <value> <name>track-performance</name> </value> </entry> <entry> <key>max-performance-metrics</key> <value> <name>max-performance-metrics</name> </value> </entry> <entry> <key>max-operation-time</key> <value> <name>max-operation-time</name> </value> </entry> <entry> <key>max-listing-time</key> <value> <name>max-listing-time</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Input Directory</key> <value>C:\Users\503067805\Desktop\IDP Project\POC - Well Path\Input</value> </entry> <entry> <key>listing-strategy</key> <value>timestamps</value> </entry> <entry> <key>Recurse Subdirectories</key> <value>true</value> </entry> <entry> <key>Input Directory Location</key> <value>Local</value> </entry> <entry> <key>File Filter</key> <value>[^\.].*</value> </entry> <entry> <key>Path Filter</key> </entry> <entry> <key>Include File Attributes</key> <value>true</value> </entry> <entry> <key>Minimum File Age</key> <value>0 sec</value> </entry> <entry> <key>Maximum File Age</key> </entry> <entry> <key>Minimum File Size</key> <value>0 B</value> </entry> <entry> <key>Maximum File Size</key> </entry> <entry> <key>Ignore Hidden Files</key> <value>true</value> </entry> <entry> <key>target-system-timestamp-precision</key> <value>auto-detect</value> </entry> <entry> <key>et-state-cache</key> </entry> <entry> <key>et-time-window</key> <value>3 hours</value> </entry> <entry> <key>et-initial-listing-target</key> <value>all</value> </entry> <entry> <key>et-node-identifier</key> <value>${hostname()}</value> </entry> <entry> <key>track-performance</key> <value>false</value> </entry> <entry> <key>max-performance-metrics</key> <value>100000</value> </entry> <entry> <key>max-operation-time</key> <value>10 secs</value> </entry> <entry> <key>max-listing-time</key> <value>3 mins</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>ListFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.ListFile</type> </processors> <processors> <id>c300524c-bcd5-3925-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>416.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Merge Strategy</key> <value> <name>Merge Strategy</name> </value> </entry> <entry> <key>Merge Format</key> <value> <name>Merge Format</name> </value> </entry> <entry> <key>Attribute Strategy</key> <value> <name>Attribute Strategy</name> </value> </entry> <entry> <key>Correlation Attribute Name</key> <value> <name>Correlation Attribute Name</name> </value> </entry> <entry> <key>mergecontent-metadata-strategy</key> <value> <name>mergecontent-metadata-strategy</name> </value> </entry> <entry> <key>Minimum Number of Entries</key> <value> <name>Minimum Number of Entries</name> </value> </entry> <entry> <key>Maximum Number of Entries</key> <value> <name>Maximum Number of Entries</name> </value> </entry> <entry> <key>Minimum Group Size</key> <value> <name>Minimum Group Size</name> </value> </entry> <entry> <key>Maximum Group Size</key> <value> <name>Maximum Group Size</name> </value> </entry> <entry> <key>Max Bin Age</key> <value> <name>Max Bin Age</name> </value> </entry> <entry> <key>Maximum number of Bins</key> <value> <name>Maximum number of Bins</name> </value> </entry> <entry> <key>Delimiter Strategy</key> <value> <name>Delimiter Strategy</name> </value> </entry> <entry> <key>Header File</key> <value> <name>Header File</name> </value> </entry> <entry> <key>Footer File</key> <value> <name>Footer File</name> </value> </entry> <entry> <key>Demarcator File</key> <value> <name>Demarcator File</name> </value> </entry> <entry> <key>Compression Level</key> <value> <name>Compression Level</name> </value> </entry> <entry> <key>Keep Path</key> <value> <name>Keep Path</name> </value> </entry> <entry> <key>Tar Modified Time</key> <value> <name>Tar Modified Time</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Merge Strategy</key> <value>Bin-Packing Algorithm</value> </entry> <entry> <key>Merge Format</key> <value>Binary Concatenation</value> </entry> <entry> <key>Attribute Strategy</key> <value>Keep Only Common Attributes</value> </entry> <entry> <key>Correlation Attribute Name</key> </entry> <entry> <key>mergecontent-metadata-strategy</key> <value>Do Not Merge Uncommon Metadata</value> </entry> <entry> <key>Minimum Number of Entries</key> <value>50</value> </entry> <entry> <key>Maximum Number of Entries</key> <value>1000</value> </entry> <entry> <key>Minimum Group Size</key> <value>50 MB</value> </entry> <entry> <key>Maximum Group Size</key> </entry> <entry> <key>Max Bin Age</key> <value>5 s</value> </entry> <entry> <key>Maximum number of Bins</key> <value>1</value> </entry> <entry> <key>Delimiter Strategy</key> <value>Filename</value> </entry> <entry> <key>Header File</key> </entry> <entry> <key>Footer File</key> </entry> <entry> <key>Demarcator File</key> </entry> <entry> <key>Compression Level</key> <value>1</value> </entry> <entry> <key>Keep Path</key> <value>false</value> </entry> <entry> <key>Tar Modified Time</key> <value>${file.lastModifiedTime}</value> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>MergeContent</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>merged</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>original</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.MergeContent</type> </processors> <processors> <id>c3c7aecc-217a-397d-0000-000000000000</id> <parentGroupId>ee4a5604-53c3-33b8-0000-000000000000</parentGroupId> <position> <x>0.0</x> <y>584.0</y> </position> <bundle> <artifact>nifi-standard-nar</artifact> <group>org.apache.nifi</group> <version>1.11.0</version> </bundle> <config> <bulletinLevel>WARN</bulletinLevel> <comments></comments> <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount> <descriptors> <entry> <key>Directory</key> <value> <name>Directory</name> </value> </entry> <entry> <key>Conflict Resolution Strategy</key> <value> <name>Conflict Resolution Strategy</name> </value> </entry> <entry> <key>Create Missing Directories</key> <value> <name>Create Missing Directories</name> </value> </entry> <entry> <key>Maximum File Count</key> <value> <name>Maximum File Count</name> </value> </entry> <entry> <key>Last Modified Time</key> <value> <name>Last Modified Time</name> </value> </entry> <entry> <key>Permissions</key> <value> <name>Permissions</name> </value> </entry> <entry> <key>Owner</key> <value> <name>Owner</name> </value> </entry> <entry> <key>Group</key> <value> <name>Group</name> </value> </entry> </descriptors> <executionNode>ALL</executionNode> <lossTolerant>false</lossTolerant> <penaltyDuration>30 sec</penaltyDuration> <properties> <entry> <key>Directory</key> <value>C:\Users\503067805\Desktop\IDP Project\POC - Well Path\Output</value> </entry> <entry> <key>Conflict Resolution Strategy</key> <value>fail</value> </entry> <entry> <key>Create Missing Directories</key> <value>true</value> </entry> <entry> <key>Maximum File Count</key> </entry> <entry> <key>Last Modified Time</key> </entry> <entry> <key>Permissions</key> </entry> <entry> <key>Owner</key> </entry> <entry> <key>Group</key> </entry> </properties> <runDurationMillis>0</runDurationMillis> <schedulingPeriod>0 sec</schedulingPeriod> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> <yieldDuration>1 sec</yieldDuration> </config> <executionNodeRestricted>false</executionNodeRestricted> <name>PutFile</name> <relationships> <autoTerminate>false</autoTerminate> <name>failure</name> </relationships> <relationships> <autoTerminate>false</autoTerminate> <name>success</name> </relationships> <state>STOPPED</state> <style/> <type>org.apache.nifi.processors.standard.PutFile</type> </processors> </snippet> <timestamp>02/12/2020 12:31:46 CST</timestamp> </template>     ... View more

Hi, I am getting below error when I run run-nifi-registry.bat registry from windows 10 Error: Could not find or load main class Project\nifi-registry-0.4.0\bin\..\conf;C:\Users\503067805\Desktop\IDP Do you know how to solve it? ... View more

Hi, I have installed virtual box (5.1.28) and imported hortonwork sandbox (HDP_2.6_1_docker). It start ok giving 3 options (Only last one works). I can then login to sandbox with id - root and pwd - hadoop. after that none of the command work (See the screenshot). even ifconfig -a or ipconfig does not work. Also unable to access 127.0.0.1:8888 or 127.0.0.1:8080 using browser. Tried all the solutions found online like - 1. Check port setting. It looks fine. 2. Changing the network adapter to "bridged adapter" and "Host Only Adapter". But none helped. 3. Shut of wirewall setting. It did not help either. I am dead in the water for last few days. Please help ... View more