Member since
09-24-2017
28
Posts
0
Kudos Received
0
Solutions
08-15-2020
09:21 PM
I have a incoming csv file which I need to route to different nifi flows based on the values in one of the column. Each csv has a column called "Country" which has same country name in all of its rows. What I want is File 1 - 50 rows (All rows will only one value in country column which will be any of the below) England, england, Britain, UK, United kingdom, USA, United States So the file which contain values like England, england, Britain, UK, United kingdom in country column should be routed to one flow and if file contain USA or United States then it should be routed to another flow. Note - One file will only contain one value in all rows. So routing based on first row value of country column for entire file is fine. How to achieve this?
... View more
Labels:
- Labels:
-
Apache NiFi
05-24-2020
09:02 PM
I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code - Table Definition - CREATE TABLE `local_db`.`table_x` ( `Unique_code` varchar(50) NOT NULL, `city` varchar(200) DEFAULT NULL, `state` varchar(50) DEFAULT NULL, `population` bigint(20) DEFAULT NULL, `Govt` varchar(50) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; ``` input_file = "input_file" csv_data = csv.reader(open(input_file)) try: connection = mysql.connector.connect(host='localhost', database='local_db', user='root', password='root', port = '3306') cursor = connection.cursor() for row in csv_data: cursor.execute(""" INSERT INTO table_x(Unique_code,city,state,population,Govt) VALUES("%s", "%s", "%s","%s", "%s") ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \ population = VALUES(population),Govt = VALUES(Govt)""") connection.commit() print(cursor.rowcount, "Record inserted successfully into table_x") cursor.close() except mysql.connector.Error as error: print("Failed to insert record into table_x table {}".format(error)) finally: if (connection.is_connected()): connection.close() print("MySQL connection is closed") ``` But I am getting below error - Failed to insert record into table_x table 1366 (22007): Incorrect integer value: '%s' for column `local_db`.`table_x`.`population` at row 1 MySQL connection is closed In other thread it was suggested to change SET sql_mode = "" But its not an option for me since I would be running it on organization server which I can not change for this only. Please suggest what code changes I can do here to handle this situation.
... View more
Labels:
- Labels:
-
Apache NiFi
05-20-2020
10:11 AM
Hi Steven, Thanks a lot for responding to my query. I helps a lot. I have downloaded your flow. I have few clarification questions. 1. In Updateattribute processor in your template - It is not adding any flow file attribute. So we need to add Code and Continent there right? 2. So we need to add anything in the properites of invokehttp processor? We will get response from invokeHTTP and that response needs to replace the values of longitude and latitude in original flow file. How to do that? 3. If original flow file is a csv file instead of json. Will the same flow work or do we need to convert csv to json first? csv file is like this Company_name, City, Country, Latitude, Longitude Abc Corp, Newyork, USA, 98.765, 67.5656
... View more
05-19-2020
11:35 PM
Hi, I have a flowfile json request which looks like this. Actually flowfile is a csv file which I am converting to json first so that another json request can be created for invokehttp processor. But if this csv to json conversion is not required for it then I would prefer to keep it csv only. { "type": "record", "name": "Location", "fields": [{ "name": "Company_name", "type": ["null", "string"] }, { "name": "City", "type": ["null", "string"] }, { "name": "Country", "type": ["null", "string"] }, { "name": "Latitude", "type": ["null", "string"] }, { "name": "Longitude", "type": ["null", "string"] } ] } I need to send a request to invokehttp processor which look like below - { "Code":"A-5678", "Latitude":"78.495603", "Longitude":"5.969756", "Continent":"" } Latitude and longitude is to be taken from flowfile fields while Code and Continent (To be sent as blank string) are not available in flowfile and has to be created. How can I create this custom request. I tried evaluatejsonpath processor but unable to add Continent as it does not take blank value. Once the request is received I need to replace the Latitude and Logitude values in flow file with the response received. Please let me know how to achieve it. The final output has to be csv file. @Shu_ashu @mburgess Please help
... View more
Labels:
- Labels:
-
Apache NiFi
02-17-2020
10:28 PM
Hi Alim, Your solution works fine for me except the xml file content is coming in one single line instead of blocks. Any idea why?
... View more
02-16-2020
01:11 PM
Hi Matt, I am able to trigger python script now but stuck on how to pass the output file to STDout so that it can come back to flowfile. Here is what I am trying. STDIN seems to be working fine but STDOUT is not working f = sys.stdin reader = csv.reader(f) *********** ##Python code for creating the output file and passing it to STDOUT xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile
... View more
02-15-2020
01:09 AM
Hi Matt, I am able to execute python script from command line. Executescript processor is not an option here because it is an experimental process. I am looking for something more robust which can handle heavy data and also I should be able to put the flow in production. Can you please see the settings in my flow and see if I am doing anything wrong here?
... View more
02-13-2020
12:07 PM
Hi Mat I can execute the script from spyder (Anaconda) and it is successful. From executestreamcommand processor its not even triggering the script it seems. Its a .py file in the folder. Can this be an issue?
... View more
02-13-2020
12:05 PM
Thank you very much
... View more
02-12-2020
09:23 PM
I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help
Below is the error in bulletin board
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
23:21:24 CST
ERROR
3bc05a73-0170-1000-6f23-80001cc130eb
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
... View more
Labels:
- Labels:
-
Apache NiFi