Member since
09-24-2017
28
Posts
0
Kudos Received
0
Solutions
09-27-2020
09:20 PM
I am running a NiFi flow where I have InvokeHTTP Processor calling a python script thru flask API. When I am processing many files (in 100's) then some files like (1-2%) fails with java.net.Socket.Exception:Broken Pipe (Write Failed) error in InvokeHTTP Processor. I have observed this happening more when I am running two separate flows accessing same flask API but different python script. What do you think I should change to avoid it. It only happens once in a while for only few files (1-2%) of total flow file.
... View more
Labels:
- Labels:
-
Apache NiFi
08-15-2020
09:21 PM
I have a incoming csv file which I need to route to different nifi flows based on the values in one of the column. Each csv has a column called "Country" which has same country name in all of its rows. What I want is File 1 - 50 rows (All rows will only one value in country column which will be any of the below) England, england, Britain, UK, United kingdom, USA, United States So the file which contain values like England, england, Britain, UK, United kingdom in country column should be routed to one flow and if file contain USA or United States then it should be routed to another flow. Note - One file will only contain one value in all rows. So routing based on first row value of country column for entire file is fine. How to achieve this?
... View more
Labels:
- Labels:
-
Apache NiFi
05-28-2020
05:35 PM
I have a csv flowfile with single record. I need to create its file name based on couple of column values in the csv file. Can you please let me know how we can do it by using the column name only not the position of the column as column position may change. Example CSV File Name , City, State, Country, Gender John, Dallas, Texas, USA, M File name should be John_USA.csv
... View more
Labels:
- Labels:
-
Apache NiFi
05-24-2020
09:02 PM
I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code - Table Definition - CREATE TABLE `local_db`.`table_x` ( `Unique_code` varchar(50) NOT NULL, `city` varchar(200) DEFAULT NULL, `state` varchar(50) DEFAULT NULL, `population` bigint(20) DEFAULT NULL, `Govt` varchar(50) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; ``` input_file = "input_file" csv_data = csv.reader(open(input_file)) try: connection = mysql.connector.connect(host='localhost', database='local_db', user='root', password='root', port = '3306') cursor = connection.cursor() for row in csv_data: cursor.execute(""" INSERT INTO table_x(Unique_code,city,state,population,Govt) VALUES("%s", "%s", "%s","%s", "%s") ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \ population = VALUES(population),Govt = VALUES(Govt)""") connection.commit() print(cursor.rowcount, "Record inserted successfully into table_x") cursor.close() except mysql.connector.Error as error: print("Failed to insert record into table_x table {}".format(error)) finally: if (connection.is_connected()): connection.close() print("MySQL connection is closed") ``` But I am getting below error - Failed to insert record into table_x table 1366 (22007): Incorrect integer value: '%s' for column `local_db`.`table_x`.`population` at row 1 MySQL connection is closed In other thread it was suggested to change SET sql_mode = "" But its not an option for me since I would be running it on organization server which I can not change for this only. Please suggest what code changes I can do here to handle this situation.
... View more
Labels:
- Labels:
-
Apache NiFi
05-20-2020
10:11 AM
Hi Steven, Thanks a lot for responding to my query. I helps a lot. I have downloaded your flow. I have few clarification questions. 1. In Updateattribute processor in your template - It is not adding any flow file attribute. So we need to add Code and Continent there right? 2. So we need to add anything in the properites of invokehttp processor? We will get response from invokeHTTP and that response needs to replace the values of longitude and latitude in original flow file. How to do that? 3. If original flow file is a csv file instead of json. Will the same flow work or do we need to convert csv to json first? csv file is like this Company_name, City, Country, Latitude, Longitude Abc Corp, Newyork, USA, 98.765, 67.5656
... View more
05-19-2020
11:35 PM
Hi, I have a flowfile json request which looks like this. Actually flowfile is a csv file which I am converting to json first so that another json request can be created for invokehttp processor. But if this csv to json conversion is not required for it then I would prefer to keep it csv only. { "type": "record", "name": "Location", "fields": [{ "name": "Company_name", "type": ["null", "string"] }, { "name": "City", "type": ["null", "string"] }, { "name": "Country", "type": ["null", "string"] }, { "name": "Latitude", "type": ["null", "string"] }, { "name": "Longitude", "type": ["null", "string"] } ] } I need to send a request to invokehttp processor which look like below - { "Code":"A-5678", "Latitude":"78.495603", "Longitude":"5.969756", "Continent":"" } Latitude and longitude is to be taken from flowfile fields while Code and Continent (To be sent as blank string) are not available in flowfile and has to be created. How can I create this custom request. I tried evaluatejsonpath processor but unable to add Continent as it does not take blank value. Once the request is received I need to replace the Latitude and Logitude values in flow file with the response received. Please let me know how to achieve it. The final output has to be csv file. @Shu_ashu @mburgess Please help
... View more
Labels:
- Labels:
-
Apache NiFi
02-17-2020
10:28 PM
Hi Alim, Your solution works fine for me except the xml file content is coming in one single line instead of blocks. Any idea why?
... View more
02-16-2020
01:11 PM
Hi Matt, I am able to trigger python script now but stuck on how to pass the output file to STDout so that it can come back to flowfile. Here is what I am trying. STDIN seems to be working fine but STDOUT is not working f = sys.stdin reader = csv.reader(f) *********** ##Python code for creating the output file and passing it to STDOUT xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile
... View more
02-16-2020
01:08 PM
Hi,
Can some one please give an example of STDIN and STDOUT in python script to take input from Nifi flowfile and paas the output file to NiFi flowfile again. I am using below script for STDIN and STDOUT. STDIN seems to be working fine but STDOUT is not working. Please help
f = sys.stdin reader = csv.reader(f)
***********
##Python code for creating the output file and passing it to STDOUT
xmlfile = open("C:\\Output_from_python\\output_file.xml",'w') xmlFile.write(doc.toprettyxml(indent = '\t')) xmlFile.close() sys.stdout = xmlFile
... View more
Labels:
- Labels:
-
Apache NiFi
02-15-2020
01:09 AM
Hi Matt, I am able to execute python script from command line. Executescript processor is not an option here because it is an experimental process. I am looking for something more robust which can handle heavy data and also I should be able to put the flow in production. Can you please see the settings in my flow and see if I am doing anything wrong here?
... View more