Member since
04-08-2020
12
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9523 | 04-17-2020 08:11 AM |
04-17-2020
08:11 AM
In order to solve this problem I did the following: Added one of the postgres drives (https://jdbc.postgresql.org/download.html) to the nifi-1.11.4/lib/ folder I also added the file to a secondary folder within lib call /java11. I safely shut down nifi and restarted my linux system. I updated my DBCPConnectionPool to say the follow: Database Connection URL jdbc:postgresql:///forestdb Database Driver Class Name org.postgresql.Driver Database Driver Location(s) No value set Kerberos Credentials Service No value set Database User admin Password 123 Max Wait Time 500 millis Max Total Connections 8 Be sure you have your tables set up and all that good stuff. Notice that I no longer included the path to the drivers locations. This should be due to Nifi loading the jar files into the system itself, by us adding it to the nifi-1.11.4/lib folder. Adding the file to both the nifi-1.11.4/lib folder and the nifi-1.11.4/lib/java11 folder is probably not necessary but i was troubleshooting. also note that I changed the Database Connection URL, no longer having localhost as part of the url.
... View more
04-16-2020
02:17 PM
This seems to be an issue for many but I didn't see much in light for Linux systems. I've done it all picked up the drivers from https://jdbc.postgresql.org/download.html placed these drives in the nifi-1.11.4/lib/ folder. I've restarted Nifi, verified the database, verified the filepath, altered the path name with "file:///", altered the path to say just lib/postgresql-42.2.12.jar, attempted to use another database user. I know that the connection information is correct because I connected to the DataGrip UI. I'm completely stumped. Database Connection URL jdbc:postgresql://localhost:5432/forestdb Database Driver Class Name org/postgresql/Driver.class Database Driver Location(s) /home/fg/Documents/nifi-1.11.4-bin/nifi-1.11.4/lib/postgresql-42.2.12.jar Database User admin Password 123 Max Wait Time 500 millis 16:03:08 CDT
ERROR
01711097-9fd7-1476-e26e-a9bf77b2dff7
StandardControllerServiceNode[service=DBCPConnectionPool[id=01711097-9fd7-1476-e26e-a9bf77b2dff7], versionedComponentId=null, processGroup=StandardProcessGroup[identifier=6428af02-0171-1000-124e-9aeed8d62c82], active=true] Failed to invoke @OnEnabled method due to org.apache.nifi.reporting.InitializationException: Can't load Database Driver: Can't load Database Drive
... View more
Labels:
- Labels:
-
Apache NiFi
04-15-2020
07:54 AM
Thanks for spotting the mistake I made with the 'text' file. after making your suggested corrections: text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
json_data = json.loads(text)
try:
self.year.append(json_data['Year'])
except KeyError:
self.year.append(2000) I'm still incurring the following error: ExecuteScript[id=01711026-9fd7-1476-8c67-e730c921e164] ExecuteScript[id=01711026-9fd7-1476-8c67-e730c921e164] failed to process due to javax.script.ScriptException: ValueError: time data 'Y/e/a/r/m/o/n/t/h/D/O/Y/H/o/u/r' does not match format '%Y/%m/%d/%H' in <script> at line number 63; rolling back session: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: ValueError: time data 'Y/e/a/r/m/o/n/t/h/D/O/Y/H/o/u/r' does not match format '%Y/%m/%d/%H' in <script> at line number 63
... View more
04-15-2020
06:43 AM
Just getting this with that updated. TypeError: unicode indices must be integers in <script> at line number 63; rolling back session: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: shifted code to say something like try:
self.month.append(json_data[1])
except KeyError:
self.month.append(11) errors continue
... View more
04-14-2020
03:34 PM
My goal is to pass the data coming in as a JSON to my script and update it to UNIX time. using python. but it is saying line 63 is being referenced before called, "json_data". This is at the session writer near the bottom Processors being used is Convert Record -Execute Script--. The message being passed looks like this: json_data = {"Year":2018,"DOY":12,"Hour":20,"HGI_Lat_of_the_S/C":7.0,"IMF_B_scalar_nT":1.03,"SW_Plasma_Speed_KMs":441.0} and should pop out like this {"HGI_Lat_of_the_S/C": 7.0, "IMF_B_scalar_nT": 1.03, "SW_Plasma_Speed_KMs": 441.0, "Unix_time": 1542074400.0} Error message: ExecuteScript[id=01711026-9fd7-1476-8c67-e730c921e164] Failed to process session due to javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before assignment in <script> at line number 63: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before assignment in <script> at line number 63 Python code: import json
from datetime import datetime
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
self.year=[]
self.day=[]
self.hour=[]
self.month=[]
self.key1_year = 'Year'
self.key4_month = 'month'
self.key2_day = 'DOY'
self.key3_hour = 'Hour'
# Write bytes that are utf-8 encoded chine word.
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
#obj = json.loads(text)
#json_data = json.dumps(text)
json_data = json.loads(json_data)
try:
self.year.append(json_data[self.key1_year])
except KeyError:
self.year.append(2000)
try:
self.month.append(json_data[self.key4_month])
except KeyError:
self.month.append(11)
try:
self.day.append(json_data[self.key2_day])
except KeyError:
self.day.append(11)
try:
self.hour.append(json_data[self.key3_hour])
except KeyError:
self.hour.append(11)
new = year+month+day+hour
# Considering date is in mm/dd/yyyy format
#converting the appendd list to strings instead of ints
b=[str(x) for x in new]
#joining all the data without adding
b = '/'.join(b)
#convert to unix
dt_object2 = datetime.strptime(b, "%Y/%m/%d/%H")
timestamp = datetime.timestamp(dt_object2)
json_data.update({'Unix_time':timestamp})
#deleting unwanted data from the dict
for func in [self.key1_year,self.key4_month,self.key2_day,self.key3_hour]:
try:
del json_data[func]
except KeyError as e:
pass
outputStream.write(bytearray(json.dumps(json_data).encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile, PyStreamCallback())
session.transfer(flowFile, REL_SUCCESS)
... View more
Labels:
- Labels:
-
Apache NiFi
04-09-2020
08:22 PM
Is there something about configuring Nifi that needs to be done when setting up?
I ask because I've worked on a few different systems and the one I just set up seems to be having issues with Regex, prepend, and other items.
... View more
Labels:
- Labels:
-
Apache NiFi
04-08-2020
01:55 PM
1 Kudo
Thanks for the effort @stevenmatison I imagine it does have to do with configuration.
... View more
04-08-2020
12:35 PM
I've made these modification and all it does is route it to failed.
... View more
04-08-2020
10:33 AM
Unfortunately this method does not work @stevenmatison I using Replace Text but the only method that has worked so far is below. this has only lead to the following results: Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_var,Time_btwn_observation_sec,Field_magnutude_average_nT,BY_nT(GSM),BZ_nT_(GSM),RMS_SD_B_scalar_nT,RMS_SD_field_vector_nT,Speed_km/s,Alfven_mach_number,Magnetosonic_Mach_number,BSN_location_Xgse_Re2019,,,1,,0,,0,51,100,,,2788,,,,164,,0.12,999999,,,,5.11,,,,2.00,,,,2.73,,,,0.08,,,,1.03,,,451.0,,,9.8,,6.5,,,13.15,2019,,,1,,0,,1,51,100,,,2810,,,,159,,0.12,,,,,37,,,,5.10,,,,2.33,,,,2.58,,,,0.11,,,,1.04,,,451.3,,,9.8,,6.5,,,13.10,2019,,,1,,0,,2,51,,80,,,2852,,,,109,,0.09,,,,,18,,,,4.86,,,,2.37,,,,2.56,,,,0.12,,,,0.56,,,454.7,,10.3,,6.7,,,13.07,2019,,,1,,0,,3,51,,67,,,2951,,,,,66,,0.06,,,,-39,,,,4.78,,,,2.21,,,,2.55,,,,0.03,,,,0.33,,,452.3,,11.0,,6.8,,,13.00,2019,,,1,,0,,4,51,100,,,3025,,,,,,7,,0.00,,,,-13,,,,4.80,,,,2.17,,,,2.37,,,,0.03,,,,0.14,,,451.4,,11.2,,6.8,,,13.00,2019,,,1,,0,,5,99,,80,,,2973,,,,111,,0.09,,,,111,,,,4.94,,,,2.68,,,,2.39,,,,0.13,,,,0.55,99999.9,999.9,99.9,,,13.19,2019,,,1,,0,,6,51,,67,,,3074,,,,,20,,0.02,,,,-40,,,,4.88,,,,2.54,,,,2.01,,,,0.02,,,,0.28,,,451.0,,,9.8,,6.5,,,13.27,2019,,,1,,0,,7 Search Value \s Replacement Value , Character Set UTF-8 Maximum Buffer Size 1 MB Replacement Strategy Regex Replace Evaluation Mode Line-by-Line Line-by-Line Evaluation Mode All
... View more
04-08-2020
08:51 AM
Hi, I'm attempting to replace values in an ASCII file that im importing from a website. I've added the header with replace text but, I'm having a hard time replacing the spaces with commas so, I can convert the file into csv and later to json. Here is what I'm dealing with.
Edit: I should add that sometimes there is one space other times there are two spaces. As of now I have two spaces in the search attribute and a comma in the replace attribute works for part of this problem.
Value
Search Value
Replacement Value
,
Character Set
UTF-8
Maximum Buffer Size
1 MB
Replacement Strategy
Regex Replace
Evaluation Mode
Line-by-Line
Line-by-Line Evaluation Mode
All
Before:
Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_var,Time_btwn_observation_sec,Field_magnutude_average_nT,BY_nT(GSM),BZ_nT_(GSM),RMS_SD_B_scalar_nT,RMS_SD_field_vector_nT,Speed_km/s,Alfven_mach_number,Magnetosonic_Mach_number,BSN_location_Xgse_Re,2019 1 0 0 51 100 2788 164 0.12 999999 5.11 2.00 2.73 0.08 1.03 451.0 9.8 6.5 13.15 2019 1 0 1 51 100 2810 159 0.12 37 5.10 2.33 2.58 0.11 1.04 451.3 9.8 6.5 13.10 2019 1 0 2 51 80 2852 109 0.09 18 4.86 2.37 2.56 0.12 0.56 454.7 10.3 6.7 13.07 2019 1 0 3 51 67 2951 66 0.06 -39 4.78 2.21 2.55 0.03 0.33 452.3 11.0 6.8 13.00 2019 1 0 4 51 100 3025 7 0.00 -13 4.80 2.17 2.37 0.03 0.14 451.4 11.2 6.8 13.00 2019 1 0 5 99 80 2973 111 0.09 111 4.94 2.68 2.39 0.13 0.55 99999.9 999.9 99.9 13.19 2019 1 0 6 51 67 3074 20 0.02 -40 4.88 2.54 2.01 0.02 0.28 451.0 9.8 6.5 13.27 2019 1 0 7 51 50 3114 9 0.00 19 4.82 2.37 2.93 0.02 0.14 451.0 9.9 6.5 13.29 2019 1 0 8 99 999 999999 999999 99.99 999999 9999.99 9999.99 9999.99 9999.99 9999.99 99999.9 999.9 99.9 9999.99 2019 1 0 9 99 100 3036 0 0.00 999999 5.16 3.34 2.44 0.00 0.00 99999.9 999.9 99.9 13.24 2019 1 0 10 99 100 3036 0 0.00 60 5.16 3.34 2.43 0.00 0.00 99999.9 999.9 99.9 13.24
after:
Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_var,Time_btwn_observation_sec,Field_magnutude_average_nT,BY_nT(GSM),BZ_nT_(GSM),RMS_SD_B_scalar_nT,RMS_SD_field_vector_nT,Speed_km/s,Alfven_mach_number,Magnetosonic_Mach_number,BSN_location_Xgse_Re,2019, 1,0,0 51 100, 2788,,164,0.12 999999,,5.11,,2.00,,2.73,,0.08,,1.03, 451.0, 9.8,6.5, 13.15 2019, 1,0,1 51 100, 2810,,159,0.12,, 37,,5.10,,2.33,,2.58,,0.11,,1.04, 451.3, 9.8,6.5, 13.10 2019, 1,0,2 51,80, 2852,,109,0.09,, 18,,4.86,,2.37,,2.56,,0.12,,0.56, 454.7,10.3,6.7, 13.07 2019, 1,0,3 51,67, 2951,, 66,0.06,,-39,,4.78,,2.21,,2.55,,0.03,,0.33, 452.3,11.0,6.8, 13.00 2019, 1,0,4 51 100, 3025,,,7,0.00,,-13,,4.80,,2.17,,2.37,,0.03,,0.14, 451.4,11.2,6.8, 13.00 2019, 1,0,5 99,80, 2973,,111,0.09,,111,,4.94,,2.68,,2.39,,0.13,,0.55 99999.9 999.9 99.9, 13.19 2019, 1,0,6 51,67, 3074,, 20,0.02,,-40,,4.88,,2.54,,2.01,,0.02,,0.28, 451.0, 9.8,6.5, 13.27 2019, 1,0,7 51,50, 3114,,,9,0.00,, 19,,4.82,,2.37,,2.93,,0.02,,0.14, 451.0, 9.9,6.5, 13.29 2019, 1,0,8 99 999 999999 999999 99.99 999999 9999.99 9999.99 9999.99 9999.99 9999.99 99999.9 999.9 99.9 9999.99 2019, 1,0,9 99 100, 3036,,,0,0.00 999999,,5.16,,3.34,,2.44,,0.00,,0.00 99999.9 999.9 99.9, 13.24 2019, 1,0 10 99 100, 3036,,,0,0.00,, 60,,5.16,,3.34,,2.43,,0.00,,0.00 99999.9 999.9 99.9, 13.24 2019, 1,0 11 99 999 999999 999999 99.99 999999 9999.99 9999.99 9999.99 9999.99 9999.99 99999.9 999.9 99.9 9999.99 2019, 1,0 12 99 999 999999 999999 99.99 999999 9999.99 9999.99 9999.99 9999.99 9999.99 99999.9 999.9 99.9 9999.99
... View more
Labels:
- Labels:
-
Apache NiFi