Member since
07-19-2018
613
Posts
100
Kudos Received
117
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3056 | 01-11-2021 05:54 AM | |
2185 | 01-11-2021 05:52 AM | |
5737 | 01-08-2021 05:23 AM | |
5428 | 01-04-2021 04:08 AM | |
25066 | 12-18-2020 05:42 AM |
09-28-2020
03:41 PM
@surajnag This is a great question. Here are some of my ideas which I have used in development and production nifi flows. During development route all flow exceptions to an output port. I call these EOL, or End of Line and increment them with #s, like EOL1, EOL2, EOL3. I use these to hold failure, retry, original, no-retry, etc type outbound connections during testing. As I move to production, some of these may be auto terminated, and some may remain. In production, I route ALL major points of failure which are not auto terminated to an output port called ERROR. In my flows I sometimes have multiple ERROR output ports for difference purposes. Depending on the use case, the error is sent to some type of event notification. For Example: send an email, or post to a slack channel. In other cases these are routed to a process group that is in a stopped/disabled state. Based on the error, I may make some change in the flow, and then enable/start the flow to create a method to REPLAY that flowfile. In reference to successful end of flow, I may just auto terminate, as I want to assume the flow is always on, and everything has succeeded if it is not routed as above in #1 or #2. In other cases the bottom or end of the flow is also routed to some type of Event Notification. However, be careful here, as you could create tons of notifications depending on your flow. I hope some of these ideas are helpful. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-25-2020
05:27 AM
@Tokolosk JAVA_HOME is not required for nifi to operate. [root@c7401 nifi-1.12.0]# ./bin/nifi.sh start
nifi.sh: JAVA_HOME not set; results may vary
Java home:
NiFi home: /root/nifi-1.12.0
Bootstrap Config File: /root/nifi-1.12.0/conf/bootstrap.conf The only commands in my node are (not ubuntu but should be similar): yum install java java-devel wget http://apache.mirrors.hoobly.com/nifi/1.12.0/nifi-1.12.0-bin.zip unzip nifi-1.12.0-bin.zip cd nifi-1.12.0 ./bin/nifi.sh start tail -f logs/nifi-app.log .... 2020-09-25 12:15:14,828 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2020-09-25 12:15:14,828 INFO [main] org.apache.nifi.web.server.JettyServer http://192.168.74.101:8080/nifi
2020-09-25 12:15:14,828 INFO [main] org.apache.nifi.web.server.JettyServer http://10.0.2.15:8080/nifi
2020-09-25 12:15:14,828 INFO [main] org.apache.nifi.web.server.JettyServer http://127.0.0.1:8080/nifi If you are having problems, be sure to tail logs/nifi-app.log and look for errors. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-24-2020
06:34 AM
1 Kudo
@MKS_AWS There are a few ways to break up JSON within a flowfile (splitJson,QueryRecord). However if its just one giant blob of JSON you may not find that very useful. Perhaps you can share some sample json to that effect. Check out this way to do SNS up to 2gb per payload: https://github.com/awslabs/amazon-sns-java-extended-client-lib If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-24-2020
06:16 AM
Check that the user executing sqoop is able to write to the target directory. You may need to create a service user and directory with proper permissions since the hdfs user cannot write to the intended directory. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-24-2020
06:01 AM
@wenfeng Check that the user hue is able to write to the target directory. You may need to create a hue user directory with proper permissions if the hue user cannot write to the hive user directory. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-24-2020
05:56 AM
@ravi_sh_DS This gets a bit high level, so forgive me, as I am not sure how you know which ID to change and what to change it too. That said, your approach could be to use QueryRecord and find the match you want, then update that match with UpdateRecord. You can also split the json image array with SplitJson, then use UpdateRecord as suggested above. In either method depending on your Use Case when you split the records and process the splits separately you may need to rejoin them downstream. Some older methods useful here are SplitJson, EvaluateJson, UpdateAttribute, AttributeToJson, but the Query Update Records are now preferred as it is possible to do things more dynamically.
... View more
09-23-2020
04:50 PM
@ravi_sh_DS The solution you are looking for is UpdateRecord. Here is a great article with full info: https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267 If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-23-2020
04:46 PM
@alex15 I suspect the issue you have is that nifi use is not able to execute the script. Make sure the user ownership of the file is correct, also confirm read/write permissions. In unix/linux these are chown and chmod. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
09-19-2020
05:10 AM
1 Kudo
@Kilynn The solution you are looking for here is QueryRecord using JsonRecordReader. With this QueryRecord processor and record reader/writers configured, you can click + in QueryRecord to create a new key => value property. In the value you can write a SQL like query against the results in the flowfile. For example: DATA_MESSAGE => SELECT logEvents FROM FLOWFILE This will create all of your message objects as individual flowfles directed away from QueryRecord for the relationship "DATA_MESSAGE". From here you can process each message according to your use case. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
09-16-2020
04:58 AM
1. Check on the user permission of the jar file you added to your class path. Make sure jar file is in right place. 2. When you add driver manually also add this to sqoop command --driver = org.postgresql.Driver
... View more