Member since
07-19-2018
613
Posts
101
Kudos Received
117
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5092 | 01-11-2021 05:54 AM | |
| 3421 | 01-11-2021 05:52 AM | |
| 8789 | 01-08-2021 05:23 AM | |
| 8383 | 01-04-2021 04:08 AM | |
| 36681 | 12-18-2020 05:42 AM |
06-10-2020
04:11 AM
@vigneshvenu You will need to send parameters to hive during execution of query. You can find some good conversation about this here: https://community.cloudera.com/t5/Support-Questions/How-are-number-of-mappers-determined-for-a-query-with-hive/m-p/94915 If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
06-10-2020
03:54 AM
@Mondi Yes you can move the paths. Just be sure to shut down hdfs, move the data, then update the paths in hdfs config. Additionally you can also symlink the old path to the new path. Be sure to complete this on all nodes reporting the limited disk size. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
06-10-2020
03:50 AM
@sbharadi You are going to need to do some Performance Tuning and High Availability in Hue to increase the speed and response of the UI for many concurrent users. I do not think a single instance will satisfy your users. You can find some info on the follow link: https://gethue.com/performance-tuning/ These are older articles (check deeper links) and some to do with Cloudera Manager, but the concepts should still apply, especially in terms of basic hue performance ideas; memory, more instances, load balancer in front of more instances, and even adjustments to Hive itself. Last but not least, you may find a better response over at the Hue Discourse. @Romain is active here but not always watching. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
06-10-2020
03:42 AM
@ShuwnYuan Thats a pretty big query. If you do not have enough memory in yarn available to the containers building the query it will fail with this error. You are going to need to increase the tez container size in Hive Configuration.
... View more
06-05-2020
05:20 AM
@JonnyL The sample above provided exact output you requested. Please inspect the template and inspect FlowFiles for root files and subdirectories files. The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed. For any files in root folder, then path is "./". Additionally the absolute path attribute is also there for all the files. This attribute can be acted against too with deeper expression language, but I found path was easier.
... View more
06-04-2020
05:40 AM
1 Kudo
Do you have the concurrency of the python set to 8 too? With only 2 cores, and concurrency set to 8, you are still in realms of pushing the limits of stability with the number of active threads. When I am building a flow, i never touch concurrency until I am ready to run very specific unit tests against each processor where I am adjusting scheduling, queue properties, and concurrency. This is done with nifi clusters with 10s,50s,100s of cores. With a dev system of 1,2, or 3 nodes and very small number of cores I do not do any tuning, just logical testing without time or speed expectations because it will break....
... View more
06-04-2020
05:29 AM
@renuu You have to build the final values you want as separate key value pairs like this: UpdateAttribute Next you will use these attributes to create a json object attribute using AttributesToJson: AttributesToJson flowfile-attribute After that, if you inspect the flowfile you will see the json object as JSONAttributes: FlowFile Attribute JSONAttributes From here you can use it as ${JSONAttributes} or if you need it to be called ${attributes} you just use another UpdateAttribute:
... View more
06-04-2020
05:15 AM
@renuu Create your attributes how you need them to be in UpdateAttribute. Next use attributesToJson to combine the attributes you want, and put them together into a single json object. In this processor set to flowfile-attribute if you need to use them as attributes later. If you need to POST the json, use flowfile-content. Once you have set your attributes in UpdateAttribute, test the flow and view the attributes, or the flowfile content. You may notice the strings are escaped. If they are escaped, and you need them to not be escaped, that is when you must use unescapeJson. You can use unescapeJson anywhere you are allowed to use expression language. You could use it in another UpdateAttribute process to make a new attribute that is unescaped. You can use it in replaceText if you want to unescape the content of the flowfile before posting.
... View more
06-04-2020
05:10 AM
@JonnyL after FetchFile each flowfile contains an attribute called ${path}. Use this path when writing the files like this: PutFile Settings Expected Output You can find my template for this test here: https://github.com/steven-dfheinz/NiFi-Templates/blob/master/ListFile_FetchFile_PutFile_Demo.xml If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
06-01-2020
06:49 AM
@renuu just add more S.... ${now():minus(600000):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "GMT")} I have updated the original snippet as well...
... View more