Member since
07-19-2018
613
Posts
101
Kudos Received
117
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5098 | 01-11-2021 05:54 AM | |
| 3422 | 01-11-2021 05:52 AM | |
| 8790 | 01-08-2021 05:23 AM | |
| 8386 | 01-04-2021 04:08 AM | |
| 36692 | 12-18-2020 05:42 AM |
06-11-2020
06:17 AM
1 Kudo
Thank you @stevenmatison appreciate the pointers provided. We are not using conventional nginx as load-balancer but instead have BigIP F5. And after examining configuration there we have identified connection limit parameter which was the culprit. We have updated it and this has made some difference in the user experience. We are looking to increase the number of Hue and HS2 instances in the cluster to boost the overall and query performance. Many thanks.
... View more
06-10-2020
08:52 AM
My YARN URL right now @stevenmatison is not accessible so I can't see it using that, is there any other way to check? thanks!
... View more
06-10-2020
06:29 AM
@stevenmatison Thanks for prompt reply. I'm using HDP-3.0.1.0 with Ambari. Here's my current Hive config: Tez Container Size: 3072 MB HiveServer2 Heap Size: 4096 MB Memory: 819.2 MB Data per Reducer: 2042.9 MB They are mostly the default values. Do they make sense? Any suggestion on which to increase / decrease for optimum performance?
... View more
06-10-2020
04:11 AM
@vigneshvenu You will need to send parameters to hive during execution of query. You can find some good conversation about this here: https://community.cloudera.com/t5/Support-Questions/How-are-number-of-mappers-determined-for-a-query-with-hive/m-p/94915 If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven @ DFHZ
... View more
06-05-2020
05:20 AM
@JonnyL The sample above provided exact output you requested. Please inspect the template and inspect FlowFiles for root files and subdirectories files. The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed. For any files in root folder, then path is "./". Additionally the absolute path attribute is also there for all the files. This attribute can be acted against too with deeper expression language, but I found path was easier.
... View more
06-04-2020
05:29 AM
@renuu You have to build the final values you want as separate key value pairs like this: UpdateAttribute Next you will use these attributes to create a json object attribute using AttributesToJson: AttributesToJson flowfile-attribute After that, if you inspect the flowfile you will see the json object as JSONAttributes: FlowFile Attribute JSONAttributes From here you can use it as ${JSONAttributes} or if you need it to be called ${attributes} you just use another UpdateAttribute:
... View more
06-04-2020
05:15 AM
@renuu Create your attributes how you need them to be in UpdateAttribute. Next use attributesToJson to combine the attributes you want, and put them together into a single json object. In this processor set to flowfile-attribute if you need to use them as attributes later. If you need to POST the json, use flowfile-content. Once you have set your attributes in UpdateAttribute, test the flow and view the attributes, or the flowfile content. You may notice the strings are escaped. If they are escaped, and you need them to not be escaped, that is when you must use unescapeJson. You can use unescapeJson anywhere you are allowed to use expression language. You could use it in another UpdateAttribute process to make a new attribute that is unescaped. You can use it in replaceText if you want to unescape the content of the flowfile before posting.
... View more
06-02-2020
12:04 PM
@stevenmatison : Thanks for the update!! This issue got fixed by adding an new line in pg_hba.conf file host all hive 192.168.0.109/32 trust Restart postgresql
... View more
06-01-2020
06:59 AM
big thank you to you @stevenmatison
... View more