Member since
02-24-2016
175
Posts
56
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1031 | 06-16-2017 10:40 AM | |
8315 | 05-27-2016 04:06 PM | |
1110 | 03-17-2016 01:29 PM |
08-18-2016
02:31 AM
Smart Solutions HDP 2.5 should be available in the next weeks.
... View more
08-22-2016
01:15 AM
2 Kudos
Hi @Smart Solutions, I think this would be sufficient to certify that the libraries are installed and your applications will be able to find them. You can find several examples that are ready to run under /usr/hdp/current/spark-client/examples/src/main/python/mllib. You can substitute python with your preferred language to find examples that correspond to the appropriate API. In terms of optimized configurations, it is hard to tune that upfront as it will be highly dependent upon on your application, dataset, and cluster.
... View more
08-15-2016
05:48 PM
There could be a problem with the certificate itself. I recommend regenerating it and trying again. You can follow instructions in the Apache Knox Users Guide to generate a self-signed certificate: http://knox.apache.org/books/knox-0-6-0/user-guide.html#Generating+a+self-signed+cert+for+use+in+testing+or+development+environments If you want to use a more legitimate certificate you can generate and sign it yourself with OpenSSL or from a CA, and follow the steps in the next section of the guide, Using a CA Signed Key Pair.
... View more
08-01-2016
01:31 PM
@Steven Hirsch I think you can try it for one application, one possibility is to simply switch off ATS for a bit that helped me once but not a second time ( Tez still tries to log to it ) So if you really want to switch it off completely you can add the following settings: tez.history.logging.service.class
= org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
and to see the logs:
tez.simple.history.logging.dir=/mylogdir
Also removing the following ATSHooks hive.exec.pre.hooks=
hive.exec.post.hooks= Also potentially reduce log levels hive.tez.log.level=ERROR And see if it makes things faster. Again if you don't see a difference you may have other issues. But its worth to rule out. ATS 1.5 has been enabled in HDP2.4. Also ATS1.0 has some tuning options. If that is really your bottleneck Hortonworks support may be able to help.
... View more
12-20-2016
11:37 AM
I have the same pb but i don't know how to delete my application from spark-history. Could you explain, please ? thanks
... View more
06-09-2016
04:33 PM
HI @Jitendra Yadav , I restarted couple of times and it worked smoothly so did not do anything else. Thank you 🙂
... View more
06-07-2016
03:49 PM
@Alex Miller Well in our scenario both the A/D are more or less replicas. Anyway I got this fixed.
... View more
05-25-2016
01:15 PM
1 Kudo
Hi @Smart Solutions. Right now... there isn't really a good, quick, clean, easy way to achieve this. You've already identified the thread that I would otherwise point you towards for some ideas. You're just too good! There are two main approaches that I would recommend thinking about, the first is ceasing to make any changes via the web-ui, and only making changes via the API, that way you can just call both your clusters one after the other to make the configuration changes. The second is to use some of the ideas from the thread you linked to, where you would continue maintaining the configs on your "master" cluster, but then extract the configs from your "master" cluster on a regular basis (or on a trigger of config changed?), diff them between the previous "master" cluster config version, and then push the resulting deltas to your "slave" cluster, again via API calls. Either way, there's quite a bit of automation that would be required there. I'd strongly suggest if you want to go down this path, doing your work out in the open, this is something I see come up now and again, so I think you may well get others who would be interested in working with you on this. Longer term, Ambari will no doubt support multi cluster, and this functionality would be a natural extension of that, but progress on those public JIRA's has been slow, with other more important items taking priority. Happy to hear if you have other ideas too, sorry I couldn't be more direct help, but let me know if you plan on cutting some code moving forward, I'm sure it'd be an interesting project. Many thanks.
... View more
05-25-2016
09:39 AM
2 Kudos
@Smart Solutions You can put several Knox instances behind a load balancer to have HA. So multiple Knox instances are required for HA Knox documentation gives an example on how to do it with Apache HTTP Server https://knox.apache.org/books/knox-0-9-0/user-guide.html#High+Availability
... View more