Member since
09-29-2015
4611
Posts
21
Kudos Received
33
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
745 | 08-11-2021 05:29 AM | |
587 | 07-07-2021 01:04 AM | |
349 | 07-06-2021 03:24 AM | |
982 | 06-07-2021 11:12 PM | |
316 | 06-02-2021 04:06 AM |
07-13-2020
03:41 AM
Hello @parinthorn , thank you for reaching out with your enquiry regarding to how to restrict access to different CDP environments. Have you seen our documentation about "Enabling admin and user access to environments", please? Should you only start to onboard your user, please follow the instructions under "Getting started as an admin". Please let us know if this is what you were looking for. Thank you: Ferenc
... View more
07-13-2020
02:46 AM
Hello @wert_1311 , thank you for reaching out about how to configure high-availability for CM. Can you please specify the version you are using, so I can send you the most relevant information? In the meantime, please find below the documentation covering your enquiry for CM6.3. Please let me know if it covers your enquiry. Please do not miss to read the Database High Availability Configuration section. Best regards: Ferenc
... View more
06-25-2020
06:03 AM
Hello @NumeroUnoNU , I've run the "alternatives --list" command on a cluster node and noticed that there is a "hadoop-conf" item, which points to a directory that has the hdfs-site.xml location. You can also discover it by: "/usr/sbin/alternatives --display hadoop-conf". This lead to me to google for "/var/lib/alternatives/hadoop-conf" and found this Community Article reply, which I believe answers your question. In short if you have e.g. gateway roles deployed for HDFS on a node, you will find the up-to-date hdfs-site.xml in /etc/hadoop/conf folder... We have a little bit diverged from the original topic in this thread. To make the conversation easier to read for future visitors, would you mind open a new thread for each major topics, please? Please let us know if the above information helped you by pressing the "Accept as Solution" button. Best regards: Ferenc
... View more
06-24-2020
08:55 AM
1 Kudo
Hello @NumeroUnoNU , yes, you either parse the contents of the hdfs-site.xml or you utilise the HDFS Client, so you do not need to worry about implementation details. I've just quickly googled for you an explanation of what is HDFS Client [1]. If you go for the parsing exercise, make sure you are not referencing the NN, otherwise on failover you should prepare your script to handle that situation. Kind regards: Ferenc [1] https://stackoverflow.com/questions/43221993/what-does-client-exactly-mean-for-hadoop-hdfs
... View more
06-24-2020
06:45 AM
Hello @NumeroUnoNU , Cloudera Manager is taking care of the Client Configuration files [1]. It makes sure that the latest configurations are deployed to all nodes where related services deployed or gateway roles for that service is configured. You will find the client configs present the node where e.g. a Datanode role is running under this folder: /var/run/cloudera-scm-agent/process/[largest number]...[Service name].../ The up-to-date configs are always in the folder which is starting with the largest number. Hope this helps! Kind regards: Ferenc [1] https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_mc_client_config.html
... View more
06-24-2020
01:02 AM
Hello @iceqboy , thank you for raising your enquiry about how to upgrade the OS version on a cluster. As a first step, please upgrade your OS. [1] points out that temporarily - while the OS upgrade is carried out - it is supported by Cloudera to run on mixed minor version releases. It means that it is less risky to run on different minor OS releases than on different OS-es. [2] describes that: "Upgrading the operating system to a higher version but within the same major release is called a minor release upgrade. For example, upgrading from Redhat 6.8 to 6.9. This is a relatively simple procedure that involves properly shutting down all the components, performing the operating system upgrade, and then restarting everything in reverse order." Once the cluster is on the same OS release, the next step is to upgrade your CM [3]. The CM version has to be higher or equal to the CDH version you are upgrading to. Then please follow our documentation on how to upgrade to CDH5.16. [4] Please let us know if we addressed your enquiry! Best regards: Ferenc [1] https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html [2] https://docs.cloudera.com/cdp/latest/upgrade-cdh/topics/ug_os_upgrade.html [3] https://docs.cloudera.com/cdp/latest/upgrade-cdh/topics/ug_cm_upgrade.html [4] https://docs.cloudera.com/cdp/latest/upgrade-cdh/topics/ug_cdh_upgrade.html
... View more
06-23-2020
02:30 AM
Hello @mhchethan , it is an internal jira. For future reference it is the DOCS-6740 [HDF3.3.0 SLES12SP3 download location is not shown]. Thank you for confirming you have all the information you need. You can close the thread by pressing "Accept as Solution" button under the message that you consider that answered your enquiry, please. Best regards: Ferenc
... View more
06-23-2020
12:43 AM
Hello @mhchethan , I have reached out to Product Management internally and the binaries for SLES12 SP1 can be used for SP3 too. Raised a jira internally, so our doc would reflect this in the future. Please let us know if you need any further information! Thank you: Ferenc
... View more
06-22-2020
06:25 AM
Hello @mhchethan , good point, let me check internally. Thank you: Ferenc
... View more
06-22-2020
03:57 AM
Hello @mhchethan , thank you for sharing your concerns with us. All the source code is reachable, however the binaries are moved behind a paywall. I understand that the wording of the sentence might be interpreted in a way as you described. In this report under page 451. there is a more clear explanation: "Cloudera notes its full commitment to open source and will continue to follow the practice of making contributions to the upstream source first, including to any new open source projects. Access to binaries, however, will only be available from Cloudera and will require a subscription agreement with the company to access, which is a departure from how it previously distributed its binaries. The reason for putting binaries behind a paywall is that it provides some level of protection for the vendor. The binaries contain Cloudera-specific IP that turns the many disparate open source projects into an enterprise-grade functioning system." Hope it helps! Best regards: Ferenc
... View more
06-22-2020
02:30 AM
Hello @NumeroUnoNU , thank you for confirming that the github repo covers your enquiries. Regarding to WebHDFS, I would use the hdfs-site.xml config file to get the URLs to the namenodes and datanodes after you've enabled it. The Apache Hadoop WebHDFS documentation describes further how the URIs are composed. Please let me know if it addresses your enquiry. Kind regards: Ferenc
... View more
06-22-2020
01:58 AM
Hello @Saimukunth , thank you for reaching out! Please note, the docker image is based on CDH5.13 and no longer maintained. You can still browse however the instructions on how to run the docker image. Going forward, we encourage you to trial our latest product line, CDP. Please let us know if you need any further input regarding to trialling CDP. Best regards: Ferenc
... View more
06-22-2020
12:14 AM
Hello @mhchethan , all the binaries now behind a paywall. For more details, please read this reply. In case you would like to trial the product, please consider downloading our sandbox release. Please let us know if we addressed your enquiry! Thank you: Ferenc
... View more
06-16-2020
03:47 AM
Hello @ni4ni , thank you for your feedback that you were successfully applied the workaround to install the 7.1.1. parcels! Best regards: Ferenc
... View more
06-15-2020
09:03 AM
Hello @ni4ni , we were able to reproduce internally the behaviour you reported. As a workaround when the installer hangs, open http://hostname:7180 and proceed with the installation. Raising this internally to investigate further. Kind regards: Ferenc
... View more
06-15-2020
06:16 AM
Hello @ni4ni , thank you for sharing with us your observations on the CDP 7.1.1. installation. Regarding to the 7.1.1. parcels, there is a workaround for the bug you hit. For your convenience pasting here the main steps: Click Parcel Repositories & Network Settings Remove that URL, and replace it with: https://archive.cloudera.com/cdh7/7.1.1.0/parcels/ Let me research your observations regarding to the installation script and get back to you. Kind regards: Ferenc
... View more
06-15-2020
01:35 AM
Hello @NumeroUnoNU , thank you for raising your enquiries about how to build a CSD. Have you seen the github repo [1] along with the instructions on how to build and deploy your custom solution, please? Please note the side-menu on the right with the links of the different instructions on e.g. how to build a parcel, how to create control scripts and so on. Please let us know if you found the information on our github repo that you were looking for! Thank you: Ferenc [1] https://github.com/cloudera/cm_ext/wiki/CSD-Overview
... View more
06-15-2020
01:07 AM
1 Kudo
Hello @Saagar , thank you for expressing your interest on downloading the Quickstart VM for CDH5.14. Unfortunately The Cloudera Quick start VM has been discontinued. You can try the docker image of Cloudera available publicly on https://hub.docker.com/r/cloudera/quickstart or simply run below command to download this on docker enabled system. docker pull cloudera/quickstart Please note, Cloudera don't support QuickStart VM Officially. The up-to-date product is Cloudera Data Platform , and you can download a trial version to install on-premises here . Best regards: Ferenc
... View more
06-12-2020
04:08 AM
Hello @ijarvis , - If you are looking to try out HDP, please consider downloading our HDP Sandbox [1]. It does not require to become a subscription customer or to have paywall credentials. - If you would like to deploy HDP in a production environment, please reach out to our Sales Team [2] to guide you further. Once you are a subscription customer, make sure you are registered to our Support Portal [3], please. Please note, the Community Portal registration is different from the Support Portal one. After logging in to the Support Portal, you can navigate to the Downloads page and follow an automated process for the paywall credentials. You will need your license key ready to generate the paywall credential, if you need a copy of the license key, you can open a non-technical case to request it (once you are registered and logged in to our Support Portal). The binaries are behind a paywall for which you need paywall credentials. Please see more about our Licencing Policy FAQ under [4]. - you can always download and compile the source code [5], which is not behind the paywall Please let me know if you need further input! Thank you: Ferenc [1] https://www.cloudera.com/downloads/hortonworks-sandbox.html [2] https://www.cloudera.com/contact-sales.html [3] https://sso.cloudera.com/register.html [4] https://www.cloudera.com/products/faq.html [5] https://github.com/hortonworks/hadoop-release/releases
... View more
06-12-2020
12:46 AM
Hello @ijarvis , thank you for reaching out reporting that you could not access to a restricted page to download HDP. Based on [1]: "Starting with the HDP 3.1.5 release, access to HDP repositories requires authentication. To access the binaries, you must first have the required authentication credentials ( username and password ). Authentication credentials for new customers and partners are provided in an email sent from Cloudera to registered support contacts. Existing users can file a non-technical case within the support portal (https://my.cloudera.com) to obtain credentials. Previously, HDP repositories were located on AWS S3. As of HDP 3.1.5 / Ambari 2.7.5 , repositories have been moved to https://archive.cloudera.com". Should you already a Cloudera Customer and have issues accessing the Support Portal as the first step, please register on the portal [2] and follow the instructions there. Please let us know if I've addressed your enquiries. Thank you: Ferenc [1] https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ch03s02s01.html [2] https://sso.cloudera.com/register.html
... View more
06-11-2020
02:19 AM
Hello @Fawze , thank you for raising your question regarding on our Common Vulnerabilities and Exposures (CVE) processes. Please see [1] and [2] articles on how we handle vulnerabilities and [3] for the Security Bulletins. Q1) Is the patching process part of the CDH versions, if i need to apply a patch do i have other ways without upgrading the CDH version? A1) for Cloudera Customers there is a patching process available, however the process is equivalent to deploying to a new minor version. Therefore we encourage applying the latest minor versions instead of patching when fix is available. There are situations when patching is not possible because of the amount of code change / redesign required. If you are a subscription customer, please file a Support ticket on our Support Portal listing all your requirements and we will guide you through the process. When you are in the process of redesigning your cluster, please involve your Account Team (Sales Team) as early as possible, as they might able to assist you to build a more robust system. Q2) Is there a dedicated board for cloudera/hadoop vulnerability issues that can be used as source of truth? A2) Yes, it is called Security Bulletins. Please see [3] Q3) Where i can get the patches? is it requesting subscription? A3) Yes, it is required to have a subscription to access Support. We understand that patching is an expensive exercise, therefore we do our best to identify options on how to resolve your issue, which might not require a patch. Please always apply the latest minor version release for your Cloudera software, as it contains the latest bug fixes. By this you can reduce the likelihood of downtime. Q4) For example, i want to follow this one https://nvd.nist.gov/vuln/detail/CVE-2019-14449 , is it mean to get this done i need to upgrade to 5.16.2 or there is a patch that i can download and run it? A4) Based on [3] there is no workaround identified, which means you have to upgrade to a version that contains the fix: Cloudera Manager 5.16.2, 6.0.2, 6.1.1, 6.2.0, 6.3.0 Since the issue affects CM only, you do not need to upgrade your CDH, only CM. Please let us know if we answered all of your enquiries and if so, please mark the answer as the solution, so it will be easier for other community members to find it. Best regards: Ferenc [1] http://blog.cloudera.com/blog/2016/05/clouderas-process-for-handling-security-vulnerabilities/ [2] http://www.apache.org/security/committers.html#vulnerability-handling [3] https://docs.cloudera.com/documentation/other/security-bulletins/topics/Security-Bulletin.html
... View more
06-09-2020
07:24 AM
Hello @BSST , thank you for sending the agent logs. Based on this because of some reason the agent cannot connect to the CM server.: [07/Jun/2020 09:30:46 +0000] 1182 MainThread agent ERROR Heartbeating to 192.168.1.8:7182 failed. error: [Errno 111] Connection refused You will need to troubleshoot the connectivity between the agent node and the CM Server node and if the port is listening on the CM node. What does the CM Server log shows at this period of time, please? (/var/log/cloudera-scm-server/cloudera-scm-server.log [1]). Do you see any exception there? Did you follow all the steps one-by-one in our documentation? This issue can be caused by multiple reasons e.g. even if the agent is configured for TLS, but the CM Server is not: Network problem, the CM Server is now on another node (check the IP address), certificate issues. Check if the CM Server has enough heap dedicated. Too low heap will result JVM to turn into heavy GC activity, rendering it unresponsive causing timeouts. Please let us know if you managed to overcome this issue. Kind regards: Ferenc [1] https://docs.cloudera.com/cloudera-manager/7.1.0/monitoring-and-diagnostics/topics/cm-logs.html
... View more
06-08-2020
02:18 AM
Hello @muslihuddin , Thank you for raising your concern about the CDP7.1.1 trial version allowing you only select CDH 6.3. and CDH 5.16 versions. The [1] documentation under "Upgrades from CDH to higher versions of CDH" explains that " Upgrades to Cloudera Manager 6.3.3 or higher now require a Cloudera Enterprise license file and a username and password." I believe this is the explanation why you only have these versions listed. We appreciate your feedback on the product, as it allows us to improve our documentation. Raising a request internally on adding this limitation under the Trial Installation section [2]. Kind regards: Ferenc [1] https://docs.cloudera.com/cdp/latest/upgrade/topics/cdpdc-upgrade-paths.html [2] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-install-runtime-using-wizard.html
... View more
06-08-2020
01:07 AM
Hello @Mondi , thank you for raising your question about why application logs being deleted from the nodes after the applications finished running and why is it happening, how to keep them in place. When log aggregation is enabled with the 'yarn.log-aggregation-enable = true' [1] you will observe the behaviour described: after the logs are aggregated to HDFS, the logs are immediately deleted from the local file system. Log aggregation does not start until the application is finished. If you need to keep the logs and some other temporary files on the local node for troubleshooting and you have log aggregation turned on then you can use yarn.nodemanager.delete.debug-delay-sec . This is set to 0 seconds by default causing the immediate delete. Should you disable log aggregation, non aggregated logs are kept for yarn.nodemanager.log.retain-seconds = 10800 seconds (3*3600 seconds or 3 hours). After that the NodeManager will delete the log files. Please let us know if your enquiries been addressed! Thank you: Ferenc [1] https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
... View more
06-03-2020
07:23 AM
1 Kudo
Hello @Mondi , thank you for sharing with us your concerns. I believe you were referring to the thread under [1]. dr.who is a username used for all unauthenticated users who submit job to YARN. You can be subject to malware attacks when your cluster is not secured and connected to the Internet. Please see [2] on this topic and how to remedy. Please always secure your cluster [3]. Make sure you enable authentication for web consoles [4] (securing your UIs). Once enabled, secured web UI will require SPNEGO to be configured on the browser you are accessing it e.g. the YARN Web UI. Please let us know if you need more information on this topic. Thank you: Ferenc [1] https://community.cloudera.com/t5/Support-Questions/HDP-2-6-1-Virus-CrytalMiner-dr-who/m-p/197497/highlight/true#M159548 [2] https://blog.cloudera.com/protecting-hadoop-clusters-from-malware-attacks/ [3] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/security.html [4] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_sg_web_auth.html
... View more
06-02-2020
12:47 AM
Hello @BSST , thank you for letting us know that you need clarification on how to set the ports used by CM and related products. Please see our "Ports Used by Cloudera Runtime Components" documentation [1] that hopefully covers your enquiry. Kind regards: Ferenc [1] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-ports-used-by-runtime.html
... View more
06-02-2020
12:10 AM
Hello @MikeZ , thank you for getting back with your analysis. We do not recommend to customise the CDH releases, as it might lead to unforeseen instabilities that might be very difficult to debug. Please consider to evaluate our CDP Trial release [1], in which the functionality you need is included and supported. Best regards: Ferenc [1] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-trial-download-information.html
... View more
05-28-2020
01:23 AM
Hello @BSST , thinking it through my impression is that the agents are not able to reach the server and vice versa. The screenshot shows that CM is waiting for the CM agent to heartbeat in, which doesn’t happen. Did you completed this step in [1], please? "Firewalls (such as iptables and firewalld) must be disabled or configured to allow access to ports used by Cloudera Manager, CDH, and related services." Another possibility is that SELinux is enabled. [1] points out that "Security-Enhanced Linux (SELinux) must not block Cloudera Manager or CDH operations" Please check the CM agent logs that would assist to find the root cause. Thank you: Ferenc [1] https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_network_and_security_requirements.html
... View more
05-28-2020
01:02 AM
Hello @BSST , thank you for the more detailed report. Checking internally if we can re-produce. Kind regards: Ferenc
... View more
05-27-2020
06:58 AM
Hello @MikeZ , thank you for raising your question about Livy on CDH6. As you mentioned Livy is supported only in CDP [1] (Cloudera Data Platform 7), however it is not supported yet in CDH6. Usually a product is not supported until Cloudera finds it stable and production ready. Therefore we encourage you to evaluate CDP [2]. The "KerberosName$NoMatchingRule: No rules applied to mzeoli@COMPANY.PRI" exception usually occurs when the hadoop.security.auth_to_local rules does not provide a mapping to the local linux user (the user has to exist on the node that is extracted from the kerberos principal). Also make sure that the krb5.conf is readable by the user. Please let us know if this information was helpful! Kind regards: Ferenc [1] https://docs.cloudera.com/runtime/7.1.0/running-spark-applications/topics/spark-submitting-applications-using-livy.html [2] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-trial-download-information.html
... View more