Question: How To enable https port for oozie web UI.    Hi Team,   We recieved the following security vulnerability from our server team for the ports 10002 and 11000 Hiveserver and oozie web UI ports 11000 HTTP Security Header Not Detected 10002 HTTP Security Header Not Detected We will disable HS2 port as it is not being used for anything. We are using CDH 5.14 version. Looks like this vulnerability is fixed in the higher versions of CDH and CDP Here is some more details about this vulnerability "Note: To better debug the results of this QID, it is requested that customers execute commands to simulate the following functionality: curl -lkL --verbose. CWE-693: Protection Mechanism Failure mentions the following - The product does not use or incorrectly uses a protection mechanism that provides sufficient defense against directed attacks against the product. A "missing" protection mechanism occurs when the application does not define any mechanism against a certain class of attack. An "insufficient" protection mechanism might provide some defenses - for example, against the most common attacks - but it does not protect against everything that is intended. Finally, an "ignored" mechanism occurs when a mechanism is available and in active use within the product, but the developer has not applied it in some code path. Customers are advised to set proper X-Frame-Options, X-XSS-Protection, X-Content-Type-Options and Strict-Transport-Security HTTP response headers. Depending on their server software, customers can set directives in their site configuration or Web.config files. Few examples are: X-Frame-Options: Apache: Header always append X-Frame-Options SAMEORIGIN nginx: add_header X-Frame-Options SAMEORIGIN; HAProxy: rspadd X-Frame-Options:\ SAMEORIGIN IIS: <HTTPPROTOCOL><CUSTOMHEADERS><ADD NAME="X-Frame-Options" VALUE="SAMEORIGIN"></ADD></CUSTOMHEADERS></HTTPPROTOCOL> X-XSS-Protection: Apache: Header always set X-XSS-Protection "1; mode=block" PHP: header("X-XSS-Protection: 1; mode=block"); X-Content-Type-Options: Apache: Header always set X-Content-Type-Options: nosniff HTTP Strict-Transport-Security: Apache: Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains" Nginx: add_header Strict-Transport-Security max-age=31536000; Note: Network devices that include a HTTP/HTTPS console for administrative/management purposes often do not include all/some of the security headers. This is a known issue and it is recommend to contact the vendor for a solution. " They have told to reach out to the vendor to fix the issues. We would like to get some fix for this:   Question: How To enable https ports for oozie web UI. It involves creating keystore file, that's why it is hard to do it.   By enabling this https will it fix this vulnerability ?   Please suggest if there is any fix available or the suggested methods would work. Please suggest the steps to do the same as well. Thank you ! ... View more

Hi All,   is the HDFS NFS Gateway service really important for a hadoop cluster to function properly. We received a security vulnerability report for the rpcbind service which is used by this NFS gateway service of HDFS.    The security team wants us to disable the rpcbind service on the server. when we disable the rpcbind service on the server, the nfs gateway service goes down on HDFS and shows as red in the cluster.    We want to know if we can run hdfs or do all other operations of the hadoop cluster without any issues by disabling the rpcbind service . can we remove this nfs service ?. or this nfs gateway service is really critical for the cluster.    Solution given by the Security team: "Configure the host's "mount" daemon to disallow root and other mount points with sensitive content that should not be publically accessible. Typically, the configuration file for the "rpc.mountd" daemon is "/etc/exports"."   Please advice on this. If you had faced the same situation , please let us know if there is any workaround.    Thank you  ... View more

Hi All,   Need help in fixing the bellow errors in our newly created CDSW cluster.   all the services of the CDSW cluster are running.   I guess I need to bring the kubernetes service up. "The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?"   health errors: CDSW StatusSuppress... Failed to run CDSW Nodes Check. * Failed to run CDSW system pods check. * Failed to run CDSW application pods check. * Failed to run CDSW services check. * Failed to run CDSW secrets check. * Failed to run CDSW persistent volumes check. * Failed to run ...   --------------------------------------------   [root@server ~]# cdsw logs -x Generating Cloudera Data Science Workbench diagnostic bundle... Collecting basic system info... Collecting kernel parameters... Collecting kernel messages... Collecting the list of kernel modules... Collecting the list of systemd units... Collecting cdsw details... Collecting application configuration... Collecting disks information... Collecting Hadoop configuration... Collecting network information... Collecting system service statuses... Collecting nfs information... Collecting Docker info... Collecting Kubernetes info... Collecting Helm info... Collecting custom patches... cp: cannot stat ‘/etc/cdsw/patches’: No such file or directory Collecting Kubelet logs... Collecting CDSW Host Controller logs... Collecting system logs... Collecting Kubernetes cluster info dump... ls: cannot access cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00/k8s-cluster-info/*/*/logs.txt: No such file or directory Exporting user ids... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? error: pod or type/name must be specified error: pod or type/name must be specified Collecting health logs... Collecting event logs... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? error: Filespec must match the canonical format: [[namespace/]pod:]file/path Exporting metrics... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? ERROR:: Unable to get service account credentials. Provide SERVICE_ACCOUNT_SECRET or run on master node.: 2 Producing logs tarball... Logs saved to: cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00.tar.gz Cleaning up...     -------------------------------------------------------------------------------------------   [root@server ~]# cdsw validate   [Validating host configuration] > Prechecking OS Version........[OK] > Prechecking kernel Version........[OK] > Prechecking that SELinux is disabled........[OK] > Prechecking scaling limits for processes........[OK] > Prechecking scaling limits for open files........ WARNING: Cloudera Data Science Workbench recommends that all users have a max-open-files limit set to 1048576. It is currently set to [1024] as per 'ulimit -n' Press enter to continue > Loading kernel module [ip_tables]... > Loading kernel module [iptable_nat]... > Loading kernel module [iptable_filter]... > Prechecking that iptables are not configured........[OK] > Prechecking kernel parameters........[OK] > Prechecking to ensure kernel memory accounting disabled:........[OK] > Prechecking Java distribution and version........[OK] > Checking unlimited Java encryption policy for AES........[OK] > Prechecking size of root volume........ WARNING: The recommended minimum root volume size is 100G. Press enter to continue [Validating networking setup] > Checking if kubelet iptables rules exist The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM] WARNING:: Verification of iptables rules failed: 1 > Checking if DNS server is running on localhost > Checking the number of DNS servers in resolv.conf > Checking DNS entries for CDSW main domain > Checking reverse DNS entries for CDSW main domain WARNING:: DNS doesn't resolve 10.127.116.126 to cdsw.cts.com; DNS is not configured properly: 1 > Checking DNS entries for CDSW wildcard domain > Checking that firewalld is disabled > Checking if ipv6 is enabled [Validating Kubernetes versions] > Checking kubernetes client version > Checking kubernetes server version WARNING:: Kubernetes server is not running, version couldn't be checked.: 1 [Validating NFS and Application Block Device setup] > Checking if nfs or nfs-server is active and enabled > Checking if rpcbind.socket is active and enabled > Checking if rpcbind.service is active and enabled > Checking if the project folder is exported over nfs WARNING:: The projects folder /var/lib/cdsw/current/projects must be exported over nfs: 1 > Checking if application mountpoint exists > Checking if the application directory is on a separate block device > Checking the root directory (/) free space WARNING:: The directory has less then 10% free capacity: 1 > Checking the application directory (/var/lib/cdsw) free space WARNING:: The directory has less then 10% free capacity: 1 [Validating Kubernetes cluster state] > Checking if we have exactly one master node WARNING:: There must be exactly one Kubernetes node labelled 'stateful=true': 1 > Checking if the Kubernetes nodes are ready > Checking kube-apiserver pod WARNING: Unable to reach k8s pod kube-apiserver. WARNING: [kube-apiserver] pod(s) are not ready under kube-system namespace. WARNING: Unable to bring up kube-apiserver in the kube-system cluster. Skipping other checks.. [Validating CDSW application] > Checking connectivity over ingress WARNING:: Could not curl the application over the ingress controller: 7 -------------------------------------------------------------------------- Errors detected. Please review the issues listed above. Further details can be collected by capturing logs from all nodes using "cdsw logs".   -------------------------------------------------------------------------- cdsw status Sending detailed logs to [/tmp/cdsw_status_rwPmR6.log] ... CDSW Version: [1.9.0.7802354:5a39a73] Installed into namespace 'default' OK: Application running as root check OK: NFS service check OK: System process check for CSD install OK: Sysctl params check OK: Kernel memory slabs check Failed to run CDSW Nodes Check. Failed to run CDSW system pods check. Failed to run CDSW application pods check. Failed to run CDSW services check. Failed to run CDSW secrets check. Failed to run CDSW persistent volumes check. Failed to run CDSW persistent volumes claims check. Failed to run CDSW Ingresses check. Checking web at url: http://cdsw.cts.com Web is not yet up. Cloudera Data Science Workbench is not ready yet ... View more

Hello,   Could someone help me in creating a Docker Block Device on my CDSW Master node. I am getting the below errors:   Checking if [conntrack-tools.x86_64] is installed...OK Setting up docker storage. ERROR:: Error in pvcreate for [/dev/sdb]: 5 ERROR:: Unable to setup docker storage.: 5 ERROR:: Unable to create storage for docker.: 5 Installed: conntrack-tools.x86_64 0:1.4.4-7.el7 Dependency Installed: libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 Complete! Setting up docker storage. ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/sde]: 1 ERROR:: Unable to create storage for docker.: 1 here's my server disks look like . shoud this be done only on Master CDSW node or even in the worker CDSW node ?     I have a 2TB sdb disk which can I give for a new /dev/sdb3 something ... how do I do this ? can you please share the commands for creating a   new disk of size 500GB-1TB for  Docker Image block device  without mounting it.    Also, would that docker image space be used later or I can follow this method of creating a files as a docker image block given in this solution. not sure if it would work and recommended.  https://community.cloudera.com/t5/Support-Questions/CDSW-installation-Entries-in-DOCKER-BLOCK-DEVICES-must-only/m-p/80978   Also, please let me know if setting up DNS wildcard is adding the *domain name on /etc/hosts or any other DNS file of the linux server. and is it only on the Master CDSW and need to do the same on the worker CDSW node as well.    Also, should Spark2 be installed manually or could be added as a service from the cluster but dont see the option in the cluster to add as a service. where are the parcels to downloaded for spark2 .. or should Install it like its shown in this doc https://docs.cloudera.com/cdsw/1.9.0/installation/topics/cdsw-configure-apache-spark-2-cdh6-or-7.html   Please help , I'm stuck with this.. Thanks!  ... View more

Hi All,   I am trying to install CDP , the CM server was succesfully installed along with CM agent and it got added in the cluster. But the Add Cluster wizard fails for all other 4 hosts of cluster while Installing the CM agents. I feel that CM server is unable to push the packages to the other hosts of the cluster.  It fails with the error   "yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: cloudera-manager"   I have internet open only on my CM server. do I need internet on all the cluster hosts or CM server internet is enough to push the packages to the other hosts.   I wont be able to get internet port open on all the hosts, so in that case should I create a local repo on CM host and install the cluster.   But I remember installing a CDH dev cluster only having internet only on CM server and without using a local repo on the CM server with the internet.   Please advice if we need internet on all the servers or CM server with internet will push the packages to the other hosts   Thanks! ... View more

Hello All, I am trying to install CDP cluster. after my Cloudera Manager is up, all my 6 hosts are detected, the user authenticated, when I go to the step of Installing agents on the nodes, Only the CM server gets succesful and rest of the nodes get failed with the error. "yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: cloudera-manager" I can sense that the Cloudera manager server is not able to connect to the other hosts due to something missing but I completed all the pre requisites. even the passwordless ssh between the hosts, thats why all the servers were detected and the installation proceeded with the user credentials. The cloudera agent logs say the below error on the succesful CM nodes 1446 Monitor-HostMonitor throttling_logger ERROR (359 skipped) ntpq: ntpq -np: not synchronized to any server 1446 MainThread agent INFO Stopping agent... With the research I did feel that the ntpq has some config issues and it is not synchronized with the other servers and I checked few things on it. I see the stratum is 16. which means it is not in sync.. and there is no star next to the network host name. ntpq -c pe remote refid st t when poll reach delay offset jitter ============================================================================== timesync.mts.co .XFAC. 16 u - 1024 0 0.000 0.000 0.000 timesync.mts.co .XFAC. 16 u - 1024 0 0.000 0.000 0.000 ntpq -c "rv 9464" associd=9464 status=8011 conf, sel_reject, 1 event, mobilize, srcadr=ctsinpunsxud.cts.com, srcport=123, dstadr=10.127.116.119, dstport=123, leap=11, stratum=16, precision=-24, rootdelay=0.000, rootdisp=0.000, refid=XFAC, I could see the /etc/yum.repos.d/ of the other cluster nodes haven't gotten the cloudera-manager repos yet. Questions: what could be the issue for the CM not connecting with the cluster nodes ? Is it the ntpq or some other issue ? please help in connecting other hosts from my CM server to install the parcels using the wizard. or should I manually install the agent packages. if I install manually how will future components or the CM-agent will communicate within the cluster. Thank you ! ... View more

Hello,   I am trying to install the cloud manager of the CDP cluster and getting Certificate ssl error during importing the signing GPG key for CDP installation. this is the command I am running   sudo rpm --import https://[username]:[password]@archive.cloudera.com/p/cm7/7.2.4/redhat7/yum/ RPM-GPG-KEY-cloudera   I tried adding sslverify=false to the /etc/yum.repos.d file.. still no luck is this something to be fixed from server side or shou Error : curl: (60) Peer's certificate issuer has been marked as not trusted by the user. More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. ... View more

Hello All,   we recieved an alert from our server team to upgrade the current version of Postgresql to the latest one. When checked its not the DB server of the cluster having Postgresql as DB but other server of the cluster and I found this package "postgresql-libs-9.2.24-4.el7_8.x86_64"   Just curious why this package exisits on only 1 server apart from the main DB server from out of 5 other cluster servers.   Can I go ahead and remove this package.. will it affect the cluster in any way ?   If i cant remove, how to upgrade this version of postgresql and update the same in the cluster so that the cluster works without any issues.    thank you !  ... View more

Hello!    One of my cluster servers has this Mysql and its a old version which needs to be upgraded as per server team because of vulnerability issues.    I see the server doesnt have any sql service or process running and our cluster uses Postgres , nor I am able to see the status of the mysql on the server   root 17280 7440 0 18:23 pts/0 00:00:00 grep --color=auto -i mysql    so, when i try to remove the package it says the cloudera-manager-agent  is dependent on this mysql, but we never use mysql but only postgres as our hadoop DB .   Question: can I remove this or not ? does the CM agent need this mysql or not ?   If It is needed and I upgrade the Mysql to latest version how do I change the upgraded config to the cluster settings ?   Dependencies Resolved ============================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================= Removing: mysql-community-client x86_64 5.7.25-1.el7 @mysql57-community 107 M mysql-community-common x86_64 5.7.25-1.el7 @mysql57-community 2.6 M mysql-community-libs x86_64 5.7.25-1.el7 @mysql57-community 9.5 M mysql-community-libs-compat x86_64 5.7.25-1.el7 @mysql57-community 9.2 M Removing for dependencies: MySQL-python x86_64 1.2.5-1.el7 @base 284 k cloudera-manager-agent x86_64 5.14.0-1.cm5140.p0.25.el7 @Cloudera-manager 76 M net-snmp x86_64 1:5.7.2-49.el7_9.1 @centos7_optional_local 850 k net-snmp-agent-libs x86_64 1:5.7.2-49.el7_9.1 @centos7_optional_local 2.1 M perl-DBD-MySQL x86_64 4.023-6.el7 @centos7_online 323 k postfix x86_64 2:2.10.1-9.el7 @32F960A7-7DB1-D008-64D9-307AB478D4B1 12 M qt-mysql x86_64 1:4.8.7-9.el7_9 @B8A1CC78-DAEF-672D-98A2-2EB8FE14490E 74 k Transaction Summary ============================================================================================================================================================================================= Remove 4 Packages (+7 Dependent packages) Installed size: 220 M Is this ok [y/N]: n Exiting on user command     ... View more

Hey Hadoopers,   We are using CDH and we are planning to save some space for the future incoming data on HDFS. So, we were thinking if we could compress/Zip old HDFS data so that we could reduce some space.   1. Is Compression or zipping of HDFS files possible ? 2. Can we compress it and store it on local file system or HDFS file system 3. If yes, how to uncompress the data and keep it back to the same HDFS in future with the exact Metadata information or path where it existed before.  4. Will it compress even with the original file path or metadata ?   Thanks a ton !      ... View more

Hi All,   We are using CDH Express editions in all our clusters and just worried about the news of Paywall subscriptions starting from Feb 2021.   With this paywall subscription thing from Jan 31, will it affect only the Trial and Enterprise edition or will it affect the express edition too ??   Please help on this as days are very few for us to make a plan and move the data.   Thank you !  ... View more