Question: How To enable https port for oozie web UI.    Hi Team,   We recieved the following security vulnerability from our server team for the ports 10002 and 11000 Hiveserver and oozie web UI ports 11000 HTTP Security Header Not Detected 10002 HTTP Security Header Not Detected We will disable HS2 port as it is not being used for anything. We are using CDH 5.14 version. Looks like this vulnerability is fixed in the higher versions of CDH and CDP Here is some more details about this vulnerability "Note: To better debug the results of this QID, it is requested that customers execute commands to simulate the following functionality: curl -lkL --verbose. CWE-693: Protection Mechanism Failure mentions the following - The product does not use or incorrectly uses a protection mechanism that provides sufficient defense against directed attacks against the product. A "missing" protection mechanism occurs when the application does not define any mechanism against a certain class of attack. An "insufficient" protection mechanism might provide some defenses - for example, against the most common attacks - but it does not protect against everything that is intended. Finally, an "ignored" mechanism occurs when a mechanism is available and in active use within the product, but the developer has not applied it in some code path. Customers are advised to set proper X-Frame-Options, X-XSS-Protection, X-Content-Type-Options and Strict-Transport-Security HTTP response headers. Depending on their server software, customers can set directives in their site configuration or Web.config files. Few examples are: X-Frame-Options: Apache: Header always append X-Frame-Options SAMEORIGIN nginx: add_header X-Frame-Options SAMEORIGIN; HAProxy: rspadd X-Frame-Options:\ SAMEORIGIN IIS: <HTTPPROTOCOL><CUSTOMHEADERS><ADD NAME="X-Frame-Options" VALUE="SAMEORIGIN"></ADD></CUSTOMHEADERS></HTTPPROTOCOL> X-XSS-Protection: Apache: Header always set X-XSS-Protection "1; mode=block" PHP: header("X-XSS-Protection: 1; mode=block"); X-Content-Type-Options: Apache: Header always set X-Content-Type-Options: nosniff HTTP Strict-Transport-Security: Apache: Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains" Nginx: add_header Strict-Transport-Security max-age=31536000; Note: Network devices that include a HTTP/HTTPS console for administrative/management purposes often do not include all/some of the security headers. This is a known issue and it is recommend to contact the vendor for a solution. " They have told to reach out to the vendor to fix the issues. We would like to get some fix for this:   Question: How To enable https ports for oozie web UI. It involves creating keystore file, that's why it is hard to do it.   By enabling this https will it fix this vulnerability ?   Please suggest if there is any fix available or the suggested methods would work. Please suggest the steps to do the same as well. Thank you ! ... View more

Hi All,   is the HDFS NFS Gateway service really important for a hadoop cluster to function properly. We received a security vulnerability report for the rpcbind service which is used by this NFS gateway service of HDFS.    The security team wants us to disable the rpcbind service on the server. when we disable the rpcbind service on the server, the nfs gateway service goes down on HDFS and shows as red in the cluster.    We want to know if we can run hdfs or do all other operations of the hadoop cluster without any issues by disabling the rpcbind service . can we remove this nfs service ?. or this nfs gateway service is really critical for the cluster.    Solution given by the Security team: "Configure the host's "mount" daemon to disallow root and other mount points with sensitive content that should not be publically accessible. Typically, the configuration file for the "rpc.mountd" daemon is "/etc/exports"."   Please advice on this. If you had faced the same situation , please let us know if there is any workaround.    Thank you  ... View more

Hi All,   I am unable to run the sqoop jobs on the oozie scheduler. it is an incremental import sqoop job so we need to to schedule this as a sqoop job but having issues on it. But normal sqoop commands are running on the oozie scheduler. The sqoop job works perfectly when run it on the server but we want to schedule the same job in Oozie. But when we run it on Oozie it throws the below error and the job fails.   "ERROR, reason: Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]"   when we run a jar file or sqoop command on oozie, it works fine. only the sqoop job fails with the error. we need to schedule several import sqoop jobs in our cluster , so we need to fix this asap.   I later added the jar file on the sqoop hdfs path hdfs dfs -put log4j-jcl-2.13.3.jar /user/oozie/share/lib/lib/sqoop After adding this file, it gives a different error. it says the job is not found..   118:16:17.086 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 18:16:17.119 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.7.7.1.4.0-203 18:16:17.170 [main] INFO org.apache.sqoop.manager.SqlManager - Using default fetchSize of 1000 18:16:17.385 [main] ERROR org.apache.sqoop.metastore.GenericJobStorage - Cannot restore job: AppRiskMASRiskCategories1 18:16:17.385 [main] ERROR org.apache.sqoop.metastore.GenericJobStorage - (No such job) 18:16:17.386 [main] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Cannot restore missing job AppRiskMASRiskCategories1 at org.apache.sqoop.metastore.GenericJobStorage.read(GenericJobStorage.java:272)   I also tried to create a sqoop job on the server where the yarn is running. but that also gives the same issue error.   I need to confirm if it is sqoop jobs are specific to the users because we have a user with which we login to the Hue and schedule the sqoop job on the hue console but the sqoop job on the server is with a different user. May be that's why it is not able to locate the sqoop job.   I want to check what are the guidelines to run/schedule a sqoop job on hue.   1. related to users 2. on which server the sqoop jobs needs to be placed 3. what are the other dependencies.   Please help on how to run a sqoop job on oozie scheduler . Thanks a lot ... View more

Hi Guys,   We are trying to incrementally import all our external rdbms tables into Hive using Sqoop jobs. There are more than 1500 tables in almost 30 Databases. We tested the incremental imports on the table based on our requirement and have written sqoop jobs accordingly. We want to do it only incrementally as and when the source Table is updated or a new record is added we need to add it to our Hive tables. The only option we see for data ingestion incrementally is sqoop Job.   So, writing so many sqoop jobs and scheduling it on daily basis and maintaining it seems to be needed a lot of efforts and will take time which is practically impossible. Since, Cloudera is targetted at Big Data there must be a solution where we can load data from a lot of different tables and ingest into the Hive. Please help us in creating a solution to import all the source tables date incrementally and schedule it daily, so that we don't miss any data.   Thanks! ... View more

Hi All, We have 1600 Rdbms tables to be imported into Hive using sqoop incremental import jobs. But, when I think of creating that many sqoop jobs and scheduling them in oozie or shell script, it seems challenging . I feel that it will be hard to maintain and find which job or script is for which table. has anyone dealt with such a scenario ? Please suggest how to schedule sqoop incremental jobs periodically which can be easily maintained and checked in case of any issues. Thanks!  ... View more

Hi Guys! I found out how to incrementally import the data using sqoop into hive. When I use --incremental append I get any new records added to the source rdbms into hive. However, when some existing record is altered and a modified date column value is updated in the rows, we need to make the import work as equally. Basically, this import should add the new rows and update the edited existing rows into hive. When I try to use sqoop incremental lastmodified it says that it doesn't work with hive. any help on achieving this scenario is much appreciated. Thank you ! ... View more

Hi All,   I am working on a solution where I am trying to check duplicates or repeated values in the Data field level like same value present in multiple columns. if same value is present we need to check if the relationship with the column is valid or not. If it doesn't have any relationship it needs to throw a flag saying check your data. Below table might be an example    ID Name Fathers Name Address 1 Alpha Bravo   2 Bravo Gamma   3 Gamma Alpha Bravo Street   In the above example if 'bravo' is repeated in any other column of the table we need to check why it is getting repeated or if the relationship is correct. Also, overall relationships between data and columns should be checked.   This activity is to have a good quality of the data we are ingesting. and we need to apply this while ingesting in hive from an external DBMS.   My question: Should I write a business logic or code to achieve these constraints or is there any Data quality tool available to check the data checks in Big data while Data Ingestion.   any leads on any tools, algorithms, code will be really helpful for me. If anyone has already seen such a thing please feel free to share. It would be of great help !   Thank you! ... View more

Hi All, We are trying to import all the tables from an external DB using sqoop with below command. It doesnt work and neither does it give any errors. sqoop-import-all-tables --connect "jdbc:sqlserver://CTSINCHNVPDTDB2;database=DemandandCapacity" --username ApplensView --password ApplensView --hive-import -create-hive-table --hs2-url "jdbc:hive2://10.127.116.119:10000/demand_and_capacity" Output: [5:39 PM] S, Venkatesh (Cognizant) Warning: /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/04/19 17:36:45 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.4.0-203 21/04/19 17:36:45 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/04/19 17:36:45 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 21/04/19 17:36:45 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 21/04/19 17:36:45 INFO manager.SqlManager: Using default fetchSize of 1000 Upon researching we could see that tables with no primary key or composite key or any varchar columns cannot be sqooped with sqoop-import-all-tables command. Is there any other way to import all the Tables of a DB in a single shot using sqoop , which has tables with no primary keys ? If there is no option in sqoop what is the alernative method to import all tables of a DB. Please help as we need to ingest Data from external DBs which has many tables within it Thank you! ... View more

Hi All,   Need help in fixing the bellow errors in our newly created CDSW cluster.   all the services of the CDSW cluster are running.   I guess I need to bring the kubernetes service up. "The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?"   health errors: CDSW StatusSuppress... Failed to run CDSW Nodes Check. * Failed to run CDSW system pods check. * Failed to run CDSW application pods check. * Failed to run CDSW services check. * Failed to run CDSW secrets check. * Failed to run CDSW persistent volumes check. * Failed to run ...   --------------------------------------------   [root@server ~]# cdsw logs -x Generating Cloudera Data Science Workbench diagnostic bundle... Collecting basic system info... Collecting kernel parameters... Collecting kernel messages... Collecting the list of kernel modules... Collecting the list of systemd units... Collecting cdsw details... Collecting application configuration... Collecting disks information... Collecting Hadoop configuration... Collecting network information... Collecting system service statuses... Collecting nfs information... Collecting Docker info... Collecting Kubernetes info... Collecting Helm info... Collecting custom patches... cp: cannot stat ‘/etc/cdsw/patches’: No such file or directory Collecting Kubelet logs... Collecting CDSW Host Controller logs... Collecting system logs... Collecting Kubernetes cluster info dump... ls: cannot access cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00/k8s-cluster-info/*/*/logs.txt: No such file or directory Exporting user ids... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? error: pod or type/name must be specified error: pod or type/name must be specified Collecting health logs... Collecting event logs... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? error: Filespec must match the canonical format: [[namespace/]pod:]file/path Exporting metrics... The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port? ERROR:: Unable to get service account credentials. Provide SERVICE_ACCOUNT_SECRET or run on master node.: 2 Producing logs tarball... Logs saved to: cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00.tar.gz Cleaning up...     -------------------------------------------------------------------------------------------   [root@server ~]# cdsw validate   [Validating host configuration] > Prechecking OS Version........[OK] > Prechecking kernel Version........[OK] > Prechecking that SELinux is disabled........[OK] > Prechecking scaling limits for processes........[OK] > Prechecking scaling limits for open files........ WARNING: Cloudera Data Science Workbench recommends that all users have a max-open-files limit set to 1048576. It is currently set to [1024] as per 'ulimit -n' Press enter to continue > Loading kernel module [ip_tables]... > Loading kernel module [iptable_nat]... > Loading kernel module [iptable_filter]... > Prechecking that iptables are not configured........[OK] > Prechecking kernel parameters........[OK] > Prechecking to ensure kernel memory accounting disabled:........[OK] > Prechecking Java distribution and version........[OK] > Checking unlimited Java encryption policy for AES........[OK] > Prechecking size of root volume........ WARNING: The recommended minimum root volume size is 100G. Press enter to continue [Validating networking setup] > Checking if kubelet iptables rules exist The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM] WARNING:: Verification of iptables rules failed: 1 > Checking if DNS server is running on localhost > Checking the number of DNS servers in resolv.conf > Checking DNS entries for CDSW main domain > Checking reverse DNS entries for CDSW main domain WARNING:: DNS doesn't resolve 10.127.116.126 to cdsw.cts.com; DNS is not configured properly: 1 > Checking DNS entries for CDSW wildcard domain > Checking that firewalld is disabled > Checking if ipv6 is enabled [Validating Kubernetes versions] > Checking kubernetes client version > Checking kubernetes server version WARNING:: Kubernetes server is not running, version couldn't be checked.: 1 [Validating NFS and Application Block Device setup] > Checking if nfs or nfs-server is active and enabled > Checking if rpcbind.socket is active and enabled > Checking if rpcbind.service is active and enabled > Checking if the project folder is exported over nfs WARNING:: The projects folder /var/lib/cdsw/current/projects must be exported over nfs: 1 > Checking if application mountpoint exists > Checking if the application directory is on a separate block device > Checking the root directory (/) free space WARNING:: The directory has less then 10% free capacity: 1 > Checking the application directory (/var/lib/cdsw) free space WARNING:: The directory has less then 10% free capacity: 1 [Validating Kubernetes cluster state] > Checking if we have exactly one master node WARNING:: There must be exactly one Kubernetes node labelled 'stateful=true': 1 > Checking if the Kubernetes nodes are ready > Checking kube-apiserver pod WARNING: Unable to reach k8s pod kube-apiserver. WARNING: [kube-apiserver] pod(s) are not ready under kube-system namespace. WARNING: Unable to bring up kube-apiserver in the kube-system cluster. Skipping other checks.. [Validating CDSW application] > Checking connectivity over ingress WARNING:: Could not curl the application over the ingress controller: 7 -------------------------------------------------------------------------- Errors detected. Please review the issues listed above. Further details can be collected by capturing logs from all nodes using "cdsw logs".   -------------------------------------------------------------------------- cdsw status Sending detailed logs to [/tmp/cdsw_status_rwPmR6.log] ... CDSW Version: [1.9.0.7802354:5a39a73] Installed into namespace 'default' OK: Application running as root check OK: NFS service check OK: System process check for CSD install OK: Sysctl params check OK: Kernel memory slabs check Failed to run CDSW Nodes Check. Failed to run CDSW system pods check. Failed to run CDSW application pods check. Failed to run CDSW services check. Failed to run CDSW secrets check. Failed to run CDSW persistent volumes check. Failed to run CDSW persistent volumes claims check. Failed to run CDSW Ingresses check. Checking web at url: http://cdsw.cts.com Web is not yet up. Cloudera Data Science Workbench is not ready yet ... View more

Hello,   Could someone help me in creating a Docker Block Device on my CDSW Master node. I am getting the below errors:   Checking if [conntrack-tools.x86_64] is installed...OK Setting up docker storage. ERROR:: Error in pvcreate for [/dev/sdb]: 5 ERROR:: Unable to setup docker storage.: 5 ERROR:: Unable to create storage for docker.: 5 Installed: conntrack-tools.x86_64 0:1.4.4-7.el7 Dependency Installed: libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 Complete! Setting up docker storage. ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/sde]: 1 ERROR:: Unable to create storage for docker.: 1 here's my server disks look like . shoud this be done only on Master CDSW node or even in the worker CDSW node ?     I have a 2TB sdb disk which can I give for a new /dev/sdb3 something ... how do I do this ? can you please share the commands for creating a   new disk of size 500GB-1TB for  Docker Image block device  without mounting it.    Also, would that docker image space be used later or I can follow this method of creating a files as a docker image block given in this solution. not sure if it would work and recommended.  https://community.cloudera.com/t5/Support-Questions/CDSW-installation-Entries-in-DOCKER-BLOCK-DEVICES-must-only/m-p/80978   Also, please let me know if setting up DNS wildcard is adding the *domain name on /etc/hosts or any other DNS file of the linux server. and is it only on the Master CDSW and need to do the same on the worker CDSW node as well.    Also, should Spark2 be installed manually or could be added as a service from the cluster but dont see the option in the cluster to add as a service. where are the parcels to downloaded for spark2 .. or should Install it like its shown in this doc https://docs.cloudera.com/cdsw/1.9.0/installation/topics/cdsw-configure-apache-spark-2-cdh6-or-7.html   Please help , I'm stuck with this.. Thanks!  ... View more

Hi All,   I am trying to install CDP , the CM server was succesfully installed along with CM agent and it got added in the cluster. But the Add Cluster wizard fails for all other 4 hosts of cluster while Installing the CM agents. I feel that CM server is unable to push the packages to the other hosts of the cluster.  It fails with the error   "yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: cloudera-manager"   I have internet open only on my CM server. do I need internet on all the cluster hosts or CM server internet is enough to push the packages to the other hosts.   I wont be able to get internet port open on all the hosts, so in that case should I create a local repo on CM host and install the cluster.   But I remember installing a CDH dev cluster only having internet only on CM server and without using a local repo on the CM server with the internet.   Please advice if we need internet on all the servers or CM server with internet will push the packages to the other hosts   Thanks! ... View more

Hello All, I am trying to install CDP cluster. after my Cloudera Manager is up, all my 6 hosts are detected, the user authenticated, when I go to the step of Installing agents on the nodes, Only the CM server gets succesful and rest of the nodes get failed with the error. "yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: cloudera-manager" I can sense that the Cloudera manager server is not able to connect to the other hosts due to something missing but I completed all the pre requisites. even the passwordless ssh between the hosts, thats why all the servers were detected and the installation proceeded with the user credentials. The cloudera agent logs say the below error on the succesful CM nodes 1446 Monitor-HostMonitor throttling_logger ERROR (359 skipped) ntpq: ntpq -np: not synchronized to any server 1446 MainThread agent INFO Stopping agent... With the research I did feel that the ntpq has some config issues and it is not synchronized with the other servers and I checked few things on it. I see the stratum is 16. which means it is not in sync.. and there is no star next to the network host name. ntpq -c pe remote refid st t when poll reach delay offset jitter ============================================================================== timesync.mts.co .XFAC. 16 u - 1024 0 0.000 0.000 0.000 timesync.mts.co .XFAC. 16 u - 1024 0 0.000 0.000 0.000 ntpq -c "rv 9464" associd=9464 status=8011 conf, sel_reject, 1 event, mobilize, srcadr=ctsinpunsxud.cts.com, srcport=123, dstadr=10.127.116.119, dstport=123, leap=11, stratum=16, precision=-24, rootdelay=0.000, rootdisp=0.000, refid=XFAC, I could see the /etc/yum.repos.d/ of the other cluster nodes haven't gotten the cloudera-manager repos yet. Questions: what could be the issue for the CM not connecting with the cluster nodes ? Is it the ntpq or some other issue ? please help in connecting other hosts from my CM server to install the parcels using the wizard. or should I manually install the agent packages. if I install manually how will future components or the CM-agent will communicate within the cluster. Thank you ! ... View more

Hello,   I am trying to install the cloud manager of the CDP cluster and getting Certificate ssl error during importing the signing GPG key for CDP installation. this is the command I am running   sudo rpm --import https://[username]:[password]@archive.cloudera.com/p/cm7/7.2.4/redhat7/yum/ RPM-GPG-KEY-cloudera   I tried adding sslverify=false to the /etc/yum.repos.d file.. still no luck is this something to be fixed from server side or shou Error : curl: (60) Peer's certificate issuer has been marked as not trusted by the user. More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. ... View more

Hello All,   I have read a lot about MetaData and how it could be useful to us Managing the Data. But everywhere they only talk about how good it is but no where I could find how to manage or collect or do the Metadata storage or format process.   We have different Databases and we are planning to do Data Governance for which Metadata is really important. we need to create a standard structure or format for the Data we have in our Databases and need to Ingest into Hadoop cluster or into the Data with the appropriate metadata formed.    Is there any videos, references, articles or any opensource tools which will aid in achieving Meta Data collection would be really helpful.   Please share the ways to do Metadata collection and manage the data using Metadata.   Thank you so much! ... View more

Hello All,   we recieved an alert from our server team to upgrade the current version of Postgresql to the latest one. When checked its not the DB server of the cluster having Postgresql as DB but other server of the cluster and I found this package "postgresql-libs-9.2.24-4.el7_8.x86_64"   Just curious why this package exisits on only 1 server apart from the main DB server from out of 5 other cluster servers.   Can I go ahead and remove this package.. will it affect the cluster in any way ?   If i cant remove, how to upgrade this version of postgresql and update the same in the cluster so that the cluster works without any issues.    thank you !  ... View more

Hello!    One of my cluster servers has this Mysql and its a old version which needs to be upgraded as per server team because of vulnerability issues.    I see the server doesnt have any sql service or process running and our cluster uses Postgres , nor I am able to see the status of the mysql on the server   root 17280 7440 0 18:23 pts/0 00:00:00 grep --color=auto -i mysql    so, when i try to remove the package it says the cloudera-manager-agent  is dependent on this mysql, but we never use mysql but only postgres as our hadoop DB .   Question: can I remove this or not ? does the CM agent need this mysql or not ?   If It is needed and I upgrade the Mysql to latest version how do I change the upgraded config to the cluster settings ?   Dependencies Resolved ============================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================= Removing: mysql-community-client x86_64 5.7.25-1.el7 @mysql57-community 107 M mysql-community-common x86_64 5.7.25-1.el7 @mysql57-community 2.6 M mysql-community-libs x86_64 5.7.25-1.el7 @mysql57-community 9.5 M mysql-community-libs-compat x86_64 5.7.25-1.el7 @mysql57-community 9.2 M Removing for dependencies: MySQL-python x86_64 1.2.5-1.el7 @base 284 k cloudera-manager-agent x86_64 5.14.0-1.cm5140.p0.25.el7 @Cloudera-manager 76 M net-snmp x86_64 1:5.7.2-49.el7_9.1 @centos7_optional_local 850 k net-snmp-agent-libs x86_64 1:5.7.2-49.el7_9.1 @centos7_optional_local 2.1 M perl-DBD-MySQL x86_64 4.023-6.el7 @centos7_online 323 k postfix x86_64 2:2.10.1-9.el7 @32F960A7-7DB1-D008-64D9-307AB478D4B1 12 M qt-mysql x86_64 1:4.8.7-9.el7_9 @B8A1CC78-DAEF-672D-98A2-2EB8FE14490E 74 k Transaction Summary ============================================================================================================================================================================================= Remove 4 Packages (+7 Dependent packages) Installed size: 220 M Is this ok [y/N]: n Exiting on user command     ... View more

Hi All,   I am looking for a tool or feature which is free to use on CDP cloudera data platform to manage the Metadata information of the Data we have stored in many of the Databases. I am looking for a way where I can manage, refer, recreated every data on our external databases or Hadoop system even with its Metadata information.   For examply, if we add data into one the Databases we should exactly know where we are placing the data based on the stored Metadata.   Even if we take some data from the DBs into HDFS we should know where we took it from by looking at the Metadata.,   Hope I am clear on my needs. please suggest some tools, framework or any CDP components to achieve this..    Thank you !  ... View more

Hey Guys! I am trying to find out the memory used for the last 6 months and I need the data for atleast 6 hours per day so that I can compare the maximum, average etc. But with the charts on Yarn I can see the data with every 6 hours only for last 2 months and if I go before that it changes to Daily on the X axis , not sure why. any thoughts ?? how to view all the data with every 6 hours granularity for all the historic data ?  is there any query to view the data for every 6 hours or its just adjustment of the data granularity option ?   Thanks       ... View more

Hey Hadoopers,   We are using CDH and we are planning to save some space for the future incoming data on HDFS. So, we were thinking if we could compress/Zip old HDFS data so that we could reduce some space.   1. Is Compression or zipping of HDFS files possible ? 2. Can we compress it and store it on local file system or HDFS file system 3. If yes, how to uncompress the data and keep it back to the same HDFS in future with the exact Metadata information or path where it existed before.  4. Will it compress even with the original file path or metadata ?   Thanks a ton !      ... View more

Hey guys !   In the YARN cluster metrics URL, I would like to know the duration for which the report is generated. http://IP:8088/ws/v1/cluster/metrics   In particular, in below report it says    <appsSubmitted> 379 </appsSubmitted> <appsCompleted> 379 </appsCompleted>   I would like to know between what duration these jobs/apps were submitted in the yarn ? and why is apps submitted and completed are same ? do Failed apps also come under completed ? as it gives an output saying it as failed ?   Also how does the number of containers get determined on Yarn ? I think its based on the size of the job or application.    <clusterMetrics> <appsSubmitted> 379 </appsSubmitted> <appsCompleted> 379 </appsCompleted> <appsPending> 0 </appsPending> <appsRunning> 0 </appsRunning> <appsFailed> 0 </appsFailed> <appsKilled> 0 </appsKilled> <reservedMB> 0 </reservedMB> <availableMB> 52224 </availableMB> <allocatedMB> 0 </allocatedMB> <reservedVirtualCores> 0 </reservedVirtualCores> <availableVirtualCores> 48 </availableVirtualCores> <allocatedVirtualCores> 0 </allocatedVirtualCores> <containersAllocated> 0 </containersAllocated> <containersReserved> 0 </containersReserved> <containersPending> 0 </containersPending> <totalMB> 52224 </totalMB> <totalVirtualCores> 48 </totalVirtualCores> <totalNodes> 3 </totalNodes> <lostNodes> 0 </lostNodes> <unhealthyNodes> 0 </unhealthyNodes> <decommissioningNodes> 0 </decommissioningNodes> <decommissionedNodes> 0 </decommissionedNodes> <rebootedNodes> 0 </rebootedNodes> <activeNodes> 3 </activeNodes> </clusterMetrics>   Thank you ! ... View more

Hi All,   We are using CDH Express editions in all our clusters and just worried about the news of Paywall subscriptions starting from Feb 2021.   With this paywall subscription thing from Jan 31, will it affect only the Trial and Enterprise edition or will it affect the express edition too ??   Please help on this as days are very few for us to make a plan and move the data.   Thank you !  ... View more