Thank you @cjervis and @BScott for the update. ... View more

@cjervis Very good news and thanks much for sharing the happy moment with us! Waiting for the update!! I have went through the FAQ about new repuation program... looks very interesting! Still I am not 100% clear about my current/existing reputation. I have contributed to Cloudera Community for 2.5 years with more than 500+ posts and received Champion award for the year 2017. But unfortunately not able to contribute in recent days (since Jan-2019) due to my new role/technologies and some additional works... Hope I can re-start my contribution in couple of months... Now my question is, Does my reputation still valid after you roll out a new reputation system? or all my effort will be voiding? Please clarify ... View more

@orak   Are you using Cloudera Enterprise by any chance? if so, you can generate report from CM -> Clusters (top menu) -> Reports -> Directory usage    For more details, pls refer https://www.cloudera.com/documentation/enterprise/5-13-x/topics/cm_dg_disk_usage_reports.html#cmug_topic_12_1   ... View more

@yunsu   It means, some services are free of cost up to 100 nodes and more than 100 nodes (mostly for commercial purposes) will be charged or free version may not allow to add more than 100 nodes. But if you/your company has a valid licensed product then you don't need to worry about it  ... View more

@Tomas79   Please increase the below parameter value as needed and try again, it may help you   Java Heap Size of Catalog Server in Bytes ... View more

@public   Are you using Cloudera Manager? if so, do you have same version of CM in your prod and qa then you can try the below   1. CM -> Backup menu -> Peers (add peer) - one time  2. CM -> Bakcup menu -> Replication schedule -> Create schedule (as many as you want) ... View more

  @sidh   Not sure it is considering different user id from beeline and spark... if so   The one reason could be a keytab if it is a kerberoized env, you can maintain your keytab somewhere and call it from your spark-submit as follows. It will make sure to execute your spark job with required kerberos ticket   spark-submit ..... --master yarn \ --keytab /home/auth/myauth.keytab \   also pls make sure the user/group has been assigned to the required sentry role , it has been granted to the required DB with create permission  ... View more

@DanielWhite   I had the similar issue long back and below was my findings   Please check the owner of HDFS folder/files for the corresponding db that you are trying to delete. if you are the owner and trying to delete the table/db from hive/impala, it will delete both metadata and hdfs file/folder. Whereas you are not the owner of hdfs folder/file but got an access in hive/impala to manage data and trying to delete it, it will just delete the metadata but not the underlined folder/files from hdfs   pls try this with a sample db/table for more understanding ... View more

@Broche   Please refer the below link, it may help you   https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_set.html   ... View more

@archit_neema   That is the reason why i've asked to use -R command    -chown           # you are talking about this, yes it will impact only once  -chown -R      # I am talking about this, this will triger the recursive ... View more

@archit_neema   you can use -R option on the parent directory, so that it will inherit the permission recursively to the child folder.    hdfs dfs -chown -R hive:hive /user/test   Note: you have to be super user to change ownership of any folder in hdfs ... View more

  @chriswalton007   According to the below link, If you have CDH 5.7 and above then you can upgrade an existing cluster to Cloudera 6. But this is for enterprise edison , are you trying it from enterprise edison?    https://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Cloudera-Enterprise-6-0-Released/m-p/79235 ... View more

@Eaks   There are different approaches, usually i do follow the below   ## Connect to beeline....   ## Run the following commands beeline> Create role role_name; beeline> DROP ROLE role_name; etc ... View more

@kush   Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link ... View more

@fil   You can get this report from Cloudera Navigator. Search by user id and apply filter as needed ... View more

@mdjedaini   There is nothing to do with cloudera on this request as there are so many other tools are available in the market. I am not sure how big your environment. In general, those who are using big environments with huge nodes will use some tools like Chef, Puppet, Terraform, Ansible, etc to achieve your requirement (for cloud there are another different set of tools like Cloudformation, etc)   In high level, you can divide them into two category: Push and Pull based a. Tools like Puppet and Chef are pull based. Agent/Client on the server periodically checks for the configuration information from central server(master) b. Ansible is Push based. Central server pushes the configuration information on target servers. You control when the changes are made on the servers     ... View more

@mohankumars   yes, if you are going to use it for your personal use (or) practice purpose... with less loading  ... View more

@phaothu   To do via CM,    Login as admin to CM -> HDFS -> Instances -> 'Federation and high availability' button -> Action -> Manual Failover ... View more

@Hichki   In general, this issue should go off after you kinit (if you are using command line).   This is just an additional check to make sure everything is good in your new cluster: Pls check whether the required kerbeors Principals are added as needed in the new cluster.  You can use the commands like  kadmin.local and klist to get the principal and make sure the similar principals are available in your new cluster (CM -> Administration(menu) -> Security -> Kerberos Credentials). Pls make sure all the principals are mapped to the corresponding host in the above link ... View more

@hadoopNoob   if the command is working on the other nodes then run the netstat command again on both the nodes (for the port starting 4040) to see the difference.    it is clear that it is not a spark issue as it is working form other nodes. so you have to identify the port open/availability status  ... View more

@hadoopNoob   yes, it may be due to port, pls try the below   export SPARK_MAJOR_VERSION=2 pyspark --master yarn --conf spark.ui.port=12888 pyspark --master yarn --conf spark.ui.port=4041 pyspark --master yarn --conf spark.ui.port=4042 etc ... View more

@vkrish   You have to follow any of those links depends upon your requirement.   1. https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cm_mc_autoconfig.html if you want to install, upgrade, add a new service, add a cluster, etc then you can follow your link. Mostly this is one time work, as it will be used for only initial setup. Also this is an optional for small clusters. May be suitable for large cluster.    2. https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_ig_yarn_tuning.html if you are using mapreduce, yarn, hive, spark with mr, sqoop, etc and if you want to tune those jobs then you can use the link that i've shared. Whenever you have install/configure your cluster, it will come with default setttings, the default may not suitable for all the scenarios, so this link will give you the instruction to customize your environment for better performance ... View more

@vkrish   Not sure which document that you have referred (in case if you have referred a different link then )   The below link has an excel sheet, you have to download and fill as needed to get the recommeded values https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_ig_yarn_tuning.html     ... View more

@Matt_   I can give you two easy steps , it may reduce your burden     1. To list the valid kerberos principal $ cd /var/run/cloudera-scm-agent/process/<pid>-hdfs-DATANODE $ klist -kt hdfs.keytab ## The klist command will list the valid kerbros principal in the following format "hdfs/<NODE_FQDN>@<OUR_REALM>" 2. to kinit with the aboev listed full path $ kinit -kt hdfs.keytab <copy paste the any one of the hdfs principal from the above klist>     ... View more

@Matt_   It may possible if your JAVA_HOME is not referring to the right path.   export JAVA_HOME=<the right path -or- (usually /usr/java)>   please check java path at node13 and set the right path and try again, it may help you   ... View more

@tavi99   What about this?   CM -> Administration (menu) -> Settings -> search for     a. "Authentication Backend order" -> choose 'External then database' ... View more

@tavi99   Not sure you are getting the user info from ldap/ad or database, or some other   1. CM -> Hue-> config -> Authentication Backend    a. if you are using ldap then it should be "desktop.auth.backend.LdapBackend"    b. if you are using database then it should be "desktop.auth.backend.AllowFirstUserDjangoBackend"   2. CM -> Administration (menu) -> Settings -> search for ldap    a. "Authentication Backend order" -> choose 'External then database'    b. make sure your ldap configuration is updated as needed   NOTE: Before you make any changes/overwrite your config, pls make sure to take a backup   Pls try this, it may help you ... View more

@saikrishnamante   Login to your mysql as root and create the required databases and grant the priviledge    Ex: create database hive;create user 'hive' identified by 'password'; grant all on hive.* to hive; flush privileges; create database rman;create user 'rman' identified by 'password'; grant all on rman.* to rman; flush privileges;     ... View more

@BTibetanMastiff   When I was in the similar siutation we tried the decommission in batches (3 to 4 nodes at a time). CM/Hadoop will internelly take care of the replication (for the deleted data from decommission node), so no need to worry about the replication part.   So below are my recommendations 1. Decommission the nodes by batches 2. CM -> HDFS -> Web UI -> Datanods -> Decommissioning -> Make sure there is no under replicated blocks 3. CM -> All Hosts -> Commission state(left side menu) -> Decommissioned -> Make sure your host is listed 4. If you have time, wait for 1 or 2 days or more then delete the host from CM ... View more