Member since
07-08-2013
548
Posts
59
Kudos Received
53
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2560 | 08-17-2019 04:05 PM | |
2541 | 07-26-2019 12:18 AM | |
8789 | 07-17-2019 09:20 AM | |
4978 | 06-18-2018 03:38 AM | |
12411 | 04-06-2018 07:13 AM |
05-20-2020
10:07 AM
rpm -qlp http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15/RPMS/x86_64/cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm | grep scm_prepare_database.sh I've noticed that the url you've given is /5.15/ that redirects to the latest maintenance release of the CM 5.15 which is 5.15.2. And you've given file "cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm" 5.15.0 which is not present in that folder. For your rpm -qlp to work try the /5.15.0/ rpm -qlp http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15.0/RPMS/x86_64/cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm | grep scm_prepare_database.sh
... View more
08-17-2019
04:05 PM
1 Kudo
This was reported as a bug, and has already been fixed in CM 6.3.0, 6.2.1 as part of OPSAPS-49111
... View more
07-26-2019
02:18 PM
> Is there any option to find empty directory using HDFS command Directly? You can get a list/find empty directories using the 'org.apache.solr.hadoop.HdfsFindTool'. And using the hdfs tool to check/test if _a_ directory is empty, you can use -du or -test; please see the FileSystemShell [0] test
Usage: hadoop fs -test -[defsz] URI
Options:
-d: f the path is a directory, return 0.
-e: if the path exists, return 0.
-f: if the path is a file, return 0.
-s: if the path is not empty, return 0.
-r: if the path exists and read permission is granted, return 0.
-w: if the path exists and write permission is granted, return 0.
-z: if the file is zero length, return 0.
Example:
hadoop fs -test -e filename du
Usage: hadoop fs -du [-s] [-h] [-x] URI [URI ...]
Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
Options:
The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files. Without the -s option, calculation is done by going 1-level deep from the given path.
The -h option will format file sizes in a “human-readable” fashion (e.g 64.0m instead of 67108864)
The -x option will exclude snapshots from the result calculation. Without the -x option (default), the result is always calculated from all INodes, including all snapshots under the given path.
The du returns three columns with the following format:
size disk_space_consumed_with_all_replicas full_path_name
Example:
hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1
Exit Code: Returns 0 on success and -1 on error. [0] https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html
... View more
07-26-2019
12:18 AM
@ADingmando you have a use case, or you have any particular issue that you'd like assistance with? The ODBC Drivers are built for CDH and HDP release. I believe there should be a release notes if you unpack the contents and compare the fixes each ODBC driver contains.
... View more
07-24-2019
04:37 AM
> Note:-only want want delete zero size dir ,not want to delete data conatains dir. One of the idea involves two step process, to generate an empty directory listing and then take the listing and -rmdir # generate the empty directory listing $ hadoop jar /opt/cloudera/parcels/CDH-*/jars/search-mr-*-job.jar org.apache.solr.hadoop.HdfsFindTool -find / -type d -empty
# produces output ... hdfs://ns1/user/impala hdfs://ns1/user/spark/applicationHistory hdfs://ns1/user/spark/spark2ApplicationHistory hdfs://ns1/user/sqoop2 ... # OPTIONAL: pick a dir and confirm that the dir is empty eg: $ hdfs dfs -du -s /user/impala 0 0 /user/impala # remove the empty dir eg: /user/impala # hdfs dfs -rmdir /user/impala
https://www.cloudera.com/documentation/enterprise/5-16-x/topics/search_hdfsfindtool.html
... View more
07-17-2019
09:20 AM
1 Kudo
> few days ago my CM started successfully. But, after restarted my server, it raised that error when I started. Unfortunately without the logs, or stack trace I won't be able to assess what caused the error. Back to the intermixed "scm" and "metastore" database issue; > "Now I tried to drop all the databases in mysql and create again, then run the following command respectively :" I think there may have been some misunderstanding, about the intention of the script "/opt/cloudera/cm/schema/scm_prepare_database.sh". The script as the documenatation state, is to create and configure a database for iteself [Cloudera Manager Server] [0] As seen in your comment, you've used for various services (amon, rman, ... metastore, ... etc) Please do check in your RDBMS (MySQL), and confirm which database name CM should be connecting to. eg: mysql> show databases; and for each database confirm which one has the correct schema with the "CM_VERSION" table in. mysql> show tables; or something like this; SELECT table_name, table_schema AS dbname
FROM INFORMATION_SCHEMA.TABLES
WHERE table_name='CM_VERSION' Note this is just a suggestion, to find the correct database where CM server can connect to. [0] https://www.cloudera.com/documentation/enterprise/latest/topics/prepare_cm_database.html#cmig_topic_5_2
... View more
07-16-2019
08:33 AM
Here's what I understand from the error; 1. I 'm using CDH-6.2.0-1.cdh6.2.0. and I use mysql as metastore. 2. Then it tells me as below:
2019-07-16 15:06:38,947 ERROR main:com.cloudera.server.cmf.Main: Server failed.
...
at com.cloudera.server.cmf.Main.main(Main.java:233)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version.
...
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'metastore.CM_VERSION' doesn't exist
Cloudera Manager uses database [0], and the database name information is under /etc/cloudera-scm-server/db.properties, normally the database name should be different and not "metastore" Hive "metastore" also uses database [1], and the database configuration settings is stored in Cloudera Manger web UI, these configuration is saved in Cloudera Manager Database. Now some discrepancies 'metastore.CM_VERSION' (note metastore) and the "Tables_in_metastore" (Mysql database is as below:) which you have provided, is listing the "Hive Metastore Database" tables, intermixed with "Cloudera Manager Database" tables. This is not ideal - as it is difficult to narrow down the issue when they happen. The "Tables_in_metastore" doesn't list the table name 'metastore.CM_VERSION' as you can see. Before we continue to look into why "CM_VERSION" table is not listed in the "metastore" database, few things to consider, was your CM server, always configured to connect to the "metastore" database in the past? Or was it connected to a different database (do # ls -ltr /etc/cloudera-scm-server/db.*) - check for db.properties backup open the files and check for the database name. And also are you aware of any recent changes in your cluster? Regards, Michalis [0] https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5 [1] https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_metastore_configure.html#topic_18_4
... View more
06-18-2018
03:38 AM
1 Kudo
The location of "scm_prepare_database.sh" will change in CM 6.x, I will check with our documentation team as this may be a doc issue. For CM 5.x please use the default location, such as "/usr/share/cmf/schema/scm_prepare_database.sh" You can check this by listing the rpm contents. # rpm -qlp http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.15/RPMS/x86_64/cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el7.x86_64.rpm | grep scm_prepare_database.sh
/usr/share/cmf/schema/scm_prepare_database.sh Let me know if this helps
... View more
04-10-2018
04:14 AM
@Youssef Process: 29977 ExecStart=/etc/init.d/cloudera-scm-server-db start (code=exited, status=1/FAILURE)
avril 06 15:05:06 SRVSTGHOST runuser[29987]: pam_unix(runuser:session): session opened for user cloudera-scm by (uid=0)
avril 06 15:05:06 SRVSTGHOST cloudera-scm-server-db[29977]: initdb : nom de locale invalide (« en_US.UTF8 ») The error _may be_ related to the embedded PostgreSQL our script is using, as this is _not_ a common message to see. The script (initialize_embedded_db.sh) attempts to initialise embedded PostgreSQL using the 'initdb' command [3] and the command is failing with the following message "initdb : nom de locale invalide (« en_US.UTF8 »)". - How is your Debian locale configured as [0]? - Have you made any locale changes recently [1]? - Also review the topic "22.1.3. Problems" in [2] using the steps I've provided below. e.g.: testing this command succeeded in my node - see [3] for what each parameter represent sudo -u cloudera-scm echo -n "password" > /tmp/generated_password.txt sudo -u cloudera-scm initdb --pgdata "/some_tmp/folder/to/test/data" --encoding=UTF8 --locale=en_US.UTF8 --auth=md5 --username="cloudera-scm" --pwfile="/tmp/generated_password.txt" [0] https://wiki.debian.org/Locale ; see "dpkg-reconfigure locales" [1] https://serverfault.com/a/491502 [2] https://www.postgresql.org/docs/9.5/static/locale.html [3] https://www.postgresql.org/docs/9.5/static/app-initdb.html
... View more
04-06-2018
07:13 AM
1 Kudo
Hi Sandy, + ktutil
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p cloudera-scm/REDACTED@PRICLUSTER.COM -k 1 -e aes256-cts:normal'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p cloudera-scm/REDACTED@PRICLUSTER.COM -k 1 -e aes128-cts:normal'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p cloudera-scm/REDACTED@PRICLUSTER.COM -k 1 -e des3-hmac-sha1:normal'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p cloudera-scm/REDACTED@PRICLUSTER.COM -k 1 -e des-hmac-sha1:normal'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p cloudera-scm/REDACTED@PRICLUSTER.COM -k 1 -e des-cbc-crc:normal'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ echo 'wkt /var/run/cloudera-scm-server/cmf8091152271730902012.keytab'
addent: Bad encryption type while adding new entry
ktutil: Unknown request "REDACTED". Type "?" for a request list.
addent: Bad encryption type while adding new entry
ktutil: Unknown request "REDACTED". Type "?" for a request list.
addent: Bad encryption type while adding new entry
ktutil: Unknown request "REDACTED". Type "?" for a request list.
addent: Bad encryption type while adding new entry
ktutil: Unknown request "REDACTED". Type "?" for a request list.
addent: Bad encryption type while adding new entry
ktutil: Unknown request "REDACTED". Type "?" for a request list.
+ chmod 600 /var/run/cloudera-scm-server/cmf8091152271730902012.keytab
chmod: cannot access `/var/run/cloudera-scm-server/cmf8091152271730902012.keytab': No such file or directory Base on the above information, I've noticed that you have set the encryption in CM UI> Administration> Setting> Kerberos> "Kerberos Encryption Types" as - aes256-cts:normal - aes128-cts:normal - des3-hmac-sha1:normal - des-hmac-sha1:normal - des-cbc-crc:normal The error I see is that while ktutil executed the command addent it failed with "Bad encryption type while adding new entry" Therefore, ktutil failed to set -e encryption_type for all 5 encryption types you've specified, so there was nothing to be written into a keytab (wkt keytab) see: 'wkt /var/run/cloudera-scm-server/cmf8091152271730902012.keytab' The encryption type combination you've specified is valid for kadmin/kadmin.local tool where the -e parameter can be specified as encryption:salt, but it is not valid for ktutil -e encryption_type Since CM script is using ktutil you may need to remove the salt suffixed ':normal'. The salt :normal is default for Kerberos Version 5, you only need to set the encryption type [0] in CM UI> Administration> Setting> Kerberos> "Kerberos Encryption Types" Encryption Type - aes256-cts - aes128-cts - des3-hmac-sha1 - des-hmac-sha1 - des-cbc-crc Let me know if this helps, Michalis [0] https://web.mit.edu/kerberos/krb5-latest/doc/admin/conf_files/kdc_conf.html#encryption-types Note: A feature request OPSAPS-29768 is in progress to not allow manual entry in "Kerberos Encryption Types"
... View more