Member since
04-22-2014
1218
Posts
342
Kudos Received
157
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 28975 | 03-03-2020 08:12 AM | |
| 19698 | 02-28-2020 10:43 AM | |
| 5443 | 11-12-2019 03:28 PM | |
| 7858 | 11-01-2019 09:01 AM | |
| 7483 | 08-12-2019 11:06 AM |
06-23-2019
08:37 PM
1 Kudo
This looks like a case of edit logs getting reordered. As @bgooley noted, it is similar to HDFS-12369, where the OP_CLOSE is appearing after OP_DELETE causing the file to be absent when replaying the edits. The simplest fix, depending on if this is the only file instance of the reordered issue in your edit logs, would be to run the NameNode manually in an edits-recovery mode and "skip" this edit when it catches the error. The rest of the edits should apply normally and let you start up your NameNode. The recovery mode of NameNode is detailed at https://blog.cloudera.com/blog/2012/05/namenode-recovery-tools-for-the-hadoop-distributed-file-system/ If you're using CM, you'll need to use the NameNode's most recent generated configuration directory under /var/run/cloudera-scm-agent/process/ on the NameNode host as the HADOOP_CONF_DIR, while logged in as 'hdfs' user, before invoking the manual NameNode startup command. Once you've followed the prompts and the NameNode appears to start up, quit out/kill it to restart from Cloudera Manager normally. If you have a Support subscription, I'd recommend filing a case for this, as the process could get more involved depending on how widespread this issue is.
... View more
06-18-2019
11:37 AM
1 Kudo
@MichalAR , Right, so SAML can be used for Authentication and then LDAP for user/group sync. If you are not using LDAP for authentication, then "Create LDAP users on login" won't impact you. If you want to prevent the creation of a Hue user for a new user login, you can set the following: [libsaml] create_users_on_login=False If you do that, though, you need to be sure that you have all of your users already in Hue before they authenticate; otherwise, they will get an error. If you would like to leave create_users_on_login "True" but change the default group membership, you can adjust the "default" group that is set for new users. To do so, set: [useradmin] default_user_group=<name_of_your_preferred_group> That way, you don't prevent users from authenticating via SAML if they don't already exist as Hue users, but you can restrict the resources they can access. It's just another thing to consider that may help you achieve the type of configuration you want.
... View more
06-03-2019
07:06 PM
Hi Ben, Appreciate your quick and kind response. I have one more question. After reading your answer, I've started to develope using "time-seriese" api. During programming, I found a problem with data before about 30 days. I'd like to obtain a every single minute's data of whole period but, It might be only available every 10 minute's( or longer) data set when I tried to get old resource data. I used this api below. Could you help me know how to get every single minute's such as, cpu/mem usage ? import time import datetime api_instance = cm_client.TimeSeriesResourceApi(api_client) from_time = datetime.datetime.fromtimestamp(time.time() - 7776000) to_time = datetime.datetime.fromtimestamp(time.time()) query = "select cpu_user_rate "\ " where entityname = 'xx' " # Retrieve time-series data from the Cloudera Manager (CM) time-series data store using a tsquery. result = api_instance.query_time_series(_from=from_time, query=query, to=to_time)#, desired_rollup='RAW', must_use_desired_rollup = 'true') ts_list = result.items[0] for ts in ts_list.time_series: print (ts.metadata.attributes['entityName'], ts.metadata.metric_name) for point in ts.data: print (point.timestamp, point.value) Appreciate your response. Ben,
... View more
05-23-2019
01:56 AM
Thanks @bgooley ! That confirms that we are not totally off-track. Best regards, rdbb
... View more
05-07-2019
01:24 AM
Hi Prash, Since i upgraded the packages in the early stage of the setup. I just created a dump (backup) for the Kerberos , and use the yum update command to upgrade it. After updated the packages, the cluster could be started.
... View more
04-24-2019
02:27 AM
@bgooley , thank you for your feedback and your clear explanation , in fact the problem was resolved by removing the contents of /var/lib/hadoop-yarn/yarn-nm-recovery/ directory and then the Nodemanager role started successfully. the solution that I've found was from : https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Yarn-NodeManager-fails-to-start-and-crashing-with-SIGBUS/m-p/66590#M3611
... View more
03-22-2019
03:20 PM
1 Kudo
@IVenkatesh, Cloudera Manager knows to which database server to connect based on what it sees in /etc/cloudera-scm-server/db.properties We would not know which server should be used. It sounds like you'll have to figure out what has changed since you tried "moving" mysql server. If you used to have more users, try accessing each 'scm' database with a mysql client and see what rows exist in the USERS table. That might give you some clues.
... View more
02-12-2019
12:00 PM
Worked like a charm. Followed the instructions and all went well Thanks a million
... View more
02-12-2019
04:15 AM
Hello @bgooley Cross-realm trust is OK. I can `kinit` principals from both MIT KDC and AD realms. Hue-LDAP authenticaion is also OK, however (for now) LDAP users can only perform action not related to HDFS, HIVE and IMPALA. My target is to have some users (humans) to be authenticated against LDAP (for Hue and all CLI hive-impala-etc actions) and some other users (oozie pipelines) as well as all services to be authenticated against MIT KDC. Now, I am reading here https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cm_sg_ldap_grp_mappings.html that: "The local user:group accounts must be mapped to LDAP for group mappings in Hadoop. You must create the users and groups for your Hadoop services in LDAP. To integrate the cluster with an LDAP service, the user:group relationships must be contained in the LDAP directory. The admin must create the user accounts and define groups for user:group relationships on each host." This is confusing, as it is supposed (https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_auth_overview.html#concept_n5q_5h2_bt__local-mit-to-active-dir-architecture) that only user principals should be configured in AD. My question is whether in this architecture I need to define services user:group relationships etc in LDAP. (for User-group mapping I am trying both LdapGroupsMapping and SSSD - none have worked yet though) Thank you, Gerasimos
... View more
02-08-2019
08:26 AM
I'm glad that was it because I couldn't figure out many other possible causes of that sort of behavior :-).
... View more
- « Previous
- Next »