Member since
01-25-2017
396
Posts
28
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
862 | 10-19-2023 04:36 PM | |
4419 | 12-08-2018 06:56 PM | |
5518 | 10-05-2018 06:28 AM | |
20064 | 04-19-2018 02:27 AM | |
20086 | 04-18-2018 09:40 AM |
10-29-2017
03:38 AM
Hi Guys, I'm getting from time to time that some NodeManagers got lost in Yarn as a result of log-dirs are bad: /var/log/hadoop-yarn/container. Looking at the disk space and don't see any issue there, at the Resource manager i see: INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_e37_1509251204123_1378_01_000001, NodeId: avpr-dhc001.lpdomain.com:8041, NodeHttpAddress: avpr-dhc001.lpdomain.com:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.16.144.140:8041 }, ] for AM appattempt_1509251204123_1378_000001 2017-10-29 05:08:22,593 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node avpr-dhc001.lpdomain.com:8041 reported UNHEALTHY with details: 1/1 log-dirs are bad: /liveperson/hadoop/log/hadoop-yarn/container 2017-10-29 05:08:22,593 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: avpr-dhc001.lpdomain.com:8041 Node Transitioned from RUNNING to UNHEALTHY I don't see any issue in the DataNode or NodeManager logs. No inode issue in the server.
... View more
Labels:
- Labels:
-
Apache YARN
10-28-2017
07:58 PM
@Geoffrey Shelton Okot I have a fully linux cluster: 5 datanodes, 1 application node for the oozie and other application like Hue, I have 2 HA nodes and one client server and 1 servers for Cloudera manager. our linux systems are using aes256 so i added this to the krb5 conf and enable it in the active directory. I'm using the Active directory as a kerberos service: I attached before the AD snapshots and here the conf that related to the HDFS. I'm able to authnticate against the active direcotry using kinit -V hdfs-conf-4.pnghdfs-conf-of-the-data-nodes-ports.pnghdfs-conf.pnghdfs-conf2.png
... View more
10-28-2017
02:02 PM
Tried but still getting the same error, Below attached my AD supported encryption ad-conf-in-ad.pngad-part-2.png
... View more
10-28-2017
01:32 PM
@Geoffrey Shelton Okot [root@aopr-dhc001 ~]# cat /etc/krb5.conf [libdefaults] default_realm = LPDOMAIN.COM
dns_lookup_kdc = true
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 5000 supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
[realms]
LPDOMAIN.COM = {
kdc = ropr-mng01.lpdomain.com admin_server = ropr-mng01.lpdomain.com
}
[domain_realm]
... View more
10-28-2017
01:12 PM
@Geoffrey Shelton Okot supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
I Have the following config also: :dfs.encrypt.data.transfer.algorithm=AES/CTR/NoPadding
dfs.encrypt.data.transfer.cipher.key.bitlength=256
Kerberos Encryption Types=rc4-hmac
seems that kinit nor working in the same you in HDP:
[root@aopr-dhc001 ~]# kinit -V -J-Dsun.security.krb5.debug=true -J-Djava.security.debug=true -k -t cloudera-scm@LPDOMAIN.COM.ktab {cloudera-scm@LPDOMAIN.COM.ktab_Principal} kinit: invalid option -- 'J'
kinit: invalid option -- '-'
kinit: invalid option -- 'D'
Bad start time value un.security.krb5.debug=true
kinit: invalid option -- 'J'
kinit: invalid option -- '-'
kinit: invalid option -- 'D'
kinit: invalid option -- 'j'
kinit: invalid option -- '.'
Bad start time value ecurity.debug=true
Usage: kinit [-V] [-l lifetime] [-s start_time]
[-r renewable_life] [-f | -F] [-p | -P] -n [-a | -A] [-C]
[-E]
[-v] [-R] [-k [-t keytab_file]] [-c cachename]
[-S service_name] [-T ticket_armor_cache]
[-X <attribute>[=<value>]] [principal]
options: -V verbose
-l lifetime
-s start time
-r renewable lifetime
-f forwardable
-F not forwardable
-p proxiable
-P not proxiable
-n anonymous
-a include addresses
-A do not include addresses
-v validate
-R renew
-C canonicalize
-E client is enterprise principal name
-k use keytab
-t filename of keytab to use
-c Kerberos 5 cache name
-S service
-T armor credential cache
-X <attribute>[=<value>]
... View more
10-28-2017
10:31 AM
Tried but with no success, indeed i'm notice such error before this error and don'w know how it might be related: KdcAccessibility: remove ropr-mng01.lpdomain.com
>>> KDCRep: init() encoding tag is 126 req type is 11
>>>KRBError:
sTime is Sat Oct 28 06:26:45 EDT 2017 1509186405000
suSec is 487082
error code is 25
error Message is Additional pre-authentication required
sname is krbtgt/LPDOMAIN.COM@LPDOMAIN.COM
eData provided.
... View more
10-28-2017
08:48 AM
Hi Geoffrey, Yes i'm using CDH but the error i'm getting is not related to CDH.
... View more