Member since
04-30-2020
15
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
721 | 09-22-2020 05:44 AM | |
1425 | 09-17-2020 05:33 AM |
12-07-2020
05:53 AM
Hi. I have cluster with 12 nodes(9 of them with datanodes). On each host about 30tb capacity, used on datanodes ~15TB (20Vcores on each and 280gb ram). After restart one of VM with DN, datanode was started about 1.5 hours. Is it normal? Or how can i decrease this time.
... View more
Labels:
- Labels:
-
HDFS
12-07-2020
05:46 AM
Hi. I have a problem with outOfMemory error on impala catalog server. Data in tables about 30TB and about 175k partitiions. I set Java Heap Size of Catalog Server in Bytes =30gb, but still have out_of_memory error in start. How can i calculate how much memory needs for catalog server? And is it have any limit to set ?
... View more
- Tags:
- impala
- outofmemory
Labels:
- Labels:
-
Apache Impala
11-17-2020
07:11 AM
Hello. I configurate TLS certs for HUE(from CA), on hdfs i used AUTO_TLS certs. My config Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini [desktop] app_blacklist= ssl_validate=false use_new_editor=true [notebook] [[interpreters]] [[ldap]] sync_groups_on_login=true show_notebooks=true [spark] livy_server_url=http://myhost:8998 interface=livy [pyspark] name=PySpark interface=livy [r] name=R interface=livy [hadoop] [[hdfs_clusters]] [[[default]]] ssl_cert_ca_verify = false [[ssl]] validate=false [impala] [[ssl]] validate=false In hue i see this:
... View more
Labels:
- Labels:
-
Cloudera Hue
-
HDFS
10-23-2020
02:35 AM
Yes, i thought about cron. But maybe smth in yarn conf can help, but i miss it. Maximum log files bk doesnt help
... View more
10-16-2020
07:51 AM
Hello. I have cluster 9 nodes (400gb memory on each, 2 masters with NN). Capacity will be increase every time(i add ~10TB every week). For 31millon files used ~9gb of 15. What maximum java heap size for NN i can set and what recommended blocksize for big cluster(now its 128mb, should i set 256/512 ?)
... View more
- Tags:
- datanode
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera Manager
10-13-2020
07:23 AM
Hello. In my cloudera cluser i'm having constantly working script(hes never stop) and logs in local dir's always grows ( local folder /yarn/nm/). [root@sdl-masternode ~]# du -s -h /yarn/nm/usercache/azat/appcache/*
658M /yarn/nm/usercache/azat/appcache/application_1599754680433_0192
2.1G /yarn/nm/usercache/azat/appcache/application_1599754680433_0203
12K /yarn/nm/usercache/azat/appcache/application_1599754680433_0210
6.7G /yarn/nm/usercache/azat/appcache/application_1599754680433_0241
605M /yarn/nm/usercache/azat/appcache/application_1599754680433_0254
40K /yarn/nm/usercache/azat/appcache/application_1600498608856_0120
2.3G /yarn/nm/usercache/azat/appcache/application_1600498608856_0150
12G /yarn/nm/usercache/azat/appcache/application_1600498608856_0155
1.2G /yarn/nm/usercache/azat/appcache/application_1600498608856_0381
4.6G /yarn/nm/usercache/azat/appcache/application_1600498608856_0386
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0398
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0400
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0613
46M /yarn/nm/usercache/azat/appcache/application_1600498608856_0688
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0966
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0974
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0983
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0986
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0989
12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0991
16K /yarn/nm/usercache/azat/appcache/application_1600498608856_1002
96K /yarn/nm/usercache/azat/appcache/application_1600498608856_1003 How can i automatically clean it, for example, if log older then 7 days - its must be retain from local.
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
09-22-2020
07:16 AM
Hello. I have activity monitor in green state. In logs I have smth like this: ps Sorry for quality, have no internet access in my cluster. But when I open navigator web ui I see this:
... View more
Labels:
- Labels:
-
Cloudera Navigator
09-19-2020
05:51 AM
Hello. I need advice there I can see log(audit) of hdfs operations. For example(mkdir, putFromLocal, delete file,change file). I tried use command from cmd "hdfs dfs -mkdir /tmp/mydir" - and don't see this log. There can I find it?
... View more
Labels:
- Labels:
-
HDFS
09-17-2020
05:33 AM
1 Kudo
Miracle! It's start work while I delete file "config.ini.orig" from folder /etc/cloudera-scm/agent/ Now there only "config.ini" file.
... View more
09-17-2020
04:53 AM
I did it, but doesn't work. idk for cloudera server - all ok, but for agents - doesn't works. (example for CM admin console, for hue - looks like this too).
... View more
09-17-2020
03:09 AM
There is some photo (sorry for quality, machine w/o internet access): 1) Log file /var/log/cloudera-scm-agent/cloudera-scm-agent.log 2) Administration-> Settings -> TLS in CM: 3)Configuration file in /etc/cloudera-scm-agent/config.ini
... View more
09-15-2020
03:25 AM
I has enabled TLS Encryption for agents, but when I change file "/etc/cloudera-scm-agent/config.ini" to use_tls=1 and use path for my CA certs - its doesn't apply, in log I see parameters "use_tls=0 and standard auto_tls path to files".
... View more
09-14-2020
07:59 AM
Hello. I'm trying to use my own certs from CA(I converted them to right format) I'm already configurated TLS for cloudera manager server (when I signed on https://{MASTER_IP}:7183 - I see it. File cm_init.txt setsettings AGENT_TLS true setsettings WEB_TLS true setsettings NEED_AGENTS_VALIDATION true setsettings AUTO_TLS_TYPE NONE setsettings KEYSTORE_PATH /opt/agentcerts/MYKEYSTORE.jks setsettings KEYSTORE_PASSWORD MYPASSWORD setsettings TRUSTSTORE_PATH /opt/agentcerts/MYTRUSTSTORE.jks setsettings TRUSTSTORE_PASSWORD MYPASSWORD Also I used them for HUE web UI and Hue Load Balancer - its also work too. But, I need to configurate certs to agents too, because I didn't see status of my agents in (hosts ->all hosts tab). I changed file /etc/cloudera-scm-agent/config.ini, there main changes on him: use_tls=1 verify_cert_file=/opt/agentcerts/MYCERT.pem client_key_file=/opt/agentcerts/MYKEY.pem client_keypw_file=/opt/agentcerts/KEYPASS.pw client_cert_file=/opt/agentcerts/MYCERT.pem verify_cert_dir=/opt/agentcerts/ But in log file /var/log/cloudera-scm-agent/cloudera-scm-agent.log I still see defaults conf while it starts: [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Agent Logging Level: INFO [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Agent config: [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.max_cert_depth = 9 [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.use_tls = 0 [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_cert_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.verify_cert_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_key_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_keypw_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.cm_auto_cert_dir = /var/lib/cloudera-scm-agent/agent-cert Especially Idk why in log I see what use_tls=0.
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Security