Re: hdfs audit operations

Yuriy_but · ‎12-07-2020

Hi. I have cluster with 12 nodes(9 of them with datanodes). On each host about 30tb capacity, used on datanodes ~15TB (20Vcores on each and 280gb ram). After restart one of VM with DN, datanode was started about 1.5 hours. Is it normal? Or how can i decrease this time.

Yuriy_but · ‎12-07-2020

Hi. I have a problem with outOfMemory error on impala catalog server. Data in tables about 30TB and about 175k partitiions. I set Java Heap Size of Catalog Server in Bytes =30gb, but still have out_of_memory error in start. How can i calculate how much memory needs for catalog server? And is it have any limit to set ?

Yuriy_but · ‎11-17-2020

Hello. I configurate TLS certs for HUE(from CA), on hdfs i used AUTO_TLS certs. My config Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini [desktop] app_blacklist= ssl_validate=false use_new_editor=true [notebook] [[interpreters]] [[ldap]] sync_groups_on_login=true show_notebooks=true [spark] livy_server_url=http://myhost:8998 interface=livy [pyspark] name=PySpark interface=livy [r] name=R interface=livy [hadoop] [[hdfs_clusters]] [[[default]]] ssl_cert_ca_verify = false [[ssl]] validate=false [impala] [[ssl]] validate=false In hue i see this:

Yuriy_but · ‎10-23-2020

Yes, i thought about cron. But maybe smth in yarn conf can help, but i miss it. Maximum log files bk doesnt help

Yuriy_but · ‎10-16-2020

Hello. I have cluster 9 nodes (400gb memory on each, 2 masters with NN). Capacity will be increase every time(i add ~10TB every week). For 31millon files used ~9gb of 15. What maximum java heap size for NN i can set and what recommended blocksize for big cluster(now its 128mb, should i set 256/512 ?)

Yuriy_but · ‎10-13-2020

Hello. In my cloudera cluser i'm having constantly working script(hes never stop) and logs in local dir's always grows ( local folder /yarn/nm/). [root@sdl-masternode ~]# du -s -h /yarn/nm/usercache/azat/appcache/* 658M /yarn/nm/usercache/azat/appcache/application_1599754680433_0192 2.1G /yarn/nm/usercache/azat/appcache/application_1599754680433_0203 12K /yarn/nm/usercache/azat/appcache/application_1599754680433_0210 6.7G /yarn/nm/usercache/azat/appcache/application_1599754680433_0241 605M /yarn/nm/usercache/azat/appcache/application_1599754680433_0254 40K /yarn/nm/usercache/azat/appcache/application_1600498608856_0120 2.3G /yarn/nm/usercache/azat/appcache/application_1600498608856_0150 12G /yarn/nm/usercache/azat/appcache/application_1600498608856_0155 1.2G /yarn/nm/usercache/azat/appcache/application_1600498608856_0381 4.6G /yarn/nm/usercache/azat/appcache/application_1600498608856_0386 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0398 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0400 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0613 46M /yarn/nm/usercache/azat/appcache/application_1600498608856_0688 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0966 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0974 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0983 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0986 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0989 12K /yarn/nm/usercache/azat/appcache/application_1600498608856_0991 16K /yarn/nm/usercache/azat/appcache/application_1600498608856_1002 96K /yarn/nm/usercache/azat/appcache/application_1600498608856_1003 How can i automatically clean it, for example, if log older then 7 days - its must be retain from local.

Yuriy_but · ‎09-22-2020

Hello. I have activity monitor in green state. In logs I have smth like this: ps Sorry for quality, have no internet access in my cluster. But when I open navigator web ui I see this:

Yuriy_but · ‎09-22-2020

It's start working while I added this parameters.

Yuriy_but · ‎09-19-2020

Hello. I need advice there I can see log(audit) of hdfs operations. For example(mkdir, putFromLocal, delete file,change file). I tried use command from cmd "hdfs dfs -mkdir /tmp/mydir" - and don't see this log. There can I find it?

Yuriy_but · ‎09-17-2020

Miracle! It's start work while I delete file "config.ini.orig" from folder /etc/cloudera-scm/agent/ Now there only "config.ini" file.

Yuriy_but · ‎09-17-2020

I did it, but doesn't work. idk for cloudera server - all ok, but for agents - doesn't works. (example for CM admin console, for hue - looks like this too).

Yuriy_but · ‎09-17-2020

There is some photo (sorry for quality, machine w/o internet access): 1) Log file /var/log/cloudera-scm-agent/cloudera-scm-agent.log 2) Administration-> Settings -> TLS in CM: 3)Configuration file in /etc/cloudera-scm-agent/config.ini

Yuriy_but · ‎09-15-2020

I has enabled TLS Encryption for agents, but when I change file "/etc/cloudera-scm-agent/config.ini" to use_tls=1 and use path for my CA certs - its doesn't apply, in log I see parameters "use_tls=0 and standard auto_tls path to files".

Yuriy_but · ‎09-14-2020

Hello. I'm trying to use my own certs from CA(I converted them to right format) I'm already configurated TLS for cloudera manager server (when I signed on https://{MASTER_IP}:7183 - I see it. File cm_init.txt setsettings AGENT_TLS true setsettings WEB_TLS true setsettings NEED_AGENTS_VALIDATION true setsettings AUTO_TLS_TYPE NONE setsettings KEYSTORE_PATH /opt/agentcerts/MYKEYSTORE.jks setsettings KEYSTORE_PASSWORD MYPASSWORD setsettings TRUSTSTORE_PATH /opt/agentcerts/MYTRUSTSTORE.jks setsettings TRUSTSTORE_PASSWORD MYPASSWORD Also I used them for HUE web UI and Hue Load Balancer - its also work too. But, I need to configurate certs to agents too, because I didn't see status of my agents in (hosts ->all hosts tab). I changed file /etc/cloudera-scm-agent/config.ini, there main changes on him: use_tls=1 verify_cert_file=/opt/agentcerts/MYCERT.pem client_key_file=/opt/agentcerts/MYKEY.pem client_keypw_file=/opt/agentcerts/KEYPASS.pw client_cert_file=/opt/agentcerts/MYCERT.pem verify_cert_dir=/opt/agentcerts/ But in log file /var/log/cloudera-scm-agent/cloudera-scm-agent.log I still see defaults conf while it starts: [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Agent Logging Level: INFO [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Agent config: [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.max_cert_depth = 9 [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.use_tls = 0 [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_cert_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.verify_cert_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_key_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.client_keypw_file = /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw [14/Sep/2020 14:09:36 +0000] 2560 MainThread agent INFO Security.cm_auto_cert_dir = /var/lib/cloudera-scm-agent/agent-cert Especially Idk why in log I see what use_tls=0.

Online	Offline
Last Visited	‎01-12-2022 11:23 AM

Date Registered	‎04-30-2020 05:14 AM
Last Visited	‎01-12-2022 11:23 AM
Total Messages Posted	15
Total Kudos Received	1

Re: hdfs audit operations

Re: cloudera agent TLS configuration

data node long start

impala catalog server JVM

Hue SSL cert error to HDFS

Re: yarn local logs

heap size of namenode/ blocksize

yarn local logs

Activity monitory(navigator) http error 503

Re: hdfs audit operations

hdfs audit operations

Re: cloudera agent TLS configuration

Re: cloudera agent TLS configuration

Re: cloudera agent TLS configuration

Re: cloudera agent TLS configuration

cloudera agent TLS configuration