Member since
02-28-2022
55
Posts
0
Kudos Received
0
Solutions
06-14-2022
01:42 PM
@MattWho / @bbahamondes I made alternate stops of the nifi nodes and with that I was able to access the nifi and configure the policies. Thank you for your help.
... View more
06-14-2022
12:37 PM
hi, @bbahamondes / @MattWho in the configuration "nifi.initial.admin.identity" in nifi, I left only the user "user.bind", stopped nifi, deleted the files "authorizations.xml" and "users.xml" and started nifi I don't know if it worked, but it's not giving the permission error anymore, but it's giving connection error on port 8080 when I try to login PS: in the configuration "Login Identity Provider: Default LDAP Identity Strategy" in the nifi is configured "USE_USERNAME" PS: when I try to login with a different user it returns the permission error: "Unknown user with identity 'user'. Contact the system administrator."
... View more
06-14-2022
10:58 AM
I'm configuring LDAP access (AD) on Nifi 1.0.1.0 being managed by Cloudera Manager, but when trying to access nifi with AD credentials, the error is returned: "Unknown user with identity 'user.bind'. Contact the system administrator." In Nifi's "nifi.initial.admin.identity" settings it looks like this: "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" What could be wrong for not being able to access?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera Manager
06-06-2022
01:10 PM
how to consult the queries that are running in HIVE by TEZ VIEW that is in AMBARI version 2.6.2.2 by API? Via web is accessing normally, we need to consult the queries via TEZ VIEW API
... View more
Labels:
05-30-2022
06:16 AM
hi @rblough With the information given earlier, what could be causing this problem?
... View more
05-27-2022
07:28 AM
hi @rblough yes, the tablets server are balanced, today there are 1443 replicas, dividing by 5 tablet server gives a total of 288 replicas per tablet server, as shown by each tablet server below: hostname04.local - replicas by tablets - RUNNING-289 hostname05.local - replicas by tablets - RUNNING-288 hostname06.local - replicas by tablets - RUNNING-289 hostname07.local - replicas by tablets - RUNNING-289 hostname08.local - replicas by tablets - RUNNING-288 there have been times when host 06 was also used above 85% to 90% of memory usage. when we restart the tablet server that have high memory usage, there are times when the tablet server itself goes back up the memory usage or another tablet server that has low memory usage starts to go up, getting above 85 % too.
... View more
05-25-2022
11:08 AM
hi @rblough 1) - answer: Masters | 3 Tablet Servers | 5 Tables | 48 Tablets | 481 Replicas | 1443 2) - answer: According to the chart "Total Tablet Size On Disk Across Kudu Replicas", it has more than 710GB 3) - answer: exist 5 4) - answer: exist 3 threads 5) - answer: memory_limit_hard_bytes = 60GB block_cache_capacity_mb = 15GB
... View more
05-25-2022
09:26 AM
kudu service has 3 kudu master and 5 kudu tablet among these 5 kudu tablet, 3 of them have high memory usage, 60GB has been set for each kudu tablet and it is using more than 85% to 90% of that memory and with that it ends up causing jobs failure below shows the memory usage behavior: TABLET SERVER: hostname04.local - USE MEMORY: 55.28G / %: 92.13% TABLET SERVER: hostname05.local - USE MEMORY: 54.22G / %: 90.38% TABLET SERVER: hostname06.local - USE MEMORY: 29.05G / %: 48.42% TABLET SERVER: hostname07.local - USE MEMORY: 28.47G / %: 47.46% TABLET SERVER: hostname08.local - USE MEMORY: 54.99G / %: 91.66% memory usage graph of all kudu tablet in cloudera manager: would like to understand why kudu is having this behavior and how can we reduce this memory usage? PS: kudu service is installed and being managed by cloudera manager express in version 5.16.2 and kudu service is in version 1.7
... View more
Labels:
- Labels:
-
Apache Kudu
-
Cloudera Manager
05-24-2022
10:43 AM
thank you @ask_bill_brooks I had seen this thread you passed, but it doesn't explicitly say which version of CM/CDH to run github's log4j script procedure. but I will consider version 5.x and 6.x to run this script.
... View more
05-24-2022
08:42 AM
Hello about log4j2 failure (CVE-2021-44228) in cloudera there is the KB: https://my.cloudera.com/knowledge/Resolution-for-TSB-2021-545---Critical-vulnerability-in-log4j2?id=332012 instructs to run the script to remove a class in the jar etc. our doubt is, do we need to run this script in any version of CM / CDH or just the version from 6.x.x?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera Manager
05-20-2022
08:38 AM
LB AWS sticky sessions settings is for LB HTTP/HTTPS only in this case the LB we are using is TCP on ports 21000 and 21050 the session timeout you say in the impala service, right? would be the "idle_query_timeout" and "idle_session_timeout" options, right?
... View more
05-20-2022
05:22 AM
hi @aakulov the cluster is on AWS and the AWS LB is being used on ports 21050 of the impala daemon, in which case this LB may be having this problem? sould it be that in this LB we can also change it to "source"?
... View more
05-19-2022
02:08 PM
queries in hue using impala are returning the following error: "results have expired, rerun the query if needed" as shown in the screenshot below: what could be causing this error and how to fix this issue?
... View more
Labels:
05-17-2022
02:19 PM
When we restart or start kudu tablet server component through cloudera manager, it takes a long time to communicate with kudu master, it takes more than 10 minutes. why does this occur? when we start the kudu tablet server component and follow the log, it keeps loading the lines as shown in the screenshot below:
... View more
- Tags:
- Apache Kudu
- kudu
Labels:
- Labels:
-
Apache Kudu
-
Cloudera Manager
05-03-2022
11:06 AM
hi @GangWar , I followed the behavior of the host and when the error appeared in the cloudera manager I ran ethtool and it returned the information without any problem.
... View more
04-29-2022
05:34 AM
hi @ChethanYM unfortunately we need to do this type of blocking only in IMPALA, in HIVE it has to work normally. when blocking by sentry, it will be replicated for any type of access that the user will use, such as hive, impala, beeline, etc.
... View more
04-28-2022
10:52 AM
we need to block the DDL (create, drop, insert, update) for whoever is using IMPALA, is it possible to do this type of blocking? ps: user should only be able to SELECT with IMPALA and nothing else. DDL should only work via HIVE. Is it possible to do this type of blocking configuration?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
-
Cloudera Manager
04-28-2022
09:02 AM
hi@ChethanYM Thank you for the link. another question that popped up: we didn't find anything saying if the coordinators can be used in virtual machines, can you tell us if there is a problem?
... View more
04-28-2022
07:44 AM
hi @GangWar , I followed the behavior of the host and when the error appeared in the cloudera manager I ran ethtool and it returned the information without any problem.
... View more
04-28-2022
05:59 AM
hello @Elias / @GangWar , how are you? I ran ethtool on all hosts in the cloudera cluster and the network interface speed and duplex mode were returned successfully, this shows that ethtool is running smoothly. with ethtool working fine, what could be generating these alerts and leaving the agent not sending a heartbeat for a few seconds?
... View more
04-26-2022
08:54 AM
we are getting a lot of "Network Interface Speed Unknown" and "Agent Status Bad" alerts on two specific hosts in the cluster and when accessing these hosts through cloudera manager we collect the details. error details: "Network Interface Speed Unknown The health test result for HOST_NETWORK_INTERFACES_SLOW_MODE has become unknown: Not enough data to test: Test of whether the host has network interfaces that appear to be operating at less than full speed. Agent Status Bad The health test result for HOST_SCM_HEALTH has become bad: This host is in contact with the Cloudera Manager Server. This host is not in contact with the Host Monitor." we checked the cloudera agent logs on both hosts and found no alerts that could be impacting these alerts. we would like to know what may be impacting to generate this alert?
... View more
Labels:
- Labels:
-
Cloudera Manager
04-25-2022
01:33 PM
accidentally deleted /config/topics directory within zookeeper zkcli. so far the kafka cluster is operational, but we need to know what this directory can interfere with in the future? Is it possible to restore this /config/topics directory in zookeeper?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Cloudera Manager
04-22-2022
08:50 AM
Is it possible to create a dynamic resource pool in YARN on Ambari? We need to limit the use of YARN resources per user. we need to configure thread queues for users separately, example: YARN has 1 TB of memory and 400 vCPU we need to configure the following: 1. The default queue occupies 50% (500 GB memory and 200 vCPU) 2. User1 queue occupies 25% (250 GB memory and 100 vCPU) 3. User2 queue occupies 25% (250 GB memory and 100 vCPU) all this configuration must be done in Ambari. how can we make this configuration?
... View more
Labels:
04-18-2022
10:31 AM
I would like to know how to calculate the number of coordinators and executors in the impala cluster? today I have 23 nodes with only impala daemon, what would be the best division of coordinators for and executors for impala?
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager
04-14-2022
01:35 PM
hi @bbahamondes, thank you for the explanation
... View more
04-14-2022
09:30 AM
hi, @mszurap, in the file "/run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE/supervisor.conf" has the following parameter "KRB5CCNAME=krb5cc_cldr" the "KRB5CCNAME" parameter is in the "environment" line of the "/run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE/supervisor.conf" file as shown below: environment=CDH_PARQUET_HOME='/usr/lib/parquet',CDH_MR2_HOME='/usr/lib/hadoop-mapreduce',CMF_PACKAGE_DIR='/opt/cloudera/cm-agent/service',CM_STATUS_CODES='STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE',MGMT_HOME='/opt/cloudera/cm',CGROUP_ROOT_CPU='/sys/fs/cgroup/cpu,cpuacct',CDH_IMPALA_HOME='/usr/lib/impala',CDH_YARN_HOME='/usr/lib/hadoop-yarn',nifi_service_principal='nifi/host-nifi-16.domain.local@TESTE.INT.BR',CGROUP_GROUP_CPU='system.slice/cloudera-scm-agent.service',CSD_JAVA_OPTS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/nifi_nifi-NIFI_NODE-ae57679df54eef6fea3afc54d3cb40ad_pid{{PID}}.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh',CDH_HDFS_HOME='/usr/lib/hadoop-hdfs',CSD_HOST='host-nifi-16.domain.local',HADOOP_CREDSTORE_PASSWORD='6d1ddiyksatrp1nwubz27h5jp',PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin',CDH_HUE_PLUGINS_HOME='/usr/lib/hadoop',PARCEL_DIRNAMES='CDH-6.3.4-1.cdh6.3.4.p0.6751098:CFM-1.0.1.0',CDH_HADOOP_BIN='/usr/bin/hadoop',KEYTRUSTEE_KP_HOME='/usr/share/keytrustee-keyprovider',CDH_KUDU_HOME='/usr/lib/kudu',CLOUDERA_ORACLE_CONNECTOR_JAR='/usr/share/java/oracle-connector-java.jar',CONF_DIR='/var/run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE',CDH_SQOOP2_HOME='/usr/lib/sqoop2',KEYTRUSTEE_SERVER_HOME='/usr/lib/keytrustee-server',JSVC_HOME='/usr/libexec/bigtop-utils',HOME='/var/lib/nifi',HIVE_DEFAULT_XML='/etc/hive/conf.dist/hive-default.xml',CUSTOM_JAVA_HOME='/usr/java/default',JAVA_HOME='/usr/java/jdk1.8.0_162',CLOUDERA_POSTGRESQL_JDBC_JAR='/opt/cloudera/cm/lib/postgresql-42.2.14.jre7.jar',CGROUP_GROUP_CPUACCT='system.slice/cloudera-scm-agent.service',CM_CSD_SCRIPT='scripts/control.sh',PARCELS_ROOT='/opt/cloudera/parcels',CDH_KMS_HOME='/usr/lib/hadoop-kms',ZK_QUORUM='host-nifi-14.domain.local:2183,host-nifi-15.domain.local:2183,host-nifi-16.domain.local:2183',CDH_HBASE_HOME='/usr/lib/hbase',CDH_SQOOP_HOME='/usr/lib/sqoop',CDH_HCAT_HOME='/usr/lib/hcatalog',CDH_OOZIE_HOME='/usr/lib/oozie',CDH_HIVE_HOME='/usr/lib/hive',CDH_ZOOKEEPER_HOME='/usr/lib/zookeeper',CDH_HUE_HOME='/usr/lib/hue',nifi_spnego_principal='HTTP/host-nifi-16.domain.local@TESTE.INT.BR',CGROUP_GROUP_DEVICES='system.slice',CGROUP_ROOT_MEMORY='/sys/fs/cgroup/memory',CLOUDERA_MYSQL_CONNECTOR_JAR='/usr/share/java/mysql-connector-java.jar',CDH_SPARK_HOME='/usr/lib/spark',CGROUP_GROUP_MEMORY='system.slice/cloudera-scm-agent.service',KRB5CCNAME='krb5cc_cldr',CDH_MR1_HOME='/usr/lib/hadoop-0.20-mapreduce',TOMCAT_HOME='/usr/lib/bigtop-tomcat',CDH_PIG_HOME='/usr/lib/pig',CDH_SENTRY_HOME='/usr/lib/sentry',CDH_VERSION='6',CGROUP_GROUP_BLKIO='system.slice/cloudera-scm-agent.service',NIFI_LOG_DIR='/var/log/nifi',CGROUP_ROOT_BLKIO='/sys/fs/cgroup/blkio',CDH_CRUNCH_HOME='/usr/lib/crunch',CDH_LLAMA_HOME='/usr/lib/llama/',CDH_HTTPFS_HOME='/usr/lib/hadoop-httpfs',CDH_HADOOP_HOME='/usr/lib/hadoop',ORACLE_HOME='/usr/share/oracle/instantclient',CMF_SERVER_ROOT='/opt/cloudera/cm',WEBHCAT_DEFAULT_XML='/etc/hive-webhcat/conf.dist/webhcat-default.xml',CDH_KAFKA_HOME='/usr/lib/kafka',CGROUP_ROOT_DEVICES='/sys/fs/cgroup/devices',CDH_HBASE_INDEXER_HOME='/usr/lib/hbase-solr',SCM_DEFINES_SCRIPTS='/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/meta/cdh_env.sh:/opt/cloudera/parcels/CFM-1.0.1.0/meta/nifi_bundle_env.sh',CDH_SOLR_HOME='/usr/lib/solr',CGROUP_ROOT_CPUACCT='/sys/fs/cgroup/cpu,cpuacct',CDH_FLUME_HOME='/usr/lib/flume-ng' so the cloudera manager itself is distributing this file "krb5cc_cldr"? PS: this file is in /tmp/krb5cc_cldr
... View more
04-14-2022
09:18 AM
hi @jeremymolina , how are you? thank you very much for the explanation.
... View more
04-14-2022
08:35 AM
hi @smdas, @shubham_sharma, @RangaReddy, @mszurap, can help with this queue issue on YARN in Ambari?
... View more
04-14-2022
08:31 AM
can we delete the files that exist in the directory configured in the "Heap Dump Directory" option of the services in CDH/CM?
... View more
Labels:
04-14-2022
08:16 AM
hi @mszurap , in the file "/run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE/supervisor.conf" has exactly the parameter "KRB5CCNAME='krb5cc_cldr'" do we need to know where he is getting this information from? we already looked in the .bashrc and .bash_profile of the root user and other users, nothing was found.
... View more