Member since
12-20-2022
87
Posts
19
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
411 | 05-08-2025 06:27 AM | |
460 | 04-02-2025 11:35 PM | |
407 | 03-23-2025 11:30 PM | |
502 | 03-06-2025 10:11 PM | |
1132 | 10-29-2024 11:53 PM |
05-08-2025
06:27 AM
Hi @anonymous_123 , Generally the RM heap calculation depends on the yarn.resourcemanager.max-completed-applications value and the number of applications running daily. Default value for yarn.resourcemanager.max-completed-applications is 10000 but if you see that you dont have enough applications running you can set this to 6000. Regarding 4GB heap that is production level RM heap and it is fine if you are not seeing any heap related errors.
... View more
04-15-2025
11:15 AM
Hi @satvaddi , If you are running in a Ranger RAZ enabled environment you don't need all these settings: > --conf "spark.hadoop.hadoop.security.authentication=KERBEROS" \ > --conf "spark.hadoop.hadoop.security.authorization=true" \ > --conf "spark.hadoop.fs.s3a.delegation.token.binding=org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding" \ > --conf "spark.hadoop.fs.s3a.idb.auth.token.enabled=true" \ > --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider" \ > --conf "spark.hadoop.fs.s3a.security.credential.provider.path=jceks://hdfs/user/infa/knox_credentials.jceks" \ > --conf "spark.hadoop.fs.s3a.endpoint=s3.amazonaws.com" \ > --conf "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \ To me it looks like you are bypassing Raz by setting this parameter: > --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider" \ This, I would check whether the instance profile (IAM Role attached to the cluster) does not have too much privileges. Like access to data. This should be controlled in Ranger instead.
... View more
04-02-2025
11:35 PM
1 Kudo
Hi @anonymous_123 , Yes you can use Iceberg Table with Spark and to authorise with Ranger. You need to set two permissions one for the Iceberg Metadata files and One for global policy to give permission to iceberg on all tables. Please follow this document https://docs.cloudera.com/runtime/7.3.1/iceberg-how-to/topics/iceberg-setup-ranger.html
... View more
03-24-2025
01:56 AM
In YARN, resource allocation discrepancies can occur due to the way resource calculation is handled. By default, resource availability is determined based on available memory. However, when CPU scheduling is enabled, resource calculation considers both available memory and vCores. As a result, in some scenarios, nodes may appear to allocate more vCores than the configured limit while simultaneously displaying lower available resources. This happens due to the way YARN dynamically assigns vCores based on workload demands rather than strictly adhering to preconfigured limits. Additionally, in cases where CPU scheduling is disabled, YARN relies solely on memory-based resource calculation. This may lead to negative values appearing in the YARN UI, which can be safely ignored, as they do not impact actual resource utilization.
... View more
Labels:
03-23-2025
11:30 PM
1 Kudo
No the job wont fail as by default the work preserve is enabled on YARN Resource Manager and Node Manager.
... View more
03-06-2025
10:11 PM
Hi @sdbags , You can recover the corrupted block if you have set the replication factor to default of 3.
... View more
11-20-2024
08:19 AM
@hadoopranger Consider tuning parameters like idle_session_timeout and idle_query_timeout which is suspected to be closing the session considering you don't have LB in place. You can also set it to 0 where session will never expire until closed manually. Moreover, consider increasing value of fe_service_threads to allow more concurrent client connections which should help you avoiding similar client connection issue in future. For more info, refer: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/impala-reference/topics/impala-recommended-configs.html
... View more
11-13-2024
03:48 PM
1 Kudo
@satz The image is the result of the last check step, which is to connect to localhost:7180 and add the cluster after installing the OS using the commands on the trial install page. In this situation, the status of the cluster shows that namenode is down. After that, no matter how many times you refresh the page or restart through the command in the OS, you will get the access denied (page cannot be opened) message when you connect to the CM. Also, I install it on one OS and proceed, but sometimes, during the process of adding a cluster, only the OS installed at the time of host inspection is installed in the cluster, and the rest of the OSs are removed. (Sometimes I can completely reinstall the OSs afterward, and sometimes I can't, but I don't know the difference. The log is so long that I want to upload the file, but there is no upload function.
... View more
10-30-2024
08:52 PM
Thanks. I guess it is not for total_io_mb but for "memory and v-core utilization". Disk io should be in yarn logs. I will try seek it.
... View more
10-10-2024
05:13 AM
1 Kudo
Hi @evanle96 This error is not an issue. Usually in the HA setup the call goes to both the NN and the Active NN acknowledges the call but the standby NN will through this warning. So you can ignore this warning here.
... View more