Member since
02-02-2021
116
Posts
2
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
745 | 08-13-2021 09:44 AM | |
3692 | 04-27-2021 04:23 PM | |
1369 | 04-26-2021 10:47 AM | |
923 | 03-29-2021 06:01 PM | |
2746 | 03-17-2021 04:53 PM |
10-15-2021
06:05 PM
Hi experts, When trying to register a new version to upgrade my current cluster, I can see the new "register version" under manage Ambari > Versions. Then when I click on Install on... next to the version I want to install, it brings me to the "Admin" tab...Stack and Versions > Versions tab. However, I only see the version that is currently installed, but do not see the version that I want to install on this page. Can someone please help or direct me to some documentation to help me resolve this issue? Thanks,
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
10-15-2021
05:59 PM
Thanks @Faizan_Ali for the explanation. So in other words, once the job completes, then these logs will be stored in HDFS, or where are the logs stored after the application job is completed? Are the local and log yarn dirs are only for temporary use, usually when a job runs in the hadoop cluster? Thanks,
... View more
10-13-2021
06:15 AM
Hi experts, I just wanted to confirm my understanding or help me better understand the yarn local and log dirs. So my understanding is that yarn will download the data locally to a filesystem so that it is more easily accessible when a job is run as well as logs for that particular application. I believe these are temporary files as they will be stored in HDFS after the job completes. [root@test01 log]# ll /hadoop/yarn/ total 0 drwxr-xr-x. 6 yarn hadoop 78 Oct 13 08:09 local drwxrwxr-x. 8 yarn hadoop 239 Oct 13 08:09 log Can someone please help confirm my understanding or help me better understand this concept? Also what is usually best practice in regards to mounting these directories onto a local filesystem or onto another hard drive or can I have this directory share a hard drive with one of the datanode directories? Any help is much appreciated. Thanks,
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN
10-07-2021
08:52 AM
Hi @pvishnu Thanks for the response. So this is a new cluster and ranger was already previously installed on the node so I modified some of the data in the ambari postgres db to make it think that Ranger is already installed. Is there any documentation on what I should do to make sure that everything is synced up? Thanks,
... View more
09-30-2021
09:51 AM
Hi experts, I am currently trying to start/stop Ranger via ambari and got the below error. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 231, in <module>
RangerAdmin().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 89, in start
import params
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/params.py", line 138, in <module>
cred_validator_file = format('{usersync_home}/native/credValidator.uexe')
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 95, in format
return ConfigurationFormatter().format(format_string, args, **result)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 59, in format
result_protected = self.vformat(format_string, args, all_params)
File "/usr/lib64/python2.7/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
File "/usr/lib64/python2.7/string.py", line 571, in _vformat
obj, arg_used = self.get_field(field_name, args, kwargs)
File "/usr/lib64/python2.7/string.py", line 632, in get_field
obj = self.get_value(first, args, kwargs)
File "/usr/lib64/python2.7/string.py", line 591, in get_value
return kwargs[key]
File "/usr/lib/python2.6/site-packages/resource_management/core/utils.py", line 63, in __getitem__
return self._convert_value(self._dict[name])
KeyError: 'usersync_home' Any help is much appreciated. Thanks,
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache Ranger
09-23-2021
05:56 AM
Ambari version 2.6.1 So I was not adding Hbase. I was trying to add a mpack so I tried to recreate the tar.gz file and install the mpack via command line. I may have been doing things too fast and may have accidentally deleted something. Not sure. But I don't think I touched Hbase.
... View more
09-22-2021
07:18 PM
Hi experts, I was doing some experimenting on this sandbox cluster to installing new components and now my ambari-server process won't start. Below is the error message i see in ambari-server.log when i try to run the ambari-server process. Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 Sep 22, 2021 9:16:02 PM com.google.inject.internal.ProxyFactory <init> WARNING: Method [public void org.apache.ambari.server.orm.dao.HostVersionDAO.create(java.lang.Object)] is synthetic and is being intercepted by [org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor@44841b43]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Sep 22, 2021 9:16:02 PM com.google.inject.internal.ProxyFactory <init> WARNING: Method [public void org.apache.ambari.server.orm.dao.RepositoryVersionDAO.create(java.lang.Object)] is synthetic and is being intercepted by [org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor@44841b43]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. Sep 22, 2021 9:16:03 PM com.google.inject.internal.ProxyFactory <init> WARNING: Method [public java.lang.Object org.apache.ambari.server.topology.tasks.ConfigureClusterTask.call() throws java.lang.Exception] is synthetic and is being intercepted by [org.apache.ambari.server.security.authorization.internal.InternalAuthenticationInterceptor@25d2f66]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all. An unexpected error occured during starting Ambari Server. com.google.inject.ProvisionException: Guice provision errors: 1) Error injecting method, java.lang.RuntimeException: Trying to create a ServiceComponent not recognized in stack info, clusterName=test, serviceName=HBASE, componentName=HBASE_REST_SERVER, stackInfo=BGTP-1.0 at org.apache.ambari.server.state.cluster.ClustersImpl.loadClustersAndHosts(ClustersImpl.java:173) at org.apache.ambari.server.state.cluster.ClustersImpl.class(ClustersImpl.java:95) while locating org.apache.ambari.server.state.cluster.ClustersImpl while locating org.apache.ambari.server.state.Clusters for parameter 0 at org.apache.ambari.server.agent.HeartBeatHandler.<init>(HeartBeatHandler.java:115) at org.apache.ambari.server.agent.HeartBeatHandler.class(HeartBeatHandler.java:79) while locating org.apache.ambari.server.agent.HeartBeatHandler 1 error at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013) at org.apache.ambari.server.controller.AmbariServer.performStaticInjection(AmbariServer.java:899) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:307) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1068) Caused by: java.lang.RuntimeException: Trying to create a ServiceComponent not recognized in stack info, clusterName=test, serviceName=HBASE, componentName=HBASE_REST_SERVER, stackInfo=BGTP-1.0 at org.apache.ambari.server.state.ServiceComponentImpl.updateComponentInfo(ServiceComponentImpl.java:146) at org.apache.ambari.server.state.ServiceComponentImpl.<init>(ServiceComponentImpl.java:170) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy19.createExisting(Unknown Source) at org.apache.ambari.server.state.ServiceImpl.<init>(ServiceImpl.java:163) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy15.createExisting(Unknown Source) at org.apache.ambari.server.state.cluster.ClusterImpl.loadServices(ClusterImpl.java:428) at org.apache.ambari.server.state.cluster.ClusterImpl.<init>(ClusterImpl.java:318) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) at com.sun.proxy.$Proxy11.create(Unknown Source) at org.apache.ambari.server.state.cluster.ClustersImpl.loadClustersAndHosts(ClustersImpl.java:181) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128) at org.apache.ambari.server.state.cluster.ClustersImpl$$FastClassByGuice$$7d58855f.invoke(<generated>) at com.google.inject.internal.cglib.reflect.$FastMethod.invoke(FastMethod.java:53) at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:56) at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:90) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ... 4 more Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1076) Any help is much appreciated. Thanks,
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache HBase
09-09-2021
09:25 AM
hi experts,
When trying to test tez, i get the following error.
[root@test01 ~]# hadoop --config /etc/hadoop/conf jar /usr/lib/tez/tez-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
21/09/09 11:23:56 INFO shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=2.10.1, majorVersion=2, minorVersion=10
21/09/09 11:23:56 INFO shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.HadoopShim28, providerName=org.apache.tez.hadoop.shim.HadoopShim28Provider, overrideProviderViaConfig=null, hadoopVersion=2.10.1, majorVersion=2, minorVersion=10
21/09/09 11:23:57 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.9.2, revision=81ad7b000cec0503b9a1d5521fdaf0129443b536, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2020-11-28T13:10:15Z ]
21/09/09 11:23:57 INFO client.RMProxy: Connecting to ResourceManager at test01.com/10.49.4.11:8050
21/09/09 11:23:57 INFO client.AHSProxy: Connecting to Application History server at test02.com/10.49.4.12:10200
21/09/09 11:23:57 INFO examples.OrderedWordCount: Running OrderedWordCount
21/09/09 11:23:57 INFO client.TezClient: Submitting DAG application with id: application_1629805664278_0020
21/09/09 11:23:57 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /bgtp/apps/1.0/tez/tez.tar.gz
21/09/09 11:23:57 INFO client.TezClientUtils: Using tez.lib.uris.classpath value from configuration: null
21/09/09 11:23:58 INFO client.TezClient: Tez system stage directory hdfs://test/tmp/root/staging/.tez/application_1629805664278_0020 doesn't exist and is created
21/09/09 11:23:58 INFO conf.Configuration: resource-types.xml not found
21/09/09 11:23:58 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
21/09/09 11:23:58 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
21/09/09 11:23:58 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
21/09/09 11:23:58 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1629805664278_0020, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
21/09/09 11:23:58 INFO impl.YarnClientImpl: Submitted application application_1629805664278_0020
21/09/09 11:23:58 INFO client.TezClient: The url to track the Tez AM: http://test01.com:8088/proxy/application_1629805664278_0020/
21/09/09 11:23:59 INFO client.TezClient: App did not succeed. Diagnostics: Application application_1629805664278_0020 failed 2 times due to AM Container for appattempt_1629805664278_0020_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-09-09 11:23:59.041]Exception from container-launch.
Container id: container_1629805664278_0020_02_000001
Exit code: 1
[2021-09-09 11:23:59.042]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster
[2021-09-09 11:23:59.042]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster
For more detailed output, check the application tracking page: http://test01.com:8088/cluster/app/application_1629805664278_0020 Then click on links to logs of each attempt.
. Failing the application.
21/09/09 11:23:59 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
21/09/09 11:23:59 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1629805664278_0020 failed 2 times due to AM Container for appattempt_1629805664278_0020_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-09-09 11:23:59.041]Exception from container-launch.
Container id: container_1629805664278_0020_02_000001
Exit code: 1
[2021-09-09 11:23:59.042]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster
[2021-09-09 11:23:59.042]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster
For more detailed output, check the application tracking page: http://test01.com:8088/cluster/app/application_1629805664278_0020 Then click on links to logs of each attempt.
. Failing the application.]
[root@test01 ~]#
Any help is much appreciated.
Thanks,
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Tez
09-09-2021
08:54 AM
Yes, I have copied tables from another cluster to this current cluster.
... View more
09-08-2021
09:12 PM
Also here is the application logs. [hive@sunnymaster01 ~]$ cat application_1629805664278_0004.log End of LogType:prelaunch.err ****************************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:prelaunch.out LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stderr LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:77 LogContents: Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster End of LogType:stderr *********************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stdout LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:723 LogContents: Heap PSYoungGen total 149504K, used 5140K [0x00000000e6700000, 0x00000000f0d80000, 0x0000000100000000) eden space 128512K, 4% used [0x00000000e6700000,0x00000000e6c05208,0x00000000ee480000) from space 20992K, 0% used [0x00000000ef900000,0x00000000ef900000,0x00000000f0d80000) to space 20992K, 0% used [0x00000000ee480000,0x00000000ee480000,0x00000000ef900000) ParOldGen total 341504K, used 0K [0x00000000b3400000, 0x00000000c8180000, 0x00000000e6700000) object space 341504K, 0% used [0x00000000b3400000,0x00000000b3400000,0x00000000c8180000) Metaspace used 2971K, capacity 4550K, committed 4864K, reserved 1056768K class space used 316K, capacity 386K, committed 512K, reserved 1048576K End of LogType:stdout *********************************************************************** End of LogType:prelaunch.err ****************************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:prelaunch.out LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stderr LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:77 LogContents: Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster End of LogType:stderr *********************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stdout LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:723 LogContents: Heap PSYoungGen total 149504K, used 5140K [0x00000000e6700000, 0x00000000f0d80000, 0x0000000100000000) eden space 128512K, 4% used [0x00000000e6700000,0x00000000e6c05208,0x00000000ee480000) from space 20992K, 0% used [0x00000000ef900000,0x00000000ef900000,0x00000000f0d80000) to space 20992K, 0% used [0x00000000ee480000,0x00000000ee480000,0x00000000ef900000) ParOldGen total 341504K, used 0K [0x00000000b3400000, 0x00000000c8180000, 0x00000000e6700000) object space 341504K, 0% used [0x00000000b3400000,0x00000000b3400000,0x00000000c8180000) Metaspace used 2973K, capacity 4550K, committed 4864K, reserved 1056768K class space used 316K, capacity 386K, committed 512K, reserved 1048576K End of LogType:stdout *********************************************************************** Any help is much appreciated. Thanks,
... View more