Member since
08-08-2018
49
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
13576 | 08-11-2018 12:05 AM | |
10376 | 08-10-2018 07:29 PM |
08-14-2018
11:17 PM
Hello, I just installed Hbase RegionServers to 2 of my datanodes. I'm getting the following error when I start them on HDP 3.0: esource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-regionserver/conf start regionserver' returned 127. -bash: /usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh: No such file or directory What am I missing? I installed them through Ambari.
... View more
Labels:
08-14-2018
04:42 PM
Hello, I'm looking to read tables from Phoenix using Zeppelin. I don't want to use the JDBC connection in favor of the plugin described here: https://phoenix.apache.org/phoenix_spark.html I believe I am not specifying the dependencies correctly because even though I added this to Zeppelin Interpreter (see zeppelin-phoenix-config.png). I'm still getting the following error: import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
error: object phoenix is not a member of package org.apache
How do I fix this? I've also added /usr/hdp/current/phoenix-server/lib/:/usr/hdp/current/phoenix-client/lib/:/usr/hdp/current/phoenix-server/ to both spark.driver.extraClassPath and spark.executor.extraClassPath in Spark2's spark.defaults.conf. Thanks in advance! P.S. I'm trying to figure all this out so that I can write a sparklyr extension to interact with Hbase data through Phoenix. Any pointers toward that goal are appreciated as well.
... View more
Labels:
08-13-2018
05:51 PM
@Akhil S Naik That worked! Nice job!
... View more
08-13-2018
03:33 PM
@Akhil S Naik Hm, I actually have all of those, any other ideas? <br>[root@NCIENSPK01 ~]# cd /usr/hdp/3.0.0.0-1634/hadoop-yarn
[root@NCIENSPK01 hadoop-yarn]# ls -l
total 12544
drwxr-xr-x 2 root root 93 Aug 10 17:51 bin
lrwxrwxrwx 1 root root 16 Aug 10 17:51 conf -> /etc/hadoop/conf
drwxr-xr-x 3 root root 30 Aug 10 17:51 etc
-rw-r--r-- 1 root root 3097487 Jul 12 16:02 hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 38 Aug 10 17:51 hadoop-yarn-api.jar -> hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 110407 Jul 12 16:02 hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 64 Aug 10 17:51 hadoop-yarn-applications-distributedshell.jar -> hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 55399 Jul 12 16:02 hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 69 Aug 10 17:51 hadoop-yarn-applications-unmanaged-am-launcher.jar -> hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 300539 Jul 12 16:02 hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 41 Aug 10 17:51 hadoop-yarn-client.jar -> hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2814263 Jul 12 16:02 hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 41 Aug 10 17:51 hadoop-yarn-common.jar -> hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 224598 Jul 12 16:02 hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 43 Aug 10 17:51 hadoop-yarn-registry.jar -> hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 294798 Jul 12 16:02 hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 67 Aug 10 17:51 hadoop-yarn-server-applicationhistoryservice.jar -> hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1336803 Jul 12 16:02 hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 48 Aug 10 17:51 hadoop-yarn-server-common.jar -> hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1293507 Jul 12 16:02 hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 53 Aug 10 17:51 hadoop-yarn-server-nodemanager.jar -> hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2172705 Jul 12 16:02 hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 57 Aug 10 17:51 hadoop-yarn-server-resourcemanager.jar -> hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 149224 Jul 12 16:02 hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 48 Aug 10 17:51 hadoop-yarn-server-router.jar -> hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 94342 Jul 12 16:02 hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 60 Aug 10 17:51 hadoop-yarn-server-sharedcachemanager.jar -> hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 43865 Jul 12 16:02 hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 47 Aug 10 17:51 hadoop-yarn-server-tests.jar -> hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 93081 Jul 12 16:02 hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 64 Aug 10 17:51 hadoop-yarn-server-timeline-pluginstorage.jar -> hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 80404 Jul 12 16:02 hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 51 Aug 10 17:51 hadoop-yarn-server-web-proxy.jar -> hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 103378 Jul 12 16:02 hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 47 Aug 10 17:51 hadoop-yarn-services-api.jar -> hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 539627 Jul 12 16:02 hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root 48 Aug 10 17:51 hadoop-yarn-services-core.jar -> hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
drwxr-xr-x 2 root root 4096 Aug 10 17:51 lib
drwxr-xr-x 2 root root 49 Aug 10 17:51 sbin
drwxr-xr-x 3 root root 4096 Aug 10 17:51 timelineservice
drwxr-xr-x 3 root root 16 Aug 10 17:51 webapps
drwxr-xr-x 5 root root 51 Aug 10 17:51 yarn-service-examples
Then in timeline service: [root@NCIENSPK01 hadoop-yarn]# ls timelineservice
hadoop-yarn-server-timelineservice-3.1.0.3.0.0.0-1634.jar hadoop-yarn-server-timelineservice-hbase-common-3.1.0.3.0.0.0-1634.jar hadoop-yarn-server-timelineservice-hbase-coprocessor.jar
hadoop-yarn-server-timelineservice-hbase-client-3.1.0.3.0.0.0-1634.jar hadoop-yarn-server-timelineservice-hbase-common.jar hadoop-yarn-server-timelineservice.jar
hadoop-yarn-server-timelineservice-hbase-client.jar
... View more
08-11-2018
01:21 AM
@amarnath reddy pappu Good idea! This is what I got: [root@DRIVER_NODE ambari-agent]# ./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
... View more
08-11-2018
12:39 AM
Interestingly I don't have anything in the logs from today at all. I have tried to restart it today.... I am looking in: /var/log/hadoop-yarn/yarn/hadoop-yarn-timelinereader-MYNODE01.log Is that right?
... View more
08-11-2018
12:08 AM
1 Kudo
Everything in my HDP is working except Timeline Service V2.0 Reader. When I try to start it I get the following error log: Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 108, in <module>
ApplicationTimelineReader().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 51, in start
hbase(action='start')
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 80, in hbase
createTables()
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 147, in createTables
logoutput=True)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 308, in _call
raise ExecuteTimeoutException(err_msg)
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds
... View more
Labels:
- Labels:
-
Apache Tez
-
Apache YARN
08-11-2018
12:05 AM
So I resolved all this. I just followed the steps here to remove all my packages, then deleted the contents of my files: rm -rf /usr/hdp/ Then in Ambari I used the "Start all Services" command and it went through and installed everything again for me. Then to solve the nodemanager issue I did the spark-yarn- install which jave me the missing jar that I needed and then just copied that dir: /usr/hdp/3.0.0.0-.../spark2/aux/ to all the other nodes in my cluster. Now all my nodemanagers are coming up and things are looking good. I'm creating another post about resolving my Timeline Service V2.0 issue which is somehow still persisting.
... View more
08-10-2018
09:20 PM
@Jay Kumar SenSharma I'm definitely in a jam now. Really hoping you can help me. A bit scared to touch anything at this point.
... View more
08-10-2018
07:29 PM
@Jay Kumar SenSharma When I try to start services now I'm getting: For HDFS Client Install RuntimeError: Failed to execute command '/usr/bin/yum -y install hadoop_3_0_0_0_1634', exited with code '1', message: 'Error unpacking rpm package hadoop_3_0_0_0_1634-3.1.0.3.0.0.0-1634.x86_64' For Hive Client Install RuntimeError: Failed to execute command '/usr/bin/yum -y install hive_3_0_0_0_1634-hcatalog', exited with code '1', message: 'Error unpacking rpm package hadoop_3_0_0_0_1634-3.1.0.3.0.0.0-1634.x86_64
... View more
- « Previous
- Next »