Member since
11-10-2016
26
Posts
3
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
693 | 12-21-2016 08:26 AM | |
658 | 12-14-2016 04:45 PM | |
633 | 12-14-2016 01:24 PM | |
2929 | 11-21-2016 12:04 PM |
02-21-2017
11:56 PM
@rich if you could help me out here, it will be really helpful 🙂
... View more
02-21-2017
11:49 PM
Hi, I am preparing for HDPCD exam and i wanted to know what kind of datasets are given during the exam like text file,parquet,json,sequencefile etc. Currently I am practicing with datasets like text files and ORCFILEs, i need to know what more are expected. Thanks, Zeeshan
... View more
12-21-2016
08:26 AM
Raised Customer message towards SAP and the resolution was: "Known issue for Spark Controller 1.6.2, so please updagrade to Spark Controller 2.0". After upgrading to Spark Controller 2.0 the installation was successful. Hence closing this thread.
... View more
12-21-2016
05:45 AM
HDP 2.3.0.0
... View more
12-21-2016
05:42 AM
Yes this is a SAP specific install for data management. Ambari : 2.4.2.0
Spark : 1.5.2.2.3
Spark Controller : 1.6.1
... View more
12-15-2016
01:55 PM
1 Kudo
When we are trying to install Spark Controller via Ambari, it is giving error. All pre-installation activities are cool. below is the error we are getting: stderr: /var/lib/ambari-agent/data/errors-403.txt File "/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/SparkController/package/scripts/controller_conf.py", line 10, in controller_conf
recursive = True
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 147, in __init__
raise Fail("%s received unsupported argument %s" % (self, key))
resource_management.core.exceptions.Fail: Directory['/usr/sap/spark/controller/conf'] received unsupported argument recursive stdout: /var/lib/ambari-agent/data/output-403.txt 2016-12-15 08:44:36,441 - Skipping installation of existing package curl
2016-12-15 08:44:36,441 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2016-12-15 08:44:36,496 - Skipping installation of existing package hdp-select
Start installing
2016-12-15 08:44:36,668 - Execute['cp -r /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/SparkController/package/files/sap/spark /usr/sap'] {}
2016-12-15 08:44:36,685 - Execute['chown hanaes:sapsys /var/log/hanaes'] {}
Configuring...
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Spark
12-14-2016
04:45 PM
creating the soft link like ln -s /usr/hdp/2.3.0.0-2557/spark spark-thriftserver at the dir /usr/hdp/current did the trick
... View more
12-14-2016
04:35 PM
We have installed HDP 2.3.0.0(2.3.0.0-2557) on our cluster managed by Ambari 2.4.2.0. When we are starting the spark thrift server, it errors out. Upon checking we found that the package directory "/usr/hdp/current/spark-thriftserver/" doesn't exist. this directory should had been created while installation but it is not. Any solutions how to deal with this.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
12-14-2016
01:24 PM
1 Kudo
Its available with HDP 2.3.4.7 Successful installation done.
... View more
12-14-2016
12:54 PM
HDP-2.3.6.0 (HDP-2.3.6.0-3796) gives Spark 1.5.1 but we need Spark 1.5.2
... View more
12-14-2016
11:53 AM
1 Kudo
Hi Guyzz, I have a business requirement to install Spark1.5.2 in our cluster as this version is compatible with Hana Spark Controller 1.6.1. So I want to know exact Ambari repository and HDP version which I should be using during the installation.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
11-21-2016
12:04 PM
Hi Changing the value for hive.security.authorization.manager = org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider worked. Changed the hive-site.xml at spark controller. hive-site conf at Hive Client is having proper authorizations. Issue Resolved.
... View more
11-21-2016
11:33 AM
Hi we have a Hana Hadoop integrated implementation using Spark Controller. When we are creating data preview at Hana for the virtual table which is created from a Hive table ,we are getting class not found error at spark controller log. While creating the data preview HANA send a select query request to hive via spark controller. Other functionalities like table details like name and schema are all coming in Hana. Only issue we are facing is it cannot fetch the data. Hana SPS10 Spark 1.4.1.2.3 Spark Controller 1.5 Patch 0 Hive 1.2.1.2.3 Error log : 2016-11-21 06:09:23,632 [DEBUG] <?x-ml version="1.0"?><hana:queryplan xmlns:hana="
SELECT'>http://www.sap.com/hana"><select><sql>SELECT "employee"."name", "employee"."dept", "employee"."level" FROM "big_poc"."employee" "employee" LIMIT 200 </sql><typeInfo><column><name>COL0</name><type>29</type></column><column><name>COL1</name><type>29</type></column><column><name>COL2</name><type>3</type></column></typeInfo><parameters/><relocators/></select></hana:queryplan> 2016-11-21 06:09:25,078 [ERROR] java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:399)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthenticator(SessionState.java:867)
at org.apache.hadoop.hive.ql.session.SessionState.getUserFromAuthenticator(SessionState.java:589)
at org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(Table.java:174)
at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:116)
at org.apache.spark.sql.hive.client.ClientWrapper.org$apache$spark$sql$hive$client$ClientWrapper$$toQlTable(ClientWrapper.scala:237)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getAllPartitions$1.apply(ClientWrapper.scala:297)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$getAllPartitions$1.apply(ClientWrapper.scala:296)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:155)
at org.apache.spark.sql.hive.client.ClientWrapper.getAllPartitions(ClientWrapper.scala:296)
at org.apache.spark.sql.hive.client.HiveTable.getAllPartitions(ClientInterface.scala:74)
at org.apache.spark.sql.hive.MetastoreRelation.<init>(HiveMetastoreCatalog.scala:645)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:248)
at org.apache.spark.sql.hive.hana.HanaESSQLContext$$anon$1.org$apache$spark$sql$hive$hana$HanaESCatalogNew$$super$lookupRelation(HanaESSQLContext.scala:27)
at org.apache.spark.sql.hive.hana.HanaESCatalogNew$class.lookupRelation(HanaESCatalogNew.scala:27)
at org.apache.spark.sql.hive.hana.HanaESSQLContext$$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HanaESSQLContext.scala:27)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:165)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:165)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:165)
at org.apache.spark.sql.hive.hana.HanaESSQLContext$$anon$1.lookupRelation(HanaESSQLContext.scala:27)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:222)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:233)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:229)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:222)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:222)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:221)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:242)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:272)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:227)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:242)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:272)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:227)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:212)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:229)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:219)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:61)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:59)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:59)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:51)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:51)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:933)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:933)
at org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:931)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:131)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:755)
at org.apache.spark.sql.hive.hana.HanaSQLContext$class.executeHANAQueryTask(HanaSQLContext.scala:115)
at org.apache.spark.sql.hive.hana.HanaESSQLContext.executeHANAQueryTask(HanaESSQLContext.scala:23)
at com.sap.hana.spark.network.CommandHandler$$anonfun$receive$2$$anonfun$applyOrElse$8.apply(CommandRouter.scala:278)
at com.sap.hana.spark.network.CommandHandler$$anonfun$receive$2$$anonfun$applyOrElse$8.apply(CommandRouter.scala:275)
at scala.collection.immutable.List.foreach(List.scala:318)
at com.sap.hana.spark.network.CommandHandler$$anonfun$receive$2.applyOrElse(CommandRouter.scala:275)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at com.sap.hana.spark.network.CommandHandler.aroundReceive(CommandRouter.scala:162)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:376)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:381)
... 91 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:154)
at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:142)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:366)
... 92 more
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
11-15-2016
04:48 PM
@Ancil McBarnett will your above solution delete the existing HDP installation also ?
... View more
11-15-2016
04:41 PM
Hi we have a business requirement to install Spark 1.4.1 on our cluster, so I want to know the exact ambari repository and HDP version which I should be installing. Previously we installed HDP-2.3.6.0-3796 and it came with SPARK 1.3.1.2 which was not desired and upgrading SPARK was a tedious task which resulted in Cluster Corruption. So I need the exact HDP version which will have 1.4.1.
... View more
Labels:
11-15-2016
04:26 PM
Hi @Vedant Jain , I performed the above steps and successfully removed spark 1.3.1 and installed spark 1.4.1, but in Ambari I don't see the newly Added Spark service. The old spark in not visible anymore. Due to project needs I need to see the current spark service in my Ambari. Is there any way I can achieve this ? Thanks in Advance.
... View more
11-15-2016
01:47 PM
Versions Installed : HDP-2.3.6.0-3796 Spark 1.3.1.2.3 HDFS 2.7.1.2.3 Ambari 2.1.0 Currently we are integrating SAP HANA with Hadoop and due to versions compatibility we have to move to only Spark 1.4.1. At this point of time we have Spark 1.3.1.2.3 so we need to upgrade to Spark 1.4.1
... View more
Labels:
- Labels:
-
Apache Spark