About bikas

bikas · ‎01-25-2018

If you have HDP 2.6.3 then you should be able to find spark 2.2 version of spark-llap available under /usr/hdp/current/ Perhaps you are using the older versions of shc using --packages and thats not compatible with spark 2.2.

bikas · ‎04-24-2017

/etc/hive/conf/hive-site.xml is the config for Hive service itself and is managed via Ambari through the Hive service config page. /usr/hdp/current/spark-client/conf/hive-site.xml actually points to /etc/spark/conf/hive-site.xml . This is the minimal hive config that Spark needs to access Hive. This is managed via Ambari through the Spark service config page. Ambari correctly configures this hive site for Kerberos. Depending upon your version of HDP you may not have the correct support in Ambari for configuring Livy.

bikas · ‎03-17-2017

When connecting via beeline did that use HiveServer2 or SparkThriftServer?

bikas · ‎03-12-2017

Is this needed even after the HDP 2.5 native Oozie Spark action?

bikas · ‎03-09-2017

Apache Spark has traditionally worked sub-optimally with ORC because ORC used to be inside Apache Hive and Apache Spark depends on a very old release of Hive 1.2.1 from mid-2015. We are working on figuring out how to best update Apache Spark's version of ORC either by upgrading Apache Spark's dependency to latest Apache Hive or depending for ORC from the new Apache ORC project.

bikas · ‎03-08-2017

For JDBC there is a built-in jar for JDBC support. No need for Simba.

bikas · ‎03-08-2017

Probably missing having hbase-site or phoenix conf in the classpath. So it cannot find the ZK info for hbase/phoenix.

bikas · ‎03-03-2017

In the Spark AM, the keytab is used to logon as the principal and use kerberos credentials to acquire new instances of all default tokens (HDFS/Hive/HBase) in HDP Spark 1.6. In Apache Spark Hbase is not acquired. There is a backport in HDP for that fix from Spark 2.0.

bikas · ‎03-01-2017

Fixed the link. It has the code.

bikas · ‎02-23-2017

What exception/error happens in code 2? Just curious. foreachRDD is the prescriptive method to write to external systems. So you should be using foreachRDD. The outer loop executes on the driver and inner loop on the executors. Executors run on remote machines in a cluster. However in the code above its not clear how dynamoConnection is available to executors since such network connections are usually not serializable. Or is the following line inadvertently missing from snippet 1. val dynamoConnection = setupDynamoClientConnection() If yes, then the slowness could stem from repeatedly creating a dynamoClientConnection for each record. The recommended pattern is to use foreachPartition() to create the connection once per partition and then rdd.foreach() to write the records using that connection. For more info please search for foreachPartition see http://spark.apache.org/docs/latest/streaming-programming-guide.html

bikas · ‎02-21-2017

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

bikas · ‎02-16-2017

Ambari should be giving the option of which HDP 2.5.x version to install. Choosing higher versions would give higher Apache versions. E.g. HDP 2.5.3 will give Apache Spark 2.0.1. The next HDP 2.5.4+ release will give 2.0.2 HDP 2.6 (not released yet) will have Apache Spark 2.1. You can try a tech preview of that on HDC

bikas · ‎02-15-2017

A full stack trace would help understand which interaction is resulting in this. If IDE based code is being used then you could try to not use the spark-assembly jar that is present on HDFS and instead use the local spark-assembly jar from the Spark build being compiled against. This could be done by overriding spark.yarn.jar config. Could be the that compile dependency of Spark in your IDE is different from the runtime dependency on HDFS. Another thing could be scala version mismatch.

bikas · ‎02-07-2017

Yes. For Spark 1.6 it is GA in HDP 2.5.3. The documentation is available from the Github site for a given SHC release tag. That is the source of truth.

bikas · ‎02-01-2017

--files will add it to the working directory of the YARN app master and container and this would mean that those files (and not jars) would be in the classpath of the app master and container. But in client mode jobs the main driver code is running in the client machine. So these --files are not available on the driver. SPARK_CLASSPATH adds these files to the driver classpath. Its an env var. So one could say the following. Note it will warn saying its deprecated and cannot be used concurrently with --driver-class-path option. More information can be found here. https://github.com/hortonworks-spark/shc export SPARK_CLASSPATH=/a/b/c/hbase-site.xml;/d/e/f/hive-site.xml

bikas · ‎01-30-2017

Unfortunately, that kind of functionality does not exist for Spark Streaming. Spark Streaming runs as a standard YARN job and YARN commands could be used to start, stop (kill) and re-submit a job. A properly written Spark streaming job should be able to support at-least once or exactly-once semantics through this lifecycle. But other than that there is no UI or other automation support for it. Zeppelin is designed for interactive analysis and running Spark streaming via Zeppelin is not recommended (other than demos for presentations).

bikas · ‎01-26-2017

This is not available in any distribution since its a package and can be used independently. The latest 1.6 release is https://github.com/hortonworks-spark/shc/tree/v1.0.1-1.6 You can build that with the hbase version that matches your env.

bikas · ‎01-23-2017

The flow seems right. Thats a good use case for livy. Assuming it goes YourApp->Livy->Spark and back. You will need to look at Livy client logs or livy logs for session id 339. Seems like the client is asking for a session (livy spark job) that does not exist anymore. Could have been not started and abandoned or lost.

bikas · ‎01-23-2017

SHC does not have a notion of listing tables in HBase. It works on the table catalog provided to the data source in the program. Hive will also not list HBase tables because they are not present in the metastore. There is some rudimentary way to add Hbase external tables in Hive but I dont think that really used. I could be wrong. To list Hbase tables, currently the only reliable way would be to use HBase API's inside the spark program to list tables.

bikas · ‎01-23-2017

Hive and HiveContext in Spark can only show the tables that are registered in the Hive Metastore and Hbase tables are usually not there because the schema of most Hbase tables are not easily defined in the metastore. To read HBase tables from Spark using DataFrame API please consider Spark HBase Connector

bikas · ‎01-23-2017

In HDP 2.5 Zeppelin JDBC interpreter can run 1 query in a paragraph. There is a limit of 10 queries that can be run simultaneously overall.

bikas · ‎01-17-2017

First of all, which Spark version are you using. Apache Spark 2.0 has support for automatically acquiring HBase security tokens correctly for that job and all its executors. Apache Spark 1.6 does not have that feature but in HDP Spark 1.6 we have backported that feature and it can acquire the HBase tokens for the jobs. The tokens are automatically acquired if 1) security is enabled and 2) hbase-site.xml is present on the client classpath 3) that hbase-site.xml has kerberos security configured. Then hbase tokens for the hbase master specified in that hbase-site.xml are acquired and used in the job. In order to obtain the tokens spark client needs to use hbase code and so specific hbase jars need to be present in the client classpath. This is documented in here on the SHC github page. Search for "secure" on that page. To access hbase inside the spark jobs the job obviously needs hbase jars to be present for the driver and/or executors. That would be part of your existing job submission for non-secure clusters, which I assume already works. If this job is going to be long running and run beyond the token expire time (typically 7 days) then you need to submit the Spark job with the --keytab and --principal option such that Spark can use that keytab to re-acquire tokens before the current ones expire.

bikas · ‎01-17-2017

If this does not work for you please open the feature request by creating an issue on the github project for SHC. /cc @wyang

bikas · ‎01-17-2017

Ideally, just before that OWN failure log, there should be an exception or error message about some task for vertex with id 1484566407737_0004_1. That could give more info. Even if more info is not there, you will be able to find the task attempt that actually failed. That task attempt can show you which machine and YARN container is ran on. Sometimes the logs dont have the error because it logged into stderr. In that case, the stderr from the containers YARN logs may show the error.

bikas · ‎01-17-2017

Is "/user/hadoopuser/streaming_2.10-1.0.0-SNAPSHOT.jar" your application code jar? If yes then you may have to reupload it because its blocks are now missing from HDFS and so Spark cannot download it to run your application code.

bikas · ‎01-13-2017

Is Spark Thrift Server installed via Ambari? Or done manually. It seems like the hive principal being used does not match the keytab being provided for Kerberos.

bikas · ‎01-12-2017

First authentication via AD/LDAP needs to be enabled in Zeppelin. You can search HCC for articles showing how to set that up. Next, for Spark jobs you have to enable and configure Livy component in the Spark service via Ambari. Then configure and enable the Livy interpreter in Zeppelin. After that authenticated users would runs their spark code in Zeppelin via the %livy.sql or %livy.spark interpreters. Livy interpreters support impersonation for spark jobs for a multi-user Zeppelin environment via the Livy service. /cc @vshukla in case we already have a definitive HCC article for this scenario.

bikas · ‎01-11-2017

+1. Thats what I mentioned in my last comment below. Copying here so everyone get the context quickly. Ranger KMS could be the issue because it causes problems for getting HDFS delegation token. If Z or L user needs to get HDFS delegation token then they also need to be super users for Ranger. You are better off trying with non-Ranger cluster or adding them to Ranger super users which is different from core site super users.

bikas · ‎01-11-2017

If you intend to adopt 2.0 then the recommendation is to use the new 2.0 APIs since the 1.0 API may either be deprecated or not have new features supported via them.

bikas · ‎01-11-2017

If you intend to adopt 2.0 then the recommendation is to use the new 2.0 APIs since the 1.0 API may either be deprecated or not have new features supported via them.

Online	Offline
Last Visited	‎09-23-2018 04:18 AM

Date Registered	‎10-09-2015 06:38 PM
Last Visited	‎09-23-2018 04:18 AM
Total Messages Posted	76
Total Kudos Received	33

Re: Dataframe Insert into ORC table is slow compar...

Re: Spark map vs foreachRdd

Re: How to configure spark-log4j-properties in Amb...

Re: Spark 2 Technical preview with patches

Re: Spark Hbase connector latest version for Spark...

Re: Read Hbase tables using Spark 2

Re: Why do I have two hive-site.xml config files o...

Re: Config settings set in hive (via ambari) are a...

Re: Oozie Spark Action to access Hive using HiveCo...

Re: Dataframe Insert into ORC table is slow compar...

Re: Connecting BI tools to Spark

Re: Using Phoenix in Spark on Zeppelin - Failing o...

Re: What is correct strategy for Spark Streaming K...

Re: Spark map vs foreachRdd

Re: Spark map vs foreachRdd

Re: How to configure spark-log4j-properties in Amb...

Re: Spark 2 Technical preview with patches

Re: Local Spark Development against a remote clust...

Re: HDP Spark HBase Connector

Re: Spark HBase Connector (SHC) job fails to conne...

Re: Managing Spark Streaming from Ambari?

Re: Spark Hbase connector latest version for Spark...

Re: Livy rest api with web app

Re: List hbase tables Spark sql

Re: List hbase tables Spark sql

Re: Does zeppelin support multiple hive queries in...

Re: Spark can't connect to HBase using Kerberos i...

Re: HDP Spark Hbase Connector Cell Versions?

Re: Vertex did not succeed due to OWN_TASK_FAILURE...

Re: Spark Streaming: How to restart spark streamin...

Re: How to run Spark Sql on Spark Thrift Server

Re: Controlling Access to HDFS data through zeppel...

Re: Zeppelin livy interpreter not working with Ker...

Re: spark backward compatibility 1.6 vs 2.0

Re: spark backward compatibility 1.6 vs 2.0