I am trying to use YARN through Knox. But HA is enabled for YARN. How can i define HA in topology? Regards, Shashi ... View more

I am running python code with custom libraries through Livy interpreter. But getting below error while importing the library. I was able to resolve the issue by setting the session level PYTHON_EGG_CACHE parameter in the code. But i wanted to set it global for all users. Any idea how to do it Regards, Shashi ... View more

Team, I want to restrict the users to access the data from Hive tables on Spark. My cluster kerberized and have Ranger setup. Can some help me out on this. -Shashi ... View more

can some one help in providing the solution to use S3 bucket as target dir in SQOOP. I am getting below error. 17/01/24 15:25:53 ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://av-horton-poc/test, expected: hdfs://ip-10-228-210-170:8020 Regards, Shashi ... View more

Hello All, I am executing a job on 2.2 billion rows table which contains groupByKey() on RDD. spark-submit --packages com.databricks:spark-csv_2.11:1.4.0 --jars /appdata/hwx/hdp/2.4.2.0-258/hadoop/lib/aws-java-sdk-1.7.4.jar,/appdata/hwx/hdp/2.4.2.0-258/hadoop/hadoop-aws-2.7.1.2.4.2.0-258.jar,/appdata/hwx/hdp/2.4.2.0-258/hadoop/hadoop-aws.jar --driver-memory 20g --master yarn --deploy-mode client --executor-memory 16g --conf spark.driver.maxResultSize=10g --num-executors 18 /appdata/hwx/scripts/python_jobs/Runs/isaxtest_Spark_code.py Below is the error: [Stage 2:> (0 + 160) / 200]16/07/31 13:08:19 ERROR YarnScheduler: Lost executor 15 on ip-10-228-211-233: Container killed by YARN for exceeding memory limits. 18.3 GB of 18 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. [Stage 2:> (0 + 150) / 200]16/07/31 13:08:21 ERROR YarnScheduler: Lost executor 9 on ip-10-228-211-244: Container k illed by YARN for exceeding memory limits. 18.4 GB of 18 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. 16/07/31 13:08:21 ERROR TransportRequestHandler: Error sending result RpcResponse{requestId=8646295513446257068, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to ip-10-228-211-76/10.228.211.76:37971; closing connection java.nio.channels.ClosedChannelException [Stage 2:> (0 + 140) / 200]16/07/31 13:08:24 ERROR YarnScheduler: Lost executor 2 on ip-10-228-211-76: Container ki lled by YARN for exceeding memory limits. 18.3 GB of 18 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. [Stage 2:> (0 + 130) / 200]16/07/31 13:08:27 ERROR YarnScheduler: Lost executor 3 on ip-10-228-211-140: Container k illed by YARN for exceeding memory limits. 18.4 GB of 18 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. ... View more

Hello Team, We are trying to create a function with joins and it will return few records. Can you please suggest how to return a record type in a function in hive 2.0. We are able to compile the function but while executing we are getting below error. Starting query Exception in thread "main" java.util.EmptyStackException at java.util.Stack.peek(Stack.java:102) at java.util.Stack.pop(Stack.java:84) at org.apache.hive.hplsql.Stmt.print(Stmt.java:1001) at org.apache.hive.hplsql.Exec.visitPrint_stmt(Exec.java:1715) at org.apache.hive.hplsql.Exec.visitPrint_stmt(Exec.java:51) at org.apache.hive.hplsql.HplsqlParser$Print_stmtContext.accept(HplsqlParser.java:10802) at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70) at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:987) Regards, Shashi ... View more