Member since
12-21-2015
32
Posts
14
Kudos Received
0
Solutions
02-19-2016
03:28 PM
1 Kudo
Given the fact that Spark is in memory processing and Hadoop is more to do with Disk based processing (higher disk I/O), I was wondering for sizing the containers and RAM needs, do we need more RAM for running same use case with Spark when compared to Hadoop Map-Reduce?
... View more
Labels:
- Labels:
-
Apache Spark
02-12-2016
11:31 AM
1 Kudo
Hi folks, One of the requirements is to redirect the all kind of logs/ ranger logs/access logs and also other component's logs to external file system. Probably on NFS and not on HDFS or in the DB. Does HDP provide out of the box solution for this? One workaround I can think of using Flume but wanted to know other approaches? Regards, DP
... View more
Labels:
02-11-2016
12:03 PM
1 Kudo
Thanks @Artem Ervits
... View more
02-11-2016
12:03 PM
1 Kudo
Thanks @Neeraj Sabharwal
... View more
02-11-2016
11:30 AM
2 Kudos
Hi folks, I am trying to run the spark PI example on the Hortonworks cluster. I could run this successfully in the local mode and in the yarn client mode . When I try to run it using YARN cluster mode I do not see any out put or error. I am not sure if it is running or is there a bug. Can you please help me understand this behavior? Here are the commands I am trying to use. spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster ../lib/spark-examples*.jar 10 Also tried with spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar 10 (One more question are above both correct ?) Both the times I see the following console log but not the PI value. ( When I run in local mode or yarn-client mode I see the value of PI printed on the console) 16/02/11 11:06:05 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/02/11 11:06:05 INFO SecurityManager: Changing view acls to: username
16/02/11 11:06:05 INFO SecurityManager: Changing modify acls to: username
16/02/11 11:06:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/02/11 11:06:06 INFO Client: Submitting application 432 to ResourceManager
16/02/11 11:06:06 INFO YarnClientImpl: Submitted application application_1454617624671_0432
16/02/11 11:06:07 INFO Client: Application report for application_1454617624671_0432 (state: ACCEPTED)
16/02/11 11:06:07 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1455188766121
final status: UNDEFINED
tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
user: username
..............
16/02/11 11:06:23 INFO Client: Application report for application_1454617624671_0432 (state: FINISHED)
16/02/11 11:06:23 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: APP_Master_host_IP
ApplicationMaster RPC port: 0
queue: default
start time: 1455188766121
final status: SUCCEEDED
tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
user: username
16/02/11 11:06:23 INFO ShutdownHookManager: Shutdown hook called
16/02/11 11:06:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-54dc94ab-cf66-4d17-9940-1c31ba7e9850
[username@remoteIP bin]$
... View more
Labels:
- Labels:
-
Apache Spark
12-23-2015
04:59 PM
@Neeraj Sabharwal I tried configuring Hive/PIG views as per the documentation. If you confirm that in the Keberized cluster and NN Highly available PIG/HIVe views not supported then I will close the thread 🙂 Thank you very much.
... View more
12-23-2015
03:39 PM
@Predrag Minovic The hive.server2.transport.mode is set to http. File explorer is working. We are on Ambari version: 2.1.2 Thank you. Is there any thing possibly missing?
... View more
12-23-2015
02:51 PM
Thanks
@Predrag Minovic
Indeed this is quite detailed. I've a user ambariserver and principal ambariserver/ambari_host_name@KDCRealm.com
I also verified following two properties are added in the custom core site.
hadoop.proxyuser.ambariserver.groups=*
hadoop.proxyuser.ambariserver.hosts=*
PIG/Hive view, I've added following two properties in the webhcat-site.xml
webhcat.proxyuser.ambariserver.groups=*
webhcat.proxyuser.ambariserver.hosts=*
Accessing the Hive View we see error.
H020 Could not establish connecton to HiveServer2_HOST:10000:org.apache.thrift.transport.TTransportException
... View more
12-23-2015
12:02 PM
Well but network is stable here. And no jobs are running on the cluster!
... View more