About Chittu

Chittu · ‎12-04-2019

Do I need to put the namenode in safe mode to execute this command? or I can execute this on live cluster? hadoop fs –setrep –w 3 -R /

Chittu · ‎12-02-2019

Hi All, I need to write Spark job output file to NFS mount point from spark2 shell. can you please let me know if there is any way to do it by defining absolute path in Spark2 shell. Thanks, CS

Chittu · ‎08-06-2019

I built Spark2 with CDH 5.16 and able to submit scala jobs with no issues. Now I am able to launch pyspark2 and when I am trying to run simple job, its throwing the below error? can you please suggest on this? what is the alternate to submit python jobs on spark jobs apart from Jupyter notebook? Pls advise [I 23:08:33.864 NotebookApp] Adapting to protocol v5.1 for kernel f8d7200b-6718-49f6-86e9-c051fb6d84a6 [Stage 0:> (0 + 0) / 2]Exception in thread "dispatcher-event-loop-0" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41) at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1853) Thanks CS 19/08/06 23:10:41 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources [Stage 0:> (0 + 0) / 2]19/08/06 23:10:47 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 2 for reason Container marked as failed: container_1565048178604_0033_01_000003 on host: ukvmlx-rdk-22.rms.com. Exit status: 1. Diagnostics: Exception from container-launch.

Chittu · ‎06-17-2019

I am unable to launch Spark with YARN Resource manager is not getting started. seeing the below error. Can you please let me know what I am missing. Spark error: client.RMProxy: Connecting to ResourceManager at <IP>8032 19/06/17 22:48:57 INFO ipc.Client: Retrying connect to server: <IP>:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 19/06/17 22:48:58 INFO ipc.Client: Ret Resource Manager Error: Error starting ResourceManagerjava.lang.NullPointerException at org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1039) at org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1036) at org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1101) at org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1093) at org.apache.hadoop.ha.ActiveStandbyElector.createWithRetries(ActiveStandbyElector.java:1036) at org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:347) at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.serviceInit(ActiveStandbyElectorBasedElectorService.java:110) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:333) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:143

Online	Offline
Last Visited	‎01-15-2020 04:37 PM

Date Registered	‎05-06-2019 04:26 PM
Last Visited	‎01-15-2020 04:37 PM
Total Messages Posted	7

Re: How can one increase replication factor to a d...

How do I write Spark job output to NFS Share inste...

Spark job getting failed with Jupyter notebook

YARN Resource manager is not getting started