Member since
01-28-2017
5
Posts
0
Kudos Received
0
Solutions
03-23-2017
03:20 AM
These are the parcels, we are using. Login to a server that has a running executor and find the process id of that executor I am not sure about the server that has a running executor, and how i can find the executor [root@node-1 ec2-user]# ulimit -aH core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257534 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 500000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 257534 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited This is 62 GB memory and 8 core machine
... View more
03-22-2017
08:51 AM
Thanks for the response I have increased the ulimit to all user to 500000. Still after certain time, the streaming job went down. How can we ensure , that we are closing all connections and files that we aer handling ? Yes we are having long running applications in spark streaming jobs. Please help.
... View more
03-09-2017
07:16 AM
Getting below error The streaming job got killed again with too many open files exception. 2017-03-07 11:14:12 WARN TaskSetManager:70 - Lost task 7.0 in stage 31767.0 (TID 99289, node-4.perf.com): java.io.FileNotFoundException: /data/yarn/nm/usercache/tdubidata/appcache/application_1488862815845_0004/blockmgr-8c444a03-fe57-4eea-b4ef-1fd8e1ba6e7f/2d/shuffle_3311_7_0.index.670e98c9-aa58-4677-ab8f-9d17035f2ece (Too many open files in system) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:171) at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:141) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:128) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Ulimit set to all users [root@node-2 ]# ulimit -aH core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257534 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 100000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 257534 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Also changes the Maximum Process File Descriptors of all parcels to 100000 Please help to find apt solution.
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
01-31-2017
10:13 PM
Hi We are using 5.8.0 version cluster. 2 days ago, we have encrypted our filesystem(GDE). Our cluster is hosted at IBM softlayer. Our cloudera manager DB(sc,nav,navfs,..etc) is not embedded one, we are using mysql server separately . Even that server has been encrypted. We have restarted the server after encryption. It was working fine for 4 hours. After the cloudera manager crashes, we are not able to get the GUI of the cloudera manager. We found that mysql service has been down. We make the services up with in an hour. After restarting the mysql service, we are getting the GUI page as fresh installation, but we can see the currently managed hosts are still there. But we are not able to pass any command related to hadoop, hbase etc. Getting unrecognizable service error Anyone can comment, why this is happening.? Can we retrieve whole data in hdfs? Please provide a solution
... View more
Labels:
- Labels:
-
Manual Installation
01-28-2017
08:54 PM
I am using a six node cluster(5.8). Recently, we have done GDE(file system encryption). Hosted on IBM softlayer environment. Encryption process done by different team. They asked a question We set the Hadoop server MTU to 1400 from the previous 1576 in order to get the packets to move through the encrypted tunnel. Is this OK for Cloudera to work? Some please respond to my query ASAP.
... View more
Labels:
- Labels:
-
Manual Installation