Getting below error   The streaming job got killed again with too many open files exception.  2017-03-07 11:14:12 WARN  TaskSetManager:70 - Lost task 7.0 in stage 31767.0 (TID 99289, node-4.perf.com): java.io.FileNotFoundException: /data/yarn/nm/usercache/tdubidata/appcache/application_1488862815845_0004/blockmgr-8c444a03-fe57-4eea-b4ef-1fd8e1ba6e7f/2d/shuffle_3311_7_0.index.670e98c9-aa58-4677-ab8f-9d17035f2ece (Too many open files in system)     at java.io.FileOutputStream.open(Native Method)     at java.io.FileOutputStream.<init>(FileOutputStream.java:221)     at java.io.FileOutputStream.<init>(FileOutputStream.java:171)     at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:141)     at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:128)     at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)     at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)     at org.apache.spark.scheduler.Task.run(Task.scala:89)     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)     at java.lang.Thread.run(Thread.java:745)   Ulimit set to all users   [root@node-2 ]# ulimit -aH core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257534 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 100000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 257534 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited   Also changes the Maximum Process File Descriptors of all parcels to 100000   Please help to find apt solution.  ... View more

Hi   We are using 5.8.0 version cluster. 2 days ago, we have encrypted our filesystem(GDE). Our cluster is hosted at IBM softlayer.  Our cloudera manager DB(sc,nav,navfs,..etc) is not embedded one, we are using mysql server separately .    Even that server has been encrypted. We have restarted the server after encryption. It was working fine for 4 hours. After the cloudera manager crashes, we are not able to get the GUI of the cloudera manager. We found that mysql service has been down. We make the services up with in an hour.   After restarting the mysql service, we are getting the GUI page as fresh installation, but we can see the currently managed hosts are still there.   But we are not able to pass any command related to hadoop, hbase etc.   Getting unrecognizable service error   Anyone can comment, why this is happening.? Can we retrieve whole data in hdfs? Please provide a solution ... View more

I am using a six node cluster(5.8). Recently, we have done GDE(file system encryption). Hosted on IBM softlayer environment.   Encryption process done by different team. They asked a question   We set the Hadoop server MTU to 1400 from the previous 1576 in order to get the packets to move through the encrypted tunnel. Is this OK for Cloudera to work?   Some please respond to my query ASAP.   ... View more