Member since
03-06-2015
61
Posts
5
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1955 | 04-30-2015 09:04 AM | |
37672 | 03-27-2015 10:31 AM | |
1962 | 03-24-2015 12:27 PM | |
987 | 03-06-2015 01:37 PM |
08-25-2017
08:18 AM
1 Kudo
There is no ntpupdate command. You can update ntp by typing # ntpdate -u {Main node sync'ed to NTP server IP address or Domain name here} Example # ntpdate -u domains.ntp.com OR # ntpdate -u 192.168.0.2 You should only have 1 node announcing NTP and other nodes getting time from main node.
... View more
08-29-2016
08:08 AM
I use puppet and foreman to configure each system the same for all ntp clients to a central ntp server. Great information here http://www.linuxsecrets.com/blog/4tutorial-and-install-howto/2014/06/26/906-how-to-install-and-configure-linux-ntp-server-and-client
... View more
04-30-2015
09:04 AM
I just did a roll back which resolved the issue. We have been testing apache hadoop as well so that's why I was delayed to answer. Thanks
... View more
04-13-2015
04:31 PM
What can I check to resolve this issue? Had someone changing stuff and have no idea what they did to kill my hadoop cloudera. 15/04/13 16:28:18 INFO mapred.FileInputFormat: Total input paths to process : 10 15/04/13 16:28:18 INFO mapred.JobClient: Running job: job_201504131620_0001 15/04/13 16:28:19 INFO mapred.JobClient: map 0% reduce 0% 15/04/13 16:28:22 INFO mapred.JobClient: Task Id : attempt_201504131620_0001_m_000011_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250) Caused by: java.io.IOException: Task process exit with nonzero status of 2. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
... View more
Labels:
- Labels:
-
Apache Hadoop
04-02-2015
10:47 AM
Hi eafonsof Updating your kernel version will resolve the issue most likely. I updated mine to kernel 3.18.x and resolved the issue.
... View more
04-02-2015
08:38 AM
I'll work on the documentation this weekend Something you can understand without having to click another link,link, link to find information :-).
... View more
03-30-2015
04:09 PM
I can document the process for CentOS if you like. It's pretty much the same with RedHat as well. Only difference is the repo and versions. Let me know if you want it documented then I'll post it somewhere for you.
... View more
03-30-2015
10:56 AM
P.S. Don't forget to create or move your current .config from previous kernel or create new .config. # cd /usr/src/kernels/linux-3.18.8 # cp ../linux-old-kernel-version/.config . # make oldconfig -j32;make modules -j32;make modules_install -j32;make install # Answer all question using just the default. Just enter through the kernel questions.
... View more
03-30-2015
10:51 AM
Just change the versioning number from kernel or update to RedHat 6.6. 6.6 works perfectly. Down grading isn't a good option in my opinion. It's simply needs a version check which you can do by just upgrading the kernel version. Easiest way instead of doing a complete reinstall or downgrade which isn't a good option. Download kernel from kernel.org # cd /usr/src/kernels # wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.8.tar.gz # tar zxvf linux-3.18.8.tar.gz # cd linux-3.18.8 # make -j32;make modules -j32;make modules_install -j32;make install # edit /boot/grub/grub.cfg Change to 0 which is the current kernel version and that's it, reboot and you're set.
... View more
03-27-2015
10:31 AM
Found why. Typo my mistake not removing the 1TB file size that was being generated by teragen 🙂 Command worked: sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hadoop-0.20-mapreduce/hadoop-examples-2.5.0-mr1-cdh5.3.1.jar terasort /home/ssd/hdfs-input /home/ssd/hdfs-output Works perfectly now.
... View more
03-27-2015
10:14 AM
Not sure what I'm doing wrong here but I keep getting the same error when I run terasort. Teragen works perfectly but terasort fails. Input path does not exist: hdfs://node0:8020/user/hdfs/10000000000 Command line used: sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hadoop-0.20-mapreduce/hadoop-examples-2.5.0-mr1-cdh5.3.1.jar terasort 10000000000 /home/ssd/hdfs-input /home/ssd/hdfs-output.
... View more
Labels:
03-24-2015
12:27 PM
Ok, the setup is simple you just create datanodes with 1 TT on namenode which took the network to 3500MB to other nodes which worked
... View more
03-17-2015
02:11 PM
I found a way to increase network performance but only for write. When I run a read dfsio it only seems to be sending to the local drive from 1 system and not reading from multiple systems. I need the system to read through the network and not locally can any body help on how to force network reads using dfsio?
... View more
03-17-2015
08:24 AM
03-16-2015
03:18 PM
I would also like to cut down on local hhd writes so I can evenly distribute data to all machines so I get more network traffic as well. @nauseous wrote: I'm trying to get maximum throughput with Cloudera on RedHat 6.6 on 6 - Dell R730's with kernel 3.18.1, and using 2 - 850MB, 3G ssd transfer per second hhd with modified drivers which have been tested. Currently I've tried decommissioning "mapReduce tasktracker" on all nodes except 1 single node as suggested but didn't really make any differences in nic speed. I want to max out the connection speed on all nodes if possible. I've tried : sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 100000 -fileSize 50 and sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 500 -fileSize 10GB without good results. I've already tested throughput with netperf but can't seem to get cloudera to perform network tests to maximum level like I have with netperf using dfsio. Any suggestions would help greatly. @nauseous wrote: I'm trying to get maximum throughput with Cloudera on RedHat 6.6 on 6 - Dell R730's with kernel 3.18.1, and using 2 - 850MB, 3G ssd transfer per second hhd with modified drivers which have been tested. Currently I've tried decommissioning "mapReduce tasktracker" on all nodes except 1 single node as suggested but didn't really make any differences in nic speed. I want to max out the connection speed on all nodes if possible. I've tried : sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 100000 -fileSize 50 and sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 500 -fileSize 10GB without good results. I've already tested throughput with netperf but can't seem to get cloudera to perform network tests to maximum level like I have with netperf using dfsio. Any suggestions would help greatly.
... View more
03-16-2015
02:13 PM
I'm trying to get maximum throughput with Cloudera on RedHat 6.6 on 6 - Dell R730's with kernel 3.18.1, and using 2 - 850MB, 3G ssd transfer per second hhd with modified drivers which have been tested. Currently I've tried decommissioning "mapReduce tasktracker" on all nodes except 1 single node as suggested but didn't really make any differences in nic speed. I want to max out the connection speed on all nodes if possible. I've tried : sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 100000 -fileSize 50 and sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hadoop-test-2.5.0-mr1-cdh5.3.1.jar TestDFSIO -write -nrFiles 500 -fileSize 10GB without good results. I've already tested throughput with netperf but can't seem to get cloudera to perform network tests to maximum level like I have with netperf using dfsio. Any suggestions would help greatly.
... View more
Labels:
03-09-2015
08:38 AM
I wasn't able to find my post sense I don't have my machine but here is a great article here http://tinyurl.com/lr3gfx4 Hope this helps 🙂
... View more
03-09-2015
08:28 AM
Great job. I try to keep the names as simple as possible so I can run thousands of scripts. My hosts files is like: 127.0.0.1 localhost Looping interfaces from multiple machines. AWS or for example Linode looping you just would use the internal looping device. Fast and quickly managed. #Cloudera Machines 192.168.2.1 n1 192.168.2.2 n2 192.168.2.3 n3 " n4 " n5 " n6 " n7 and so on to make it easier for changes across machines. Such as: for i in {1..300}; do ssh n$i date; done <-- Checks dates on all machines to make sure each machine is sync'ed. Makes life easier to make it simple.
... View more
03-09-2015
08:22 AM
Great job. I try to keep the names as simple as possible so I can run thousands of scripts. My hosts files is like: 127.0.0.1
... View more
03-06-2015
03:56 PM
Sounds like your nsswitch is wrong. It should be "files dns" not "dns files". I would definately check it out and verify you don't have it setup wrong.
... View more
03-06-2015
01:37 PM
I found it's location under JobTracker Handler Count mapred.job.tracker.handler.count in mapreduce.
... View more
03-06-2015
12:49 PM
1 Kudo
I've install it to RedHat 6.6 and works perfectly without issues. I actually created an article with prerequisites and they just posted it if you need that link let me know.
... View more
03-06-2015
12:23 PM
Hi, I've been searching for some time but related articles just don't explain where the location of the Launched Maps 100 which shows in the Cloudera web interface. I just to need to change it from 100 to 0. I've looked everywhere and finally I am here
... View more