Member since
12-04-2013
16
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 13068 | 01-06-2014 04:52 PM | |
| 38366 | 12-25-2013 01:33 PM |
02-12-2014
03:22 PM
Yep, reinstalling the components was as you two said, but I only needed to copy the config files that yum saved over the new config files. This was not a disaster. What I should have done is only uninstall the zookeeper-server package, but I also uninstalled the zookeeper package. Thanks!
... View more
02-05-2014
06:05 PM
I should have obviously double checked, but I was trying to get zookeeper off of my slave and one master nodes by issuing: yum remove zookeeper I just said yes without looking because I didn't figure this one command could wipe all Hadoop related processes from my nodes. Here is a sample of what happened: Two questions: Can I do anything to get my cluster back without a total reinstall? Why does it do this in the first place? When I removed HBase it didn't decide to remove all the other programs. Thank you
... View more
Labels:
01-14-2014
06:06 PM
The "Using Hive with HBase", linked here, contains the following instructions: To allow Hive scripts to use HBase, add the following statements to the top of each script. Replace the <component_version> strings with current version numbers for CDH, Guava and the Hive HBase handler....
ADD JAR /usr/lib/hive/lib/zookeeper.jar;
ADD JAR /usr/lib/hive/lib/hive-hbase-handler-<Hive-HBase-Handler_version>-cdh<CDH_version>.jar
ADD JAR /usr/lib/hive/lib/guava-<Guava_version>.jar;
For example,
ADD JAR /usr/lib/hive/lib/zookeeper.jar;
ADD JAR /usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.4.0.jar
ADD JAR /usr/lib/hive/lib/guava-11.0.2.jar;
However, when issuing a INSERT OVERWRITE TABLE hbase_table ... command in hive, I received the following error: Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase This was fixed by adding the following jar in addition to the above jars: ADD JAR /usr/lib/hive/lib/hbase.jar And I think you should also point out the fact that people using Hive with HBase don't need to issue the ADD JAR command at the begining of every single script; all they need to do is edit the hive-site.xml file on each node they execute Hive from and include the following property: <property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar,file:///usr/lib/hive/lib/hbase.jar</value>
</property>
... View more
01-13-2014
01:25 PM
Ok great. I restarted all MapReduce and Hbase daemons in addition to the HDFS daemons and it is working properly now. Thank you! Previously I had only restarted HDFS.
... View more
01-10-2014
01:49 PM
I have now installed a 5 node cluster with the following configuration: Master 1: NameNode Master 2: Secondary NameNode, JobTracker, HMaster, Hive MetaStore, Slave 1: TaskTracker, DataNode, HRegionServer Slave 2: TaskTracker, DataNode, HRegionServer Slave 3: TaskTracker, DataNode, HRegionServer I installed Ooize on my Master 2, along with a MySQL database, and ran the same steps. For Master 1 (NameNode) and Master 2 (JobTracker), I added the following properties to the core-site.xml as before: <property
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property> And restarted HDFS on both of the nodes. Now, I can issue the oozie job ... command and from my bash it executes successfully with no errors. However, when I log into the Oozie Web Console, I am told that the job cannot successfully because "JA009: User: oozie is not allowed to impersonate root". Here are the logs: 2014-01-10 13:58:05,677 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@:start:] Start action [0000003-140110133137421-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2014-01-10 13:58:05,678 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@:start:] [***0000003-140110133137421-oozie-oozi-W@:start:***]Action status=DONE 2014-01-10 13:58:05,678 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@:start:] [***0000003-140110133137421-oozie-oozi-W@:start:***]Action updated in DB! 2014-01-10 13:58:05,791 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Start action [0000003-140110133137421-oozie-oozi-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2014-01-10 13:58:06,083 WARN MapReduceActionExecutor:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] credentials is null for the action 2014-01-10 13:58:06,500 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: User: oozie is not allowed to impersonate root] org.apache.oozie.action.ActionExecutorException: JA009: User: oozie is not allowed to impersonate root at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418) at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:773) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:927) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59) at org.apache.oozie.command.XCommand.call(XCommand.java:277) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: oozie is not allowed to impersonate root at org.apache.hadoop.ipc.Client.call(Client.java:1238) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225) at org.apache.hadoop.mapred.$Proxy30.getDelegationToken(Unknown Source) at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:2125) at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:372) at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:970) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723) ... 10 more 2014-01-10 13:58:06,501 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Next Retry, Attempt Number [1] in [60,000] milliseconds 2014-01-10 13:59:06,556 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Start action [0000003-140110133137421-oozie-oozi-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2014-01-10 13:59:06,692 WARN MapReduceActionExecutor:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] credentials is null for the action 2014-01-10 13:59:07,028 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: User: oozie is not allowed to impersonate root] org.apache.oozie.action.ActionExecutorException: JA009: User: oozie is not allowed to impersonate root at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418) at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:773) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:927) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59) at org.apache.oozie.command.XCommand.call(XCommand.java:277) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: oozie is not allowed to impersonate root at org.apache.hadoop.ipc.Client.call(Client.java:1238) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225) at org.apache.hadoop.mapred.$Proxy30.getDelegationToken(Unknown Source) at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:2125) at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:372) at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:970) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723) ... 8 more 2014-01-10 13:59:07,029 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000003-140110133137421-oozie-oozi-W] ACTION[0000003-140110133137421-oozie-oozi-W@mr-node] Next Retry, Attempt Number [2] in [60,000] milliseconds Any ideas? Thank you.
... View more
01-09-2014
10:37 AM
Yesterday I configured a 5 node cluster via Cloudera Manager. I then added another node, and was about to make a "Gateway Template" to assign to it through the Add Node workflow, when my Firefox crashed. I loaded it back up and manually assigned all the gateway roles to it (HDFS, MapReduce, HBase, Hive). However, when I issue "hadoop fs -ls" on the node, it basically prints out the results of "ls -la" on my local filesystem. As in, whichever folder I am in on the local filesystem, "hadoop fs -ls" will show me all the files, their permissions, and hidden files. What is the fix for this? Would it hurt to make another node, and instead of assigning it Gateway roles through Cloudera Manager or downloading any Cloudera Agent, to simply download the client packages for HDFS, MapReduce, Hive, HBase, Pig, Oozie, etc. and manually connect it as a "gateway" node? Thank you.
... View more
Labels:
01-06-2014
04:52 PM
1 Kudo
Hi Smark; thanks for your reply. service cloudera-scm-server-db status pg_ctl: no server running psql -U scm -p 7432 psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.7432"? service cloudera-scm-server-db start DB initialization done. waiting for server to start....... done server started service cloudera-scm-server-db status pg_ctl: server is running (PID: 24118) /usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data" After turning on the server, it worked. Thanks!
... View more
01-06-2014
02:09 PM
I haven't logged into the Clouder Manager web portal in a few weeks so I don't know for how long this has happened, but when I tried to log into it today via a web browser, it took a long time and responded with a Server Not Found. I SSHed into the node with cloudera manager and ran the following commands, receiving the following responses: curl localhost:7180 <head><meta http-equiv="refresh" content="0;url=/cmf/"></head> wget localhost:7180 HTTP request sent, awaiting response... No data received. Retrying. I was able to "telnet 7180" successfully. netstat -nltp | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 10875/java Here is some sample output from /var/log/cloudera-scm/server/cloudera-scm-server.log: 2014-01-06 12:09:44,361 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@7090b499:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5a670992 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) ... Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ... 2014-01-06 12:26:34,367 INFO [591785153@scm-web-5346:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-SERVICEMONITOR-7e2762fc6af4771300e7773daf64993b 2014-01-06 12:26:34,379 WARN [773346975@scm-web-5366:spi.SqlExceptionHelper@143] SQL Error: 0, SQLState: null 2014-01-06 12:26:34,388 ERROR [773346975@scm-web-5366:spi.SqlExceptionHelper@144] Connections could not be acquired from the underlying database! 2014-01-06 12:26:34,389 INFO [773346975@scm-web-5366:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-EVENTSERVER-7e2762fc6af4771300e7773daf64993b 2014-01-06 12:27:03,384 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@4995f35:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@255f13b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) Seeing there was a problem with PostgreSQL, issuing "psql" returned the following: psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? Issuing "service postgresql status" returned: postmaster is stopped Issuing "service postgresql start" returned: /var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first. [FAILED] Any ideas on why this occurred and what I may do to fix it? Thank you.
... View more
Labels:
- Labels:
-
Cloudera Manager
12-28-2013
05:57 PM
I currently only have a three node cluster, so I have the JobTracker and NameNode on a single node (which I call "The Master Node"). Thank you for that fact, however, as I plan on attempting to manually install a larger cluster. Would it be necessary to add to my core-site.xml's on my Slave Nodes as well? I am practicing manual installations just to understand the inner workings, but when my company decides to move into production with our POCs I will definitely use Cloudera Manager. Would you happen to know if there is a list of properties configured out of the box in Cloudera Manager that one has to look out for when doing manual installations? Thank you Harsh J.
... View more
12-25-2013
01:33 PM
2 Kudos
Ok, well after I restarted HDFS via: for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x restart ; done The proxy settings that I added to core-site.xml appeared to have kicked in and I could then run the example. However, when I go to MY-VM-ALIAS:11000/oozie, the job's status is KILLED immediately. If I double click on the job, and then double click on the action item with the name of fail, I can see that the error message is: Map/Reduce failed, error message[rRuntimeException: Error in configuing object] Clicking on the Job Log tab, I saw this: Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:134)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:174)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:38)
... 29 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1680)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:127)
... 31 more Google suggested that I put the "hadoop-lzo.jar in /var/lib/oozie/ and [restart] Oozie." So I issued (on my master node with the Oozie server): find / -name hadoop-lzo.jar
cp /usr/lib/hadoop/lib/hadoop-lzo.jar /var/lib/oozie/
sudo service oozie restart and my job ran and succeeded!
... View more