Created on 11-20-2014 03:50 PM - edited 09-16-2022 02:13 AM
Ok, I am using the quick start VM version 5.2.0. THis iscludes Hue, Hive, Sqoop, and other tools. I have not changed any configuration files.
When I run Hive queries in the Hive editor, they run fine. Also when I execute this Hive SQL command:
It runs fine, the tables it shows are the same in the left hand column.
Then I create an Oozie workflow with a Hive script. The script contains only one command:
The workflow runs with no problem. The submitter is Cloudera.
Then I use Hue and downlaod all the samples for Oozie and the other parts of the framework. I submit the sample workflow that runs a Hive step. The workflow does not succeed. Here is the error:
ava.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
Here is more detail:
2014-11-20 15:06:06,926 WARN HiveActionExecutor:544 - SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[Hive] JOB[0000004-141120121239453-oozie-oozi-W] ACTION[0000004-141120121239453-oozie-oozi-W@Hive] Launcher exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
Here are the settings for the metastore. The file is in /etc/hive/config.dist/hive-stie.xml
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property>
I am trying to allow Oozie to aaccess the metastore. So I access mysql, but I do not know the password for the local instalation of MySQL. Here is what I have tried:
[cloudera@quickstart ~]$ mysql -u root -p Enter password: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) [cloudera@quickstart ~]$ mysql -u root -p Enter password: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) [cloudera@quickstart ~]$ mysql -u root ERROR 1045 (28000): Access denied for user
'root'@'localhost' (using password: NO)
[cloudera@quickstart ~]$
Should I try to make the MySQL instance accessable to Oozie? What config file is Oozie using? I hear some people changing to a Derby database, but if Hue comes with the Quick Start, it should work with the Quick Start, right? Does the Horton Sandbox have the same problems?
Created 11-21-2014 08:13 AM
So I found the issue. The example has a Hive configuration hard-coded in it that uses Derby as the metastore. I'll get this fixed in the next release, but to work around it now you can overwrite it with the real Hive configuration as follows:
sudo cp /etc/hive/conf/hive-site.xml /usr/lib/hue/apps/oozie/examples/managed/hive/hive-config.xml sudo cp /etc/hive/conf/hive-site.xml /usr/lib/hue/apps/oozie/examples/unmanaged/hive/hive-config.xml # And if you already installed the example in Hue... sudo -u hdfs hadoop fs -rm /user/hue/oozie/workspaces/managed/hive/hive-config.xml sudo -u hdfs hadoop fs -copyFromLocal /etc/hive/conf/hive-site.xml /user/hue/oozie/workspaces/managed/hive/hive-config.xml
sudo -u hdfs hadoop fs -rm /user/hue/oozie/workspaces/unmanaged/hive/hive-config.xml sudo -u hdfs hadoop fs -copyFromLocal /etc/hive/conf/hive-site.xml /user/hue/oozie/workspaces/unmanaged/hive/hive-config.xml
Thanks for the feedback!
Created 11-21-2014 09:49 AM
I didn't check any logs (other than the ones you already found). I view the configuration for that example in the Oozie app in Hue by going to edit it, and saw that it reference hive-config.xml (which caught my eye because as you noticed, the real copy is called hive-site.xml). So I searched for files with that name in the native file system and in Hadoop, and saw that those files were bundled in Hue and included a copy of the default Hive configuration that uses a Derby metastore:
find / -name hive-config.xml 2>/dev/null sudo -u hdfs hadoop fs -ls -R | grep hive-config.xml
Created 11-21-2014 12:19 PM
One thing to mention. If a person has already run a workflow, just running it again won't work, the old settings are still there. More is needed.
1. Use the Job Browser to find the id of the workflow, or the use the Oozie dashboard for workflows.
2. Select on the ID
3. Select the Details tab, then click on the application path.
4. Now youare in the File Browser, select hive-config.xml and check the hive configurations in that file.
Created on 11-21-2014 07:32 AM - edited 11-21-2014 07:34 AM
Per the documentation, the password for all MySQL users including root, hive, hue and oozie is 'cloudera'. In addition to the snippet you posted from hive-site.xml, you should also see those credentials configured there:
<property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>cloudera</value> </property>
Oozie's configuration is in /etc/oozie/conf, and it too appears to be correctly configure to talk to MySQL, but the log does indicate that it's trying to use Derby and failing. I'll need to look into why.
Created 11-21-2014 08:13 AM
So I found the issue. The example has a Hive configuration hard-coded in it that uses Derby as the metastore. I'll get this fixed in the next release, but to work around it now you can overwrite it with the real Hive configuration as follows:
sudo cp /etc/hive/conf/hive-site.xml /usr/lib/hue/apps/oozie/examples/managed/hive/hive-config.xml sudo cp /etc/hive/conf/hive-site.xml /usr/lib/hue/apps/oozie/examples/unmanaged/hive/hive-config.xml # And if you already installed the example in Hue... sudo -u hdfs hadoop fs -rm /user/hue/oozie/workspaces/managed/hive/hive-config.xml sudo -u hdfs hadoop fs -copyFromLocal /etc/hive/conf/hive-site.xml /user/hue/oozie/workspaces/managed/hive/hive-config.xml
sudo -u hdfs hadoop fs -rm /user/hue/oozie/workspaces/unmanaged/hive/hive-config.xml sudo -u hdfs hadoop fs -copyFromLocal /etc/hive/conf/hive-site.xml /user/hue/oozie/workspaces/unmanaged/hive/hive-config.xml
Thanks for the feedback!
Created 11-21-2014 09:45 AM
Which log did you check and how did you check it?
Created 11-21-2014 09:49 AM
I didn't check any logs (other than the ones you already found). I view the configuration for that example in the Oozie app in Hue by going to edit it, and saw that it reference hive-config.xml (which caught my eye because as you noticed, the real copy is called hive-site.xml). So I searched for files with that name in the native file system and in Hadoop, and saw that those files were bundled in Hue and included a copy of the default Hive configuration that uses a Derby metastore:
find / -name hive-config.xml 2>/dev/null sudo -u hdfs hadoop fs -ls -R | grep hive-config.xml
Created 11-21-2014 12:19 PM
One thing to mention. If a person has already run a workflow, just running it again won't work, the old settings are still there. More is needed.
1. Use the Job Browser to find the id of the workflow, or the use the Oozie dashboard for workflows.
2. Select on the ID
3. Select the Details tab, then click on the application path.
4. Now youare in the File Browser, select hive-config.xml and check the hive configurations in that file.