Member since
02-12-2016
37
Posts
6
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 24551 | 08-08-2016 08:35 AM | |
| 4257 | 04-06-2016 03:29 PM | |
| 25705 | 02-25-2016 07:25 AM |
01-04-2019
10:25 AM
Thanks @cjervis For informations I get the same things with an Ubuntu 18. https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_cdh5_install.html
... View more
01-02-2019
08:03 AM
Hi, Any update ? I still have the same issue. Thanks.
... View more
10-03-2016
02:38 PM
I try to show the the config of one of our job. Same error. Launch since my client workstation. bash: sqoop job --show sqoop_job_name
Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/10/03 17:36:42 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.1
16/10/03 17:36:43 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: Passed Null for map null
java.lang.IllegalArgumentException: Passed Null for map null
at org.apache.sqoop.util.SqoopJsonUtil.getMapforJsonString(SqoopJsonUtil.java:48)
at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:621)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
at org.apache.sqoop.tool.JobTool.showJob(JobTool.java:232)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:287)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
... View more
10-03-2016
02:21 PM
So "sqoop job --list" commans works. But not "sqoop job --exec" not... Are my parameters are wrong ? I try to replace the <commande> by <arg> syntax. Same error.
... View more
10-03-2016
01:17 PM
Hi gezapeti, I try but it doesn't work anymore. But I find something else. I try in my worlfow to change the sqoop job. I replace it by a simple action as "sqoop job --list" The worflow run successfully but the output is empty. Now if I keep the same "sqoop job --list" command, but this time I add the "--meta-connect" option with the url of our sqoop-metatsore, I have this error: 3557 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
3577 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.7.1
3676 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Exception creating SQL connection
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:216)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.open(HsqldbJobStorage.java:161)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:274)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:388)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:302)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: socket creation error
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
at org.hsqldb.jdbcDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:174)
... 29 more
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] As we see, its can come from a missing librairie ou something else. It can't find the a driver to connect on sqoop-metastore. But I find the hsql driver on multiple path, on hdfs and system folder... I thing its to close to find the solution but so far too !! Hope it can give you any idea !
... View more
09-29-2016
04:02 PM
Sorry gezapeti, I can't give you an answer. When I make a post, it not publish, without error message... But you configuration doesn't work. I try but have another error... 3755 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 3776 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.7.1 3948 [uber-SubtaskRunner] ERROR org.apache.sqoop.Sqoop - Got exception running Sqoop: java.lang.IllegalArgumentException: Passed Null for map null Intercepting System.exit(1)
... View more
09-29-2016
04:00 PM
I give you more informations with value of our workflow. Name Value
db_pass password
db_user username
hive_db_schema test
job_tracker host:8032
job_xml /cache/sqoop-site.xml
mapreduce.job.user.name user1
name_node hdfs://namenodeha
oozie.libpath /user/oozie/share/lib/lib_20160808152436/
oozie.use.system.libpath true
oozie.wf.application.path hdfs://namenodeha/user/user1/test//workflow-extract.xml
target_dir /user/user1/test//test_sqoop
user.name user1
wf_path /user/user1/test/ Maybe something wrong inside, but I don't see it... Or problem still present and don't find any solution 😞 I thinks the problem is very easy, but I don't find it. On our labs cluster with 5.5.1 version, nothing specific to do to allow oozie exec sqoop actions. All configurations are identical between us. So weird... Configuation from oozie in CM ? Variable need to be declare ?
... View more
09-29-2016
03:45 PM
HI gezapeti, Thanks for you answer, but it doesn't work. We use since the begining the <job-xml> to specifiy the file hive-site.xml. So as you suggest, I try to add this file separated by a comma, but it doesn't work. So I try to replace the hive-site.xml by the sqoop-site.xml without more success. Here te return: Sqoop command arguments :
job
--exec
sqoop_job_name
--
--target-dir
/path/to/destination
--username
db_user
--password
db_password
--hive-import
--hive-table
db.table
Fetching child yarn jobs
tag id : oozie-e6933cd082a6f0d7ab69512a867cbac3
Child yarn jobs are found -
=================================================================
>>> Invoking Sqoop command line now >>>
3755 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
3776 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.7.1
3948 [uber-SubtaskRunner] ERROR org.apache.sqoop.Sqoop - Got exception running Sqoop: java.lang.IllegalArgumentException: Passed Null for map null
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Have you another idea ?
... View more
09-23-2016
11:14 AM
Prroblem still present... Any ideas why and how to solve this problem ?
... View more
09-22-2016
01:10 PM
I try to start our sqoop command directlry from the oozie server. It doesn't work, and I have this error: Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/22 20:06:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.1
16/09/22 20:06:48 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: Passed Null for map null
java.lang.IllegalArgumentException: Passed Null for map null
at org.apache.sqoop.util.SqoopJsonUtil.getMapforJsonString(SqoopJsonUtil.java:48)
at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:621)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236) My command: sqoop job --exec sqoo_job_name -- --target-dir /user/user1/test/dstdir --username db_user --password db_password I remove the part "hive import" to simply the command and it can have a result. Problem still here, but with another error when I try to launch the sqoop comand only fron oozie server.
... View more
09-22-2016
12:29 PM
Hi all, I encount a problem with our new cluster on Cloudera 5.7.1. All installation is done, all service are up and running. HA is enable for namdenode. We try some job, and it works fine as: - sqoop import - oozie workflows with hive job But this one won't to work: - oozie worflow with sqoop import. My sqoop-metastore is runing on "node1", and oozie server on "node2". The sqoop-metastore run with init.d from the installation. If I'm on the node2, and do : sudo -u oozie sqoop job --list we can seel all stored job in the sqoop-metastore. But we start an simple oozie worflo, which only start a sqoop import, it dosen't work. I get this error: >>> Invoking Sqoop command line now >>>
3603 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
3624 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.7.1
3873 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - Cannot restore job: sqoop_job_name
3873 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - (No such job)
3873 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Cannot restore missing job sqoop_job_name But I don't know hy, cause it works with the command line as oozie user... I try top copy the sqoop-site.xml in the /etc/oozie/conf dir, but it doesn't work. Only one value is set in the sqoop-site,xml <property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://host.domain.ltd:16000/sqoop</value>
<description>Port that this metastore should listen on.</description>
</property> Do I forget something? Do I have something more to do to allow oozie start sqoop job ? Thanks for you help ! Regards,
... View more
09-22-2016
10:46 AM
Forget my last try,, the yarn manager is not the this machine, so the job can't start. I keep my config where the Yarn Manage is running, but kee the same problem...
... View more
09-22-2016
09:14 AM
Update: We use HA. And in the xml configuration file, we give these informations <property><name>job_tracker</name><value>activenode.domain.com:8032</value></property>
<property><name>job_xml</name><value>/cache/hive-site.xml</value></property>
<property><name>name_node</name><value>hdfs://namenodeha</value></property>
<property><name>oozie.use.system.libpath</name><value>true</value></property>
<property><name>oozie.libpath</name><value>/user/oozie/share/lib/lib_20160808152436</value></property>
<property><name>wf_path</name><value>/user/user1/test/</value></property>
<property><name>oozie.wf.application.path</name><value>${name_node}${wf_path}/workflow-extract.xml</value></property>
<property><name>db_user</name><value>db_login</value></property>
<property><name>db_pass</name><value>db_password</value></property>
<property><name>target_dir</name><value>${wf_path}/test_sqoop</value></property>
<property><name>hive_db_schema</name><value>test</value></property> But the "job_tracker" was set to the "backup" node int the HA configuration. I try to set it with the "active" node, the job start but, go to "running" status, but the task stay in "PREP" status... Our oozie workflow: <workflow-app xmlns="uri:oozie:workflow:0.5" name="cba_1333-extract">
<global>
<job-tracker>${job_tracker}</job-tracker>
<name-node>${name_node}</name-node>
<job-xml>${job_xml}</job-xml>
</global>
<start to="extract"/>
<action name="extract">
<sqoop xmlns="uri:oozie:sqoop-action:0.4">
<command>
job --exec sqoop_job_name -- --target-dir /user/user1/test/db_import --username db_user --password db_password --hive-import --hive-table db.table
</command>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app> Any idea for what the the workflow dosen't work ? If we start a sqoop alone, job, it works, If we start a an oozie, with hive job, it works... Any idea ?
... View more
09-20-2016
08:04 AM
Hi all, I come back with a problem with Oozie and Sqoop. We are on Cloudera 5.7.1. For information, the sqoop-metastore run on a psecific machine, and Oozie server on another one, without firewall between us. node1: sqoop-metastore server node2: oozie server If we start a sqoop job from the cluster in Hue or our client machine, everything fine. Sqoop start, and import from Mysql into Hive is done successfully. Now we use Oozie to create a workflow with a step to do the same import. So what I do: - Run the sqoop-metastore as service on node1 on 16000 port - Copy the sqoop-site.xml with these value from /etc/sqoop/conf/sqoop-site.xml to /etc/oozie/conf/sqoop-site.xml on node2. To check if the connection to the metastore works, I launch this command from the node2 sudo -u oozie sqoop job --list It works fine. All sqoop job configuration are displays. But when we try to start a simple Oozie worflow with just this import, we have an error: >>> Invoking Sqoop command line now >>>
3297 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
3311 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.7.1
3549 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - Cannot restore job: sqoop_job_name
3549 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - (No such job)
3550 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Cannot restore missing job sqoop_job_name
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:256)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:388)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:302)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] I don't understand why it can find the job while the ssh command from the oozie server node works... In the sqoop-site.xml, we set these parameters: <property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://remotehost:16000/sqoop</value>;
<description>Port that this metastore should listen on.</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>SA</value>
<description>The username to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
<description>The password to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.record.password</name>
<value>true</value>
<description>If true, allow saved passwords in the metastore.
</description>
</property> The workflow configuration: <action name="extract">
<sqoop xmlns="uri:oozie:sqoop-action:0.4">
<command>
job --exec sqoop_job_name -- --target-dir /path/to/dir --username username --password password --hive-import --hive-table database.table
</command>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action> We try many settings, but nothing works.... And we can't continue to works without create our workflow... Somebody already encount this problem ? Or have an idea from what or where the problem come from ? Thanks a lot for your help.
... View more
- Tags:
- Sqoop
Labels:
- Labels:
-
Oozie
08-08-2016
08:35 AM
1 Kudo
It works !! I find the workaround. After read again the log, I find weird the python error. So I would like to check again this error. The user zookeeper have the uid 167 and gid 153. I change the gid to 167 in /etc/passwd and /etc/group, and the db test works now !! I can now continue the setup. But I don't know why the setup wasn't good... Maybe a bug in Redhat package for the 5.7.1 version. I don't know. But finally, it works, I finish my setup and my cluster is now live ! Now I'g going to configure and optimize the configuration ! Maybe the post can help somebody. Regards,
... View more
08-08-2016
06:46 AM
Hi bgolley, Thanks for you help. Ok, il can check only on the client so. But I've only the Cloudera agent log which give me an output. And it's like that: [08/Aug/2016 13:39:02 +0000] 13735 MainThread agent INFO Flood daemon (re)start attempt
[08/Aug/2016 13:39:02 +0000] 13735 MainThread agent ERROR Failed to handle Heartbeat Response: {u'firehoses': [], u'rm_enabled': False, u'client_configs': [], u'create_parcel_symlinks': True, u'server_managed_parcels': [], u'extra_configs': None, u'host_collection_config_data': [{u'config_name': u'host_network_interface_collection_filter', u'config_value': u'^lo$'}, {u'config_name': u'host_disk_collection_filter', u'config_value': u'^$'}, {u'config_name': u'host_fs_collection_filter', u'config_value': u'^$'}, {u'config_name': u'host_log_tailing_config', u'config_value': u'{}\n'}, {u'config_name': u'host_dns_resolution_duration_thresholds', u'config_value': u'{"critical":"never","warning":"1000.0"}'}, {u'config_name': u'host_dns_resolution_enabled', u'config_value': u'true'}, {u'config_name': u'host_clock_offset_thresholds', u'config_value': u'{"critical":"10000.0","warning":"3000.0"}'}], u'apply_parcel_users_groups_permissions': True, u'flood_torrent_port': 7191, u'log_tailing_config': u'{}\n', u'active_parcels': {}, u'flood_rack_peers': [u'10.2.0.34:7191', u'10.2.0.31:7191', u'10.2.0.30:7191', u'10.2.0.33:7191', u'10.2.0.29:7191'], u'retain_parcels_in_cache': True, u'processes': [{u'status_links': {}, u'name': u'OOZIE.OOZIE_SERVER-test-db-connection', u'config_generation': 0, u'configuration_data': 'PK\x03\x04\x14\x00\x08\x08\x08\x00\xd7l\x08I\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\r\x00\x00\x00db.propertiesu\xcc1\n\x800\x0c@\xd1\xdd\xbb\x18*\x08\xe2\xd0\x0bx\x01\xe7\xd8\x04,\xb4Mm\xac\xa2\xa7wW\x9c\xdf\xe7;\x89\xe0\x82T\xe2\x82@\x0b$\x8clEn\xcf\x8d{QF\xd5S\n\xd9\xf1\x98]\x97\x8ei\xf0\xbc|\xaa\xfd\xcal\xe3\xa5[\xf8PU.?\xefUt\xb7\x9a|\xbb"\x89\xe4\xd6\x98\x1eB%\x8f\x10\xb9y\x00PK\x07\x08\xf5\xc6w;b\x00\x00\x00\xa4\x00\x00\x00PK\x01\x02\x14\x00\x14\x00\x08\x08\x08\x00\xd7l\x08I\xf5\xc6w;b\x00\x00\x00\xa4\x00\x00\x00\r\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00db.propertiesPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00;\x00\x00\x00\x9d\x00\x00\x00\x00\x00', u'refresh_files': [], u'user': u'root', u'parcels': {}, u'auto_restart': False, u'run_generation': 1, u'extra_groups': [], u'environment': {}, u'optional_tags': [], u'running': True, u'program': u'dbconnection/test_db_connection.sh', u'required_tags': [], u'arguments': [u'db.properties'], u'special_file_info': [], u'group': u'root', u'id': 191, u'resources': [], u'one_off': True}], u'server_manages_parcels': True, u'heartbeat_interval': 15, u'parcels_directory': u'/opt/cloudera/parcels', u'host_id': u'549b5395-76f6-43ee-86d2-ef7091a71ce1', u'eventserver_host': None, u'enabled_metric_reporters': [u'ACCUMULO16', u'ACCUMULO16', u'KEYTRUSTEE-KMS_KEYTRUSTEE', u'KMS_KEYTRUSTEE', u'SPARK_ON_YARN-SPARK_YARN_HISTORY_SERVER', u'SPARK_YARN_HISTORY_SERVER', u'SOLR-SOLR_SERVER', u'SOLR_SERVER', u'HBASE-HBASERESTSERVER', u'HBASERESTSERVER', u'HOST', u'KEYTRUSTEE_SERVER-KEYTRUSTEE_PASSIVE_SERVER', u'KEYTRUSTEE_PASSIVE_SERVER', u'IMPALA-STATESTORE', u'STATESTORE', u'SPARK', u'SPARK', u'HBASE', u'HBASE', u'ACCUMULO-ACCUMULO_TRACER', u'ACCUMULO_TRACER', u'HDFS-DATANODE', u'DATANODE', u'ACCUMULO-ACCUMULO_MASTER', u'ACCUMULO_MASTER', u'YARN-RESOURCEMANAGER', u'RESOURCEMANAGER', u'HUE-HUE_SERVER', u'HUE_SERVER', u'ACCUMULO-ACCUMULO_MONITOR', u'ACCUMULO_MONITOR', u'MGMT-EVENTSERVER', u'EVENTSERVER', u'MGMT-NAVIGATORMETASERVER', u'NAVIGATORMETASERVER', u'HBASE-MASTER', u'MASTER', u'KAFKA-KAFKA_BROKER', u'KAFKA_BROKER', u'KEYTRUSTEE_SERVER-DB_PASSIVE', u'DB_PASSIVE', u'HBASE-REGIONSERVER', u'REGIONSERVER', u'SPARK_ON_YARN', u'SPARK_ON_YARN', u'MGMT-REPORTSMANAGER', u'REPORTSMANAGER', u'MGMT-SERVICEMONITOR', u'SERVICEMONITOR', u'IMPALA-IMPALAD', u'IMPALAD', u'MGMT-ALERTPUBLISHER', u'ALERTPUBLISHER', u'HIVE-HIVESERVER2', u'HIVESERVER2', u'MGMT-ACTIVITYMONITOR', u'ACTIVITYMONITOR', u'ISILON', u'ISILON', u'YARN-NODEMANAGER', u'NODEMANAGER', u'MAPREDUCE-FAILOVERCONTROLLER', u'FAILOVERCONTROLLER', u'ACCUMULO', u'ACCUMULO', u'MAPREDUCE', u'MAPREDUCE', u'ZOOKEEPER', u'ZOOKEEPER', u'KMS', u'KMS', u'ACCUMULO16-ACCUMULO16_TRACER', u'ACCUMULO16_TRACER', u'ACCUMULO16-ACCUMULO16_MONITOR', u'ACCUMULO16_MONITOR', u'MGMT-HOSTMONITOR', u'HOSTMONITOR', u'YARN-JOBHISTORY', u'JOBHISTORY', u'KEYTRUSTEE', u'KEYTRUSTEE', u'HDFS-JOURNALNODE', u'JOURNALNODE', u'KAFKA', u'KAFKA', u'IMPALA', u'IMPALA', u'SPARK-SPARK_HISTORY_SERVER', u'SPARK_HISTORY_SERVER', u'KEYTRUSTEE_SERVER-KEYTRUSTEE_ACTIVE_SERVER', u'KEYTRUSTEE_ACTIVE_SERVER', u'HDFS-NAMENODE', u'NAMENODE', u'HUE-BEESWAX_SERVER', u'BEESWAX_SERVER', u'SOLR', u'SOLR', u'ACCUMULO16-ACCUMULO16_TSERVER', u'ACCUMULO16_TSERVER', u'MAPREDUCE-TASKTRACKER', u'TASKTRACKER', u'IMPALA-CATALOGSERVER', u'CATALOGSERVER', u'HDFS-DSSDDATANODE', u'DSSDDATANODE', u'SENTRY', u'SENTRY', u'ACCUMULO16-ACCUMULO16_GC', u'ACCUMULO16_GC', u'MGMT-NAVIGATOR', u'NAVIGATOR', u'HIVE', u'HIVE', u'HBASE-HBASETHRIFTSERVER', u'HBASETHRIFTSERVER', u'SQOOP-SQOOP_SERVER', u'SQOOP_SERVER', u'KAFKA-KAFKA_MIRROR_MAKER', u'KAFKA_MIRROR_MAKER', u'FLUME', u'FLUME', u'HUE', u'HUE', u'HDFS-SECONDARYNAMENODE', u'SECONDARYNAMENODE', u'SENTRY-SENTRY_SERVER', u'SENTRY_SERVER', u'ACCUMULO-ACCUMULO_TSERVER', u'ACCUMULO_TSERVER', u'ACCUMULO-ACCUMULO_GC', u'ACCUMULO_GC', u'HIVE-HIVEMETASTORE', u'HIVEMETASTORE', u'IMPALA-LLAMA', u'LLAMA', u'ACCUMULO16-ACCUMULO16_MASTER', u'ACCUMULO16_MASTER', u'SPARK-SPARK_WORKER', u'SPARK_WORKER', u'MGMT', u'MGMT', u'HIVE-WEBHCAT', u'WEBHCAT', u'SQOOP', u'SQOOP', u'HUE-HUE_LOAD_BALANCER', u'HUE_LOAD_BALANCER', u'ACCUMULO-ACCUMULO_LOGGER', u'ACCUMULO_LOGGER', u'HDFS', u'HDFS', u'FLUME-AGENT', u'AGENT', u'OOZIE', u'OOZIE', u'SQOOP_CLIENT', u'SQOOP_CLIENT', u'OOZIE-OOZIE_SERVER', u'OOZIE_SERVER', u'KMS-KMS', u'KMS', u'HDFS-FAILOVERCONTROLLER', u'FAILOVERCONTROLLER', u'KS_INDEXER', u'KS_INDEXER', u'SPARK-SPARK_MASTER', u'SPARK_MASTER', u'YARN', u'YARN', u'ZOOKEEPER-SERVER', u'SERVER', u'HDFS-NFSGATEWAY', u'NFSGATEWAY', u'HDFS-HTTPFS', u'HTTPFS', u'HUE-KT_RENEWER', u'KT_RENEWER', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER-DB_ACTIVE', u'DB_ACTIVE', u'MAPREDUCE-JOBTRACKER', u'JOBTRACKER', u'KS_INDEXER-HBASE_INDEXER', u'HBASE_INDEXER'], u'flood_seed_timeout': 100, u'eventserver_port': None}
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1335, in handle_heartbeat_response
self._handle_heartbeat_response(response)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1357, in _handle_heartbeat_response
response["flood_torrent_port"])
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1823, in handle_heartbeat_flood
self.mkabsdir(flood_dir, user=FLOOD_FS_USER, group=FLOOD_FS_GROUP, mode=0755)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1918, in mkabsdir
path_grp = grp.getgrgid(stat_info.st_gid)[0]
KeyError: 'getgrgid(): gid not found: 167' I find nothing which can help me for this error. In the logs details, we can see this line: host_network_interface_collection_filter', u'config_value': u'^lo$' But not fir sure it's the mistake. In the Cloudera wizard, I give for each database the good remote host name ans mysql information. So I don't know it the pattern ^lo$ is the problem. I check the /var/log/{oozie,hive} log directory, but nohting. No log. And I'm on the machine where I try to install the Hive and Oozie service. Maybe it can give you an idea from the origin of the error ! Thanks again bgooley
... View more
08-05-2016
02:01 PM
Not again... I remove again the node and add it again. Now they appears in the "Hosts" list and with the "CDH5" version. But if I try to create a new cluster, select all node, affect service to each node, then I have to give database informations for Hive, Oozie and Activity Monitor. All information are ok, but when I push "Test connection", only the "Activity Monitor" works... I download again manually the mysql-connector-java.jar and put in the /ush/share/java, create a symlink in the /var/lib/hive/ and /var/lib/oozie folder, but nothing... The database connection for these 2 databse stay under connection and never terminate... Any help for that ? Always, if I install mysql-client and try connection to my databases with same informations, it works fine.
... View more
08-05-2016
01:35 PM
Not for sure... I'm lost... I don't know why my node can't connect to the database. Only the Cloudera Manager nodeconnect with success to database. So I try to remove all cluster and node against, and follow the procedure to clean all node, execpt the Cloudera Manager. Next I select "Host" and "Add new hosts to cluster". I give all range of my new node, and eveyrthing fine. The wizard, install jdk, cloudera manager agent, and cdh package with success. Now they appears in the le the hosts list, aith the Cloudera Manager but I get this: As we see, all node have a non-detetced version of CDH. The third, is the Cloudera Manager. If we check package, all have the 5.7.1 version. Then, if I would like to create a new cluster, I all of my hosts in the "Currently Managed hosts". Then I select Cloudera Express, then "Use package" . And in the next section: Detecting CDH versions on all hosts
All hosts have the same CDH version.
The following host(s) are running CDH 5.7.1: clouderamanagernode.domain.ltd So it only see the Cloudera Manager node and not the others... I search again but I don't find why... The installation is done with success of all package in our node. We have the same 5.7..1 version on each one. So why it can find the CDH version ?? Maybe if we find, we can do the services configuration after with success. Hope somebody can help. Have a good week-end !
... View more
08-05-2016
12:01 PM
Problem solved ! I remove the "Cloudera Manager Management" cluster, and the "Cluster 1". I decommission and remove all node and then, add it again with the wizard. I continue to have this message in the log but now the wizard continue and I can go to next step. But now I encount another problem, maybe always in relation with this one... I configure my differents services on each node, and click next. I'm now on the "Database configuration" page. I have to configure the "Hive" "Activity Monitor" and "Oozie" database parameters. So i give all informations, but for Hive and Oozie, it can't connect. It stay on test connection, and nothing append. I try to install mysql client on different host where I configure Hive and Oozie, and the connection works fine. So I add the mysql-connector-java.jat in the /usr/share/java, then create a symlink in the /var/lib/oozie folder, and in the other host, in the /var/lib/hive/. But still the same state... I launch a tcpdump on my MySQL server, but nothing. No connection try to be establish betweeen my "oozie server" node and my MySQL server... I try to restart my node but still the same. Si it works with mysql client, but not with the mysql-connector-java.jar file. Do I forget something about that ? FYI, I install the 5.1.39 version. I install it by the "yum" command, but see in the documentation that it wasn't recommanded, so I remove it and download manually the jar file. Hope somebody can help me !
... View more
08-04-2016
01:45 PM
More informations. From the CLoundera Manager, I launch a new "Inspector hosts". For my Cloudera Manager, it's ok. But for my other node, I got this error: Command aborted because of exception: Command timed-out after 150 seconds But here , I can click on "Full log file" And I got a new page with his error: HTTP ERROR 403
Problem accessing /cmf/process/81/logs. Reason:
http://mynode.domain.ltd:9000/process/81-host-inspector/files/logs/stderr.log
The server declined access to the page or resource. I'm going to check if I can find something about that, but maybe encount this error before ...
... View more
08-04-2016
11:24 AM
I try to remove the "CLoudera Manager Management" service. I launch with the wizard a new instllation of "CLoudera Manager Management" service. During installation it ask me again only about the "Activity Monitor" Not ask about the "Service Monitor" and "Host Monitor". So my database stay empty... We have a small lab, running with the 5.5.2 version, and when I install it, it ask for each service, not only the "Activity Monitor"... I still don't know why I'm not ask about these databases and still conitnue to have my new node in red with this error message... Nothing seems to be interesting in the Cloudera Manager log server.
... View more
08-04-2016
10:23 AM
A small feedback. During the installation. it ask me about the login/password, and database name for the "Activity monitor" but not for others services. I add these informations manually later in the Cloudera Manager for each services in the "Dtabases legacy mysql parameters". But if I go the to database, I see many tables in the "Activity Monitor" but zero table in others. So it don't populate the database... How can I do that ? Maybe my problem come from here... If somebody have an idea ! I take it !
... View more
08-03-2016
02:02 PM
Hi all ! Its now time to create replace our Cloudera 4 by Cloudera 5 ! I start the first,as install new hardware ! Good thing done ! Now I start the application installation/configuration. FYI, I will install Cloudera 5.7.1. I have a machine for the Cloudera Manager, one for the a MySQL server with "monitor/metastore". I install the Manager with the MySQL information so fr the "scm" database. Everything fine. Then I intregrate the Manager machine to the a new cluster, configure the Host Monitor to access to the monitor database. It works fine, now all Cloudera Manager Service works fine. I think ! Now I would like to integrate 4 new node to cluster. 2 will serve for "HA" name node, ans 2 others as datanode. Six others nodes will join the cluster later. So with Cloudera Manager, I launch the wizard to add the new hosts. Answer to all question, and the installation start. Everything fine and installation is done with success. But after that, it have to run the "inspector job" on all hosts. And here I encount this warning: newnode.domain.ltd: Command aborted because of exception: Command timed-out after 150 seconds
4 hosts are reporting with NONE CDH version
There are mismatched versions across the system, which will cause failures. See below for details on which hosts are running what versions of components. I use a local reposync, and with sync only the 5.7.1 version. So no problems with differents version, 5.7.1 is installed everywhere. Then, if I go to the Manager interface, I can see all hosts int the "Hosts" section. But all are in red status. If I select one, I have this message: This host is in contact with the Cloudera Manager Server. This host is not in contact with the Host Monitor. So the Cloudera Manager Agent seems to be ok, but it seems to can't contact the "Host Monitor"... But the "Host Monitor" is install and configured on the same server. So... And the status of ths "Host Monitor" is green, and can contact my remote MySQL database. Si I don't know why I get this error. No firewall between my machine, no Selinux. I don't know why they can contact the "Cloudera Manager" but not the "Host Monitor" The /etc/cloudera-scm-agent/config.ini config file is ok. Good IP and port. server_host=10.x.x.x
server_port=7182 On cloudera Manager, the "cloudera-scm-server" and "cloudera-scm-agent" run nicely. On my new node, "cloudera-scm-agent" run nicely too. But I get this error in log. But not sure it could be the reason, and if it is, I don't know how to solve it. [03/Aug/2016 20:56:32 +0000] 158533 MainThread agent INFO Flood daemon (re)start attempt
[03/Aug/2016 20:56:32 +0000] 158533 MainThread agent ERROR Failed to handle Heartbeat Response: {u'firehoses': [{u'roletype': u'ACTIVITYMONITOR', u'rolename': u'mgmt-ACTIVITYMONITOR-728c1b31088c1d8ddc2547d70b884cf7', u'port': 9999, u'report_interval': 60, u'address': u'clouderamanager.domain.ltd'}, {u'roletype': u'SERVICEMONITOR', u'rolename': u'mgmt-SERVICEMONITOR-728c1b31088c1d8ddc2547d70b884cf7', u'port': 9997, u'report_interval': 60, u'address': u'clouderamanager.domain.ltd'}, {u'roletype': u'HOSTMONITOR', u'rolename': u'mgmt-HOSTMONITOR-728c1b31088c1d8ddc2547d70b884cf7', u'port': 9995, u'report_interval': 60, u'address': u'clouderamanager.domain.ltd'}], u'rm_enabled': False, u'client_configs': [], u'create_parcel_symlinks': True, u'server_managed_parcels': [], u'extra_configs': None, u'host_collection_config_data': [{u'config_name': u'host_network_interface_collection_filter', u'config_value': u'^lo$'}, {u'config_name': u'host_disk_collection_filter', u'config_value': u'^$'}, {u'config_name': u'host_fs_collection_filter', u'config_value': u'^$'}, {u'config_name': u'host_log_tailing_config', u'config_value': u'{}\n'}, {u'config_name': u'host_dns_resolution_duration_thresholds', u'config_value': u'{"critical":"never","warning":"1000.0"}'}, {u'config_name': u'host_dns_resolution_enabled', u'config_value': u'true'}, {u'config_name': u'host_clock_offset_thresholds', u'config_value': u'{"critical":"10000.0","warning":"3000.0"}'}], u'apply_parcel_users_groups_permissions': True, u'flood_torrent_port': 7191, u'log_tailing_config': u'{}\n', u'active_parcels': {}, u'flood_rack_peers': [u'10.2.0.33:7191', u'10.2.0.31:7191', u'10.2.0.34:7191', u'10.2.0.29:7191', u'10.2.0.30:7191'], u'retain_parcels_in_cache': True, u'processes': [{u'status_links': {}, u'name': u'cluster-host-inspector', u'config_generation': 0, u'configuration_data': 'PK\x03\x04\x14\x00\x08\x08\x08\x00\x83\x90\x03I\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\n\x00\x00\x00input.json\xb5\xd2\xbb\x0e\xc2 \x18\x05\xe0\xbdOA\x98[\x02\xbd\x98\xe8\xd6\xe8\xd0\xc5\xd4\xb8\x1a\x07\x14\x92\x12)4\xa5\x9d\x9a\xbe\xbb\x80q\x04\xbb8r\xfe\xc3\x07\t,\t\x00\x90J\xd9h3\x19\x08\x0e\xe0\x06\x16\x1b\xd9\xb0\xb3\x89\xa2=w!4Jd\x1deZ\x0f\x19\xc69\x923\x13\x14\xd9Y\xfa\xe9\n\xe6Z\xd5w5\xd4\x8c\x8d\xdcx\x0f\x12\x8cr\x84Q\x81\xa1\x9d\xae\xe9o~\x17\xe0\xf3(_n\xe5I\x80/c|\xbe\xdf\xcaW\x01\xbe\x88\xde\xbe\xd8\xca\x17\x01\x9eDy\xe2ypw%(\x94\x19\xf8s\x12Z\xf9\x92\x9a\xa5\xf4\xf9\x8b\x8f\x0f>js\xe5T\xf6~{S\x9f\xda\xf6\x82\x8e\xed\xd9\x1f\x06\xa7N\x18\xf7Q\xdc\xf0\xaf\xcf\x98\xacoPK\x07\x089m\\\xdd\xbe\x00\x00\x00\x98\x02\x00\x00PK\x01\x02\x14\x00\x14\x00\x08\x08\x08\x00\x83\x90\x03I9m\\\xdd\xbe\x00\x00\x00\x98\x02\x00\x00\n\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00input.jsonPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x008\x00\x00\x00\xf6\x00\x00\x00\x00\x00', u'refresh_files': [], u'user': u'root', u'parcels': {}, u'auto_restart': False, u'run_generation': 2, u'extra_groups': [], u'environment': {}, u'optional_tags': [], u'running': False, u'program': u'mgmt/mgmt.sh', u'required_tags': [], u'arguments': [u'inspector', u'input.json', u'output.json', u'DEFAULT'], u'special_file_info': [], u'group': u'root', u'id': 34, u'resources': [], u'one_off': True}], u'server_manages_parcels': True, u'heartbeat_interval': 15, u'parcels_directory': u'/opt/cloudera/parcels', u'host_id': u'8a18c6eb-7f32-4e90-a6f9-88d1feeacd21', u'eventserver_host': u'clouderamanager.domain.ltd', u'enabled_metric_reporters': [u'ACCUMULO16', u'ACCUMULO16', u'KEYTRUSTEE-KMS_KEYTRUSTEE', u'KMS_KEYTRUSTEE', u'SPARK_ON_YARN-SPARK_YARN_HISTORY_SERVER', u'SPARK_YARN_HISTORY_SERVER', u'SOLR-SOLR_SERVER', u'SOLR_SERVER', u'HBASE-HBASERESTSERVER', u'HBASERESTSERVER', u'HOST', u'KEYTRUSTEE_SERVER-KEYTRUSTEE_PASSIVE_SERVER', u'KEYTRUSTEE_PASSIVE_SERVER', u'IMPALA-STATESTORE', u'STATESTORE', u'SPARK', u'SPARK', u'HBASE', u'HBASE', u'ACCUMULO-ACCUMULO_TRACER', u'ACCUMULO_TRACER', u'HDFS-DATANODE', u'DATANODE', u'ACCUMULO-ACCUMULO_MASTER', u'ACCUMULO_MASTER', u'YARN-RESOURCEMANAGER', u'RESOURCEMANAGER', u'HUE-HUE_SERVER', u'HUE_SERVER', u'ACCUMULO-ACCUMULO_MONITOR', u'ACCUMULO_MONITOR', u'MGMT-EVENTSERVER', u'EVENTSERVER', u'MGMT-NAVIGATORMETASERVER', u'NAVIGATORMETASERVER', u'HBASE-MASTER', u'MASTER', u'KAFKA-KAFKA_BROKER', u'KAFKA_BROKER', u'KEYTRUSTEE_SERVER-DB_PASSIVE', u'DB_PASSIVE', u'HBASE-REGIONSERVER', u'REGIONSERVER', u'SPARK_ON_YARN', u'SPARK_ON_YARN', u'MGMT-REPORTSMANAGER', u'REPORTSMANAGER', u'MGMT-SERVICEMONITOR', u'SERVICEMONITOR', u'IMPALA-IMPALAD', u'IMPALAD', u'MGMT-ALERTPUBLISHER', u'ALERTPUBLISHER', u'HIVE-HIVESERVER2', u'HIVESERVER2', u'MGMT-ACTIVITYMONITOR', u'ACTIVITYMONITOR', u'ISILON', u'ISILON', u'YARN-NODEMANAGER', u'NODEMANAGER', u'MAPREDUCE-FAILOVERCONTROLLER', u'FAILOVERCONTROLLER', u'ACCUMULO', u'ACCUMULO', u'MAPREDUCE', u'MAPREDUCE', u'ZOOKEEPER', u'ZOOKEEPER', u'KMS', u'KMS', u'ACCUMULO16-ACCUMULO16_TRACER', u'ACCUMULO16_TRACER', u'ACCUMULO16-ACCUMULO16_MONITOR', u'ACCUMULO16_MONITOR', u'MGMT-HOSTMONITOR', u'HOSTMONITOR', u'YARN-JOBHISTORY', u'JOBHISTORY', u'KEYTRUSTEE', u'KEYTRUSTEE', u'HDFS-JOURNALNODE', u'JOURNALNODE', u'KAFKA', u'KAFKA', u'IMPALA', u'IMPALA', u'SPARK-SPARK_HISTORY_SERVER', u'SPARK_HISTORY_SERVER', u'KEYTRUSTEE_SERVER-KEYTRUSTEE_ACTIVE_SERVER', u'KEYTRUSTEE_ACTIVE_SERVER', u'HDFS-NAMENODE', u'NAMENODE', u'HUE-BEESWAX_SERVER', u'BEESWAX_SERVER', u'SOLR', u'SOLR', u'ACCUMULO16-ACCUMULO16_TSERVER', u'ACCUMULO16_TSERVER', u'MAPREDUCE-TASKTRACKER', u'TASKTRACKER', u'IMPALA-CATALOGSERVER', u'CATALOGSERVER', u'HDFS-DSSDDATANODE', u'DSSDDATANODE', u'SENTRY', u'SENTRY', u'ACCUMULO16-ACCUMULO16_GC', u'ACCUMULO16_GC', u'MGMT-NAVIGATOR', u'NAVIGATOR', u'HIVE', u'HIVE', u'HBASE-HBASETHRIFTSERVER', u'HBASETHRIFTSERVER', u'SQOOP-SQOOP_SERVER', u'SQOOP_SERVER', u'KAFKA-KAFKA_MIRROR_MAKER', u'KAFKA_MIRROR_MAKER', u'FLUME', u'FLUME', u'HUE', u'HUE', u'HDFS-SECONDARYNAMENODE', u'SECONDARYNAMENODE', u'SENTRY-SENTRY_SERVER', u'SENTRY_SERVER', u'ACCUMULO-ACCUMULO_TSERVER', u'ACCUMULO_TSERVER', u'ACCUMULO-ACCUMULO_GC', u'ACCUMULO_GC', u'HIVE-HIVEMETASTORE', u'HIVEMETASTORE', u'IMPALA-LLAMA', u'LLAMA', u'ACCUMULO16-ACCUMULO16_MASTER', u'ACCUMULO16_MASTER', u'SPARK-SPARK_WORKER', u'SPARK_WORKER', u'MGMT', u'MGMT', u'HIVE-WEBHCAT', u'WEBHCAT', u'SQOOP', u'SQOOP', u'HUE-HUE_LOAD_BALANCER', u'HUE_LOAD_BALANCER', u'ACCUMULO-ACCUMULO_LOGGER', u'ACCUMULO_LOGGER', u'HDFS', u'HDFS', u'FLUME-AGENT', u'AGENT', u'OOZIE', u'OOZIE', u'SQOOP_CLIENT', u'SQOOP_CLIENT', u'OOZIE-OOZIE_SERVER', u'OOZIE_SERVER', u'KMS-KMS', u'KMS', u'HDFS-FAILOVERCONTROLLER', u'FAILOVERCONTROLLER', u'KS_INDEXER', u'KS_INDEXER', u'SPARK-SPARK_MASTER', u'SPARK_MASTER', u'YARN', u'YARN', u'ZOOKEEPER-SERVER', u'SERVER', u'HDFS-NFSGATEWAY', u'NFSGATEWAY', u'HDFS-HTTPFS', u'HTTPFS', u'HUE-KT_RENEWER', u'KT_RENEWER', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER-DB_ACTIVE', u'DB_ACTIVE', u'MAPREDUCE-JOBTRACKER', u'JOBTRACKER', u'KS_INDEXER-HBASE_INDEXER', u'HBASE_INDEXER'], u'flood_seed_timeout': 100, u'eventserver_port': 7185}
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1335, in handle_heartbeat_response
self._handle_heartbeat_response(response)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1357, in _handle_heartbeat_response
response["flood_torrent_port"])
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1823, in handle_heartbeat_flood
self.mkabsdir(flood_dir, user=FLOOD_FS_USER, group=FLOOD_FS_GROUP, mode=0755)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 1918, in mkabsdir
path_grp = grp.getgrgid(stat_info.st_gid)[0]
KeyError: 'getgrgid(): gid not found: 167 Do you have an idea why this error ? Hope somebody can help me to have a good configuration and my node are green. Regards, Fabien
... View more
04-06-2016
03:29 PM
Hi tseader, Sorry I wasn't avaiable ! For update, It works. The problem was the "Dynamic ressrouce pool". I create a resource pool for my username, and now the job is starting and runing. It was different from our Cloudera 4 in how it works... So now the job is runing, doing the sqoop and the hive job, and terminate successfuly ! Great news! But it very slow for a small table import, I think there is something to do in Dynamic resource pool or yarn setting to use more resource cause, during the job, cpu/emory of my 2 datanode was very less... Maybe you can give me some informations on how to calculate the the max container possible ? To give you some answer: - Yes sqoop was working alone. - Yes our analytics use <args> cause sometime in CDH4 with <command>, they were some error with specific caracters. - Now yes, sqoop/oozie/hive works now. We will try Impala now - No we doesn't try to create a workflow since Hue. I will see with our dev about that. - Not, didn't try with another db. As you thinking, the problem wasn't come from the workflow but the configuration. I'm new in Cloudera/Hadoop, so I learn! I discover the configuration with time! Now I've to find the best configuration to a better usage of our datanode... Thanks again tseader!
... View more
04-01-2016
04:47 PM
Yes we always used the real fqdn to start job. And we find the mistake. The "job_tracker" was on "name_node" job and vice-versa... So we change them to be valid, and the process start but it doesn't complish anymore. We see the worklows in "running" status, and see a job "oozie:launcher" on running state. It create an "oozie:action" task, but this last stay on the "accepted" status. And I don't find why. I try some settings in Yarn memory configuration with no success. In RessourceManager, I can find this log about he job: >>> Invoking Sqoop command line now >>>
4624 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
4654 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.5.2
4671 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.BaseSqoopTool - Setting your password on the command-line is insecure. Consider using -P instead.
4672 [uber-SubtaskRunner] INFO org.apache.sqoop.tool.BaseSqoopTool - Using Hive-specific delimiters for output. You can override
4672 [uber-SubtaskRunner] INFO org.apache.sqoop.tool.BaseSqoopTool - delimiters with --fields-terminated-by, etc.
4690 [uber-SubtaskRunner] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
4816 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.MySQLManager - Preparing to use a MySQL streaming resultset.
4820 [uber-SubtaskRunner] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code generation
5360 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `table` AS t LIMIT 1
5521 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `table` AS t LIMIT 1
5616 [uber-SubtaskRunner] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
7274 [uber-SubtaskRunner] INFO org.apache.sqoop.orm.CompilationManager - Writing jar file: /tmp/sqoop-yarn/compile/f695dd68db2ed1ecf703a5405d308df5/table.jar
7282 [uber-SubtaskRunner] WARN org.apache.sqoop.manager.MySQLManager - It looks like you are importing from mysql.
7282 [uber-SubtaskRunner] WARN org.apache.sqoop.manager.MySQLManager - This transfer can be faster! Use the --direct
7282 [uber-SubtaskRunner] WARN org.apache.sqoop.manager.MySQLManager - option to exercise a MySQL-specific fast path.
7282 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.MySQLManager - Setting zero DATETIME behavior to convertToNull (mysql)
7284 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of game_session
7398 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
8187 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation
8211 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat - BoundingValsQuery: SELECT MIN(`session_id`), MAX(`session_id`) FROM `table`
8237 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.IntegerSplitter - Split size: 9811415567004; Num splits: 4 from: 14556292800030657 to: 14595538462298675
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat The status is looping on "Heart beat" logs. I don't know if it can come from memory configuration, or anything else... Have you an idea about that ?
... View more
03-31-2016
04:23 PM
Hi tseader ! Thanks for your help ! Good eyes! Yes I think it could be a error ans stop the process. I modify these 2 settings but the problem still present. It seems to be before. It seems to read this file, but never start the mysql connection process. I have something else in oozie logs. When I launch the command on my VM, the workflow appears in Hue. But the log start with theses 2 lines: 2016-03-31 19:04:18,709 WARN org.apache.oozie.util.ParameterVerifier: SERVER[hostname] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2016-03-31 19:04:18,744 WARN org.apache.oozie.service.LiteWorkflowAppService: SERVER[hostname] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://hostname.domain.com:8020/path/to/oozie/lib] does not exist Or its not the libpath that I give in th my job file... The complete log, since the job is started to the end. 2016-03-31 19:04:18,709 WARN org.apache.oozie.util.ParameterVerifier: SERVER[hostname] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2016-03-31 19:04:18,744 WARN org.apache.oozie.service.LiteWorkflowAppService: SERVER[hostname] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://hostname.domain.com:8020/path/to/oozie/lib] does not exist
2016-03-31 19:04:18,805 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@:start:] Start action [0000001-160331185825562-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-31 19:04:18,809 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@:start:] [***0000001-160331185825562-oozie-oozi-W@:start:***]Action status=DONE
2016-03-31 19:04:18,809 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@:start:] [***0000001-160331185825562-oozie-oozi-W@:start:***]Action updated in DB!
2016-03-31 19:04:18,898 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@extract] Start action [0000001-160331185825562-oozie-oozi-W@extract] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-31 19:04:18,907 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@extract] [***0000001-160331185825562-oozie-oozi-W@extract***]Action
2016-03-31 19:04:18,907 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@extract] [***0000001-160331185825562-oozie-oozi-W@extract***]Action updated in DB!
2016-03-31 19:04:19,077 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@session] Start action [0000001-160331185825562-oozie-oozi-W@session] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-31 19:04:22,804 WARN org.apache.hadoop.security.UserGroupInformation: SERVER[hostname] PriviledgedActionException as:username (auth:PROXY) via oozie (auth:SIMPLE) cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: httpstatus=DONE
2016-03-31 19:04:22,805 WARN org.apache.hadoop.security.UserGroupInformation: SERVER[hostname] PriviledgedActionException as:username (auth:PROXY) via oozie (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
2016-03-31 19:04:22,805 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostname] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000001-160331185825562-oozie-oozi-W] ACTION[0000001-160331185825562-oozie-oozi-W@session] Error starting action [session]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:454)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:434)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1032)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:450)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:436)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:434)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1246)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:980) Maybe it can give you an idea ! I check hostname in configuration file, but it seems to be ok. This error message is not very clear...
... View more
03-29-2016
01:38 PM
Hello everyone, I come with an error from one of our job which is not really explicit... I find another topic about the same error but seems to haven't the same origin. To reproduce the problem I start from my VM this oozie job (we have a standalone labs in Cloudera 5.5.2 remote server) The command to start the job: oozie job -oozie http://host.domain.com:11000/oozie -config config-default.xml -run The content of config-default.xml file: <configuration>
<property><name>job_tracker</name><value>host.domain.com:8032</value></property>
<property><name>job_xml</name><value>/path/to/file/hive-site.xml</value></property>
<property><name>name_node</name><value>hdfs://host.domain.com:8020</value></property>
<property><name>oozie.libpath</name><value>${name_node}/user/oozie/share/lib/lib_20160216173849</value></property>
<property><name>oozie.use.system.libpath</name><value>true</value></property>
<property><name>oozie.wf.application.path</name><value>${name_node}/path/to/file/simple-etl-wf.xml</value></property>
<property><name>db_user</name><value>user</value></property>
<property><name>db_pass</name><value>password</value></property>
<property><name>target_dir</name><value>/path/to/destination</value></property>
<property><name>hive_db_schema</name><value>default</value></property>
<property><name>table_suffix</name><value>specific_suffix</value></property>
</configuration> Try to set the "job"tracker" with http:// but we have the same error. The content of the simple-etl-wf.xml file <workflow-app xmlns="uri:oozie:workflow:0.5" name="simple-etl-wf">
<global>
<job-tracker>${name_node}</job-tracker>
<name-node>${job_tracker}</name-node>
<job-xml>${job_xml}</job-xml>
</global>
<start to="extract"/>
<fork name="extract">
<path start="table" />
</fork>
<action name="table">
<sqoop xmlns="uri:oozie:sqoop-action:0.4">
<arg>import</arg>
<arg>--connect</arg>
<arg>jdbc:mysql://db.domain.com/database</arg>
<arg>username</arg>
<arg>${db_user}</arg>
<arg>password</arg>
<arg>${db_pass}</arg>
<arg>--table</arg>
<arg>table</arg>
<arg>--target-dir</arg>
<arg>${target_dir}/table</arg>
<arg>--split-by</arg>
<arg>column</arg>
<arg>--hive-import</arg>
<arg>--hive-overwrite</arg>
<arg>--hive-table</arg>
<arg>${hive_db_schema}.table_${table_suffix}</arg>
</sqoop>
<ok to="join"/>
<error to="fail"/>
</action>
<join name="join" to="transform" />
<action name="transform">
<hive xmlns="uri:oozie:hive-action:0.4">
<script>script.hql</script>
<param>hive_db_schema=${hive_db_schema}</param>
<param>table_suffix=${table_suffix}</param>
</hive>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app> The job is start, but it block to 20% about. And we have this error: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. 2016-03-29 15:45:17,149 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[host.domain.com] USER[username] GROUP[-] TOKEN[] APP[simple-etl-wf] JOB[0000004-160325161246127-oozie-oozi-W] ACTION[0000004-160325161246127-oozie-oozi-W@session] Error starting action [session]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:454)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:434)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1032)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:450)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:436)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:434)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1246)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:980)
... 10 more Or the job_tracker and name_node have the good url and path. The mysql-connector-java.jar is present in the sharlib folder. I put oozie un debug mode but no more information about that. The "mapreduce.framework.name" is set to "yarn on each xml configuration file the the cluster." Have you any idea about this error ?
... View more
Labels:
- Labels:
-
Oozie
02-25-2016
07:25 AM
5 Kudos
It works! As we see in the outpulog, we see the HADOOP_CLASSPATH variable. Or we don't have any path for libs in hive directory... I try once to add in HADOOP_CLASSPATH the his folder but it doesn't works. The solution is to add the the folder and /* to take all jar... So I add this one in .bash_profile: export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/lib/hive/lib/* Then source ~/.bash_profile And now it works. Date were imported in Hive! Now we can continue our labs with Cloudera 5! Thanks!
... View more
02-22-2016
02:57 PM
On my client machine, I try to find the class and find it in a jar file. I go to /usr/lib/hive/lib folder, and look inside the hive-common.jar with this command: jar tf hive-common.jar At the end, I can see this line: org/apache/hadoop/hive/conf/HiveConf.class So the class is present. So why he can't find it when it start the import ? The HIVE_HOME is set to /usr/lib/hive, so, the path is valid... I continue to search, but maybe it can give you more informations why and how to solve that !
... View more
02-22-2016
07:14 AM
Hi Matt, Thanks. But my problem still present... Maybe someone else with a fresh install of Cloudera Manager/CDH 5.5 can have the same problem. For test, I try a fresh install on an another single machine. Same error ! So maybe the problem came from the client configuration. To do the installation I use our Cloudera Manager/CDH repository which are sync every day. So I use the package and the not the parcels during installation. My test VM are on CentOS 6.6. A supported version. To start the command, I launch it since my machine (Ubuntu) I install these services to work: sudo apt-get install hadoop-client hive oozie-client sqoop I add these variable in my ".bash_profile" export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/share/java/slf4j-simple.jar
export HIVE_HOME=/usr/lib/hive
export PATH=$PATH:$HIVE_HOME/bin Then I do an "scp" to recover the "/etc/sqoop", "/etc/hive", "/etc/hadoop/", The configuration seems to be ok so. If not, the command can't start. I tried to add the the HIVE_CONF_DIR variable in differents file: - sqoop-env.sh - hadoop-env.sh - hive-env.sh Without any success. The process starts, but the error still present. Hope somebody can help me !
... View more
