Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Oozie Sqoop action throwing java.io.IOException: No columns to generate for ClassWriter

avatar
Expert Contributor

I'm trying to test Oozie's Sqoop action in the following environment:

  • HDP2.3.2
  • Sqoop 1.4.6
  • Oozie 4.2.0

Via the command line, the following sqoop command works:

sqoop import \
	-D mapred.task.timeout=0 \
	--connect jdbc:sqlserver://x.x.x.x:1433;database=CEMHistorical \
 	--table MsgCallArrival \
	--username hadoop \
	--password-file hdfs:///user/sqoop/.adg.password \
	--hive-import \
	--create-hive-table \
	--hive-table develop.oozie \
	--split-by TimeStamp \
	--map-column-hive Call_ID=STRING,Stream_ID=STRING

But when I try to execute the same command via Oozie, I'm running into java.io.IOException: No columns to generate for ClassWriter

Below are my `job.properties` and `workflow.xml`:

nameNode=hdfs://host.vitro.com:8020
jobTracker=host.vitro.com:8050
projectRoot=${nameNode}/user/${user.name}/tmp/sqoop-test/
oozie.use.system.libpath=true
oozie.wf.application.path=${projectRoot}



<workflow-app name="sqoop-test-wf" xmlns="uri:oozie:workflow:0.4">
    <start to="sqoop-import"/>

    <action name="sqoop-import" retry-max="10" retry-interval="1">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
        <command>import -D mapred.task.timeout=0 --connect jdbc:sqlserver://x.x.x.x:1433;database=CEMHistorical --table MsgCallArrival --username hadoop --password-file hdfs:///user/sqoop/.adg.password --hive-import --create-hive-table --hive-table develop.oozie --split-by TimeStamp --map-column-hive Call_ID=STRING,Stream_ID=STRING</command>
        </sqoop>
        <ok to="end"/>
        <error to="errorcleanup"/>
    </action>
    <kill name="errorcleanup">
      <message>Sqoop Test WF failed. [${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name ="end"/>
</workflow-app>

I've attached the full log, but here's an excerpt:

2016-01-05 11:29:21,415 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
    at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
    at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

I've been struggling with this problem for quite some time now, any help would be greatly appreciated!

1 ACCEPTED SOLUTION

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
11 REPLIES 11

avatar

This error "No columns to generate forClassWriter" is thrown when no results are retrieved by the command.

So there is something wrong with the command. It appears that it could be the jdbc url.

As per MSDN, the correct syntax for named database uses the term "databaseName" and not "database" (like you have it in the snippet above). So try changing that.

Connect to a named database on a remote server:

jdbc:sqlserver://localhost;databaseName=AdventureWorks;integratedSecurity=true;

avatar
Expert Contributor

I tried your suggestion but I'm still getting the same result. In any case, I don't think it could have been the connection URL because as I mentioned above, the command works when used via command-line. It's just when I use it with Oozie that I encounter this error.

avatar
Master Mentor

@Luis Antonio Torres has this been resolved? Can you provide your solution or accept the best answer?

avatar
Expert Contributor

@Artem Ervits I never got sqoop's hive-import to work in an Oozie workflow, so I came up with a workaround instead. Will provide my workaround as an answer. Thanks.

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

As far as I see sqoop action above, I don't see hive-site.xml file. I guess you added it into lib directory in the deployment directory, which will keep Hive action from running and it gives you error something like hive-site.xml permssion error. You should add the hive-site.xml file in "Files" in the Sqoop action.

avatar
Expert Contributor

@Shigeru Takehara I did try adding hive-site.xml and placed it in the root of the workflow directory on HDFS, but I was running into the error and the error message in the logs is this:

ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: Hive exited with status 1

I eventually had to go with my workaround because I couldn't get hive import to work and I had deadlines to meet. I'd still like to try and get hive import to work though

avatar
Expert Contributor

Where did you copy your jdbc driver for Sqoop action?

avatar
Expert Contributor

Hi @Shigeru Takehara, the jdbc driver was in the Oozie sharelib. Also, the import was working fine - I see the data within HDFS. It's just the loading into Hive that fails. That's why I just opted to break the import into two separate steps.