CDH 5.3 and Sqoop2 startup and use error

[#1]

Hello.

I had a CDH 5.1.3 installation where I was able to use the Sqoop2 client to import data into Hive and HBase. After upgrading to CDH 5.3 I am unable to get neither the Sqoop2 service to start successfully, nor am I able to use the Sqoop2 client. What do I need to do to get this to work as it did before the upgrade?

The errors reported in /var/log/sqoop2/catalna log are as follows:

... ... ...

Jan 20, 2015 4:10:54 PM org.apache.tomcat.util.digester.Digester endElement
WARNING:   No rules found matching 'HTML/BODY'.
Jan 20, 2015 4:10:54 PM org.apache.tomcat.util.digester.Digester endElement
WARNING:   No rules found matching 'HTML'.
Jan 20, 2015 4:10:54 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 59 ms
Jan 20, 2015 4:10:54 PM org.apache.catalina.startup.Catalina start
SEVERE: Cannot start server. Server instance is not configured.

And, the Sqoop2 client throws an exception, and following is the output it displays:

15/01/20 19:58:13 FATAL conf.Configuration: error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
        at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2375)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2337)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2254)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:861)
        at org.apache.sqoop.tool.SqoopTool.loadPluginsFromConfDir(SqoopTool.java:170)
        at org.apache.sqoop.tool.SqoopTool.loadPlugins(SqoopTool.java:140)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:208)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2493)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2337)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2254)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:861)
        at org.apache.sqoop.tool.SqoopTool.loadPluginsFromConfDir(SqoopTool.java:170)
        at org.apache.sqoop.tool.SqoopTool.loadPlugins(SqoopTool.java:140)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:208)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
        at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2375)
        ... 8 more

[#2]

Thanks for your reply.

I reinstalled the CDH 5.3.0 via Cloudera Manager, and found log under:
/var/log/sqoop2/sqoop-cmf-sqoop-SQOOP_SERVER-slc05hwl.us.oracle.com.log.out

error saying:


2015-01-30 09:03:04,822 ERROR org.apache.sqoop.server.ServerInitializer: Sqoop server failed to start
java.lang.RuntimeException: Failure in server initialization

.....

Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
......

Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
.....
Caused by: java.sql.SQLException: No suitable driver found for jdbc:derby:/var/lib/sqoop2/repository/db;create=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)
... 31 more



And I added derby-10.11.1.1.jar to /var/lib/sqoop2/

then re-ran the Cluster Settings and error came out as you did mentioned


Do you have any idea about this?

Thank you very much

[#3]

Not exactly same but I had similar installation issue with Sqoop2. In my case I had to reinstall the OS and get past the issue. It was the most time saving solution for me after spending good amount of time trying to analyze the problem.

[#4]

well , that is lucky for you.

I re-imaged the server and issue still exists... no idea.


Any way, thank you ~~

[#5]

I've upgraded CDH from 5.2.1 to 5.3.2 and got the same errors as in the first message. Sqoop2 service could not be started.

After several hours of investigations I've found out that new sqoop2 is not compatible with xercesImpl.jar from libxerces2-java (Debian Wheezy). It cannot be removed because other packages depend on it.

The workaround is to remove /usr/share/java/*.jar from common.loader=... in /var/lib/sqoop2/tomcat-deployment/conf/catalina.properties.

[#6]

Many Thanks, it worked for me!

I followed Yugune's reply.

The workaround is to remove /usr/share/java/*.jar from common.loader=... in /var/lib/sqoop2/tomcat-deployment/conf/catalina.properties.

[#7]

Encountered this myself on Red Hat Enterprise Linux Server release 7.2 (Maipo). Seems to be a conflict with this Red Hat's release of xceres-j2.