Reply
New Contributor
Posts: 1
Registered: ‎01-20-2015

Re: Creating Sqoop 2 Database

We overcame the issue, Sqoop2 Service would not start after upgrade, by removing jars (from /var/lib/sqoop2 ) pulled from the Web/latest Apache Derby libs which were a MUST for Sqoop Upgrade then restarting Sqoop Service.

 

 

I ran into issue during upgrade: 5.1.0 to 5.3.0

Latest Apache Derby libs, derbyclient.jar and derby.jar which I pulled from the Web, as per earlier suggestion, took care of the upgrade bug.

But the service would not start. Not until we removed those manually placed for upgrade libraries from /var/lib/sqoop2. Once removed Sqoop service started up just fine. We had case open with Cloudera: 51398 in case you interested.

 

/var/log/sqoop2/derbyrepo.log: start up error

 

Mon Jan 26 13:06:59 PST 2015 Thread[main,5,main] Mon Jan 26 13:06:59 PST 2015 Thread[main,5,main] Startup failed due to missing functionality for org.apache.derby.jdbc.InternalDriver. Please ensure your classpath includes the correct Derby software.

ERROR XBM02: Startup failed due to missing functionality for org.apache.derby.jdbc.InternalDriver. Please ensure your classpath includes the correct Derby software.

                  at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)

                  at org.apache.derby.iapi.services.monitor.Monitor.missingImplementation(Unknown Source)

                  at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)

                  at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)

                  at org.apache.derby.impl.services.monitor.BaseMonitor.startServices(Unknown Source)

                  at org.apache.derby.impl.services.monitor.BaseMonitor.runWithState(Unknown Source)

                  at org.apache.derby.impl.services.monitor.FileMonitor.<init>(Unknown Source)

                  at org.apache.derby.iapi.services.monitor.Monitor.startMonitor(Unknown Source)

                  at org.apache.derby.iapi.jdbc.JDBCBoot.boot(Unknown Source)

                  at org.apache.derby.jdbc.EmbeddedDriver.boot(Unknown Source)

                  at org.apache.derby.jdbc.EmbeddedDriver.<clinit>(Unknown Source)

                  at java.lang.Class.forName0(Native Method)

                  at java.lang.Class.forName(Class.java:190)

                  at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:47)

                  at org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:122)

                  at org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:65)

                  at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:119)

                  at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:77)

                  at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53)

                  at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)

                  at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)

                  at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)

                  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)

                  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)

                  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)

                  at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)

                  at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)

                  at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)

 

var/log/sqoop2: upgrade error

 

2015-01-26 14:52:11,040 ERROR org.apache.sqoop.tools.tool.UpgradeTool: Can't finish upgrading RepositoryManager, Driver and ConnectionManager:

org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link

                  at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)

                  at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)

                  at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)

                  at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)

                  at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)

                  at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)

                  at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)

                  at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                  at java.lang.reflect.Method.invoke(Method.java:606)

                  at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                  at java.lang.reflect.Method.invoke(Method.java:606)

                  at org.apache.catalina.startup.Tool.main(Tool.java:225)

Caused by: java.sql.SQLException: No suitable driver found for jdbc:derby:/var/lib/sqoop2/repository/db;create=true

                  at java.sql.DriverManager.getConnection(DriverManager.java:596)

                  at java.sql.DriverManager.getConnection(DriverManager.java:187)

                  at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:51)

                  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)

                  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)

                  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)

                  at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)

                  ... 17 more

2015-01-26 14:52:11,043 INFO org.apache.sqoop.core.PropertiesConfigurationProvider: Shutting down configuration poller thread

Explorer
Posts: 21
Registered: ‎12-16-2014

Re: Creating Sqoop 2 Database

Hi Audi,

 

I am still have the same problem. I couldn't find any solution. Still waiting for someone to answer.

Explorer
Posts: 12
Registered: ‎02-04-2015

Re: Creating Sqoop 2 Database

Hi~Dkumar:

 

Explorer
Posts: 12
Registered: ‎02-04-2015

Re: Creating Sqoop 2 Database

Hi~Dkumar:

     I have fixed the problem.You can try my step.

1.Download db-derby-10.11.1.1-bin.zip.And install it.

2.Copy derby.jar into /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/

3.Delete /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby-{version}.jar soft link.

4.Make soft link /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar to /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/derby.jar

                                                                           Audi

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: Creating Sqoop 2 Database

Hey folks,

A few notes on getting Sqoop2 to work and give more detail if there is an error:
1. The Derby jar is required for Sqoop2 to work. It does come packaged with Sqoop2, so do not put it into /var/lib/sqoop2.
2. You can use "set option --name verbose --value true" to see more error information. You may be having different errors for all we know.
New Contributor
Posts: 4
Registered: ‎01-29-2015

Re: Creating Sqoop 2 Database

Hi Audi,
Thank you so much ! Great help~~~
New Contributor
Posts: 1
Registered: ‎02-19-2015

Re: Creating Sqoop 2 Database

Hi,

 

I am facing the exact same issue with Cloudera Manager V. 5.3.1

The folder structure does not exists - /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/ does not exist for my case to copy the derby.jar file.

 

Please guide.

 

Thanks !

DKay

Explorer
Posts: 12
Registered: ‎02-04-2015

Re: Creating Sqoop 2 Database

 Hi~Dkay:

      Did you install CDH with parcels?If yes, your CDH folder should be in /opt/cloudera/parcels.

Maybe it is /opt/cloudera/parcels/CDH-5.3.1.xxxx/jars.Good luck!

                                                       Audi

New Contributor
Posts: 1
Registered: ‎07-22-2015

Re: Creating Sqoop 2 Database

Thank you very much for the solution. It helped me to resolve same issue.

Cloudera Employee
Posts: 6
Registered: ‎06-25-2014

Re: Creating Sqoop 2 Database

Hello Audi et al,

 

I have seen the error below when the lock files in the Derby database directory are owned by a user other than Sqoop2.  This can happen if you manually connect to the database using the Derby "ij" tool.

 

Server startup failure
org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
	at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:64)
	<SNIP>
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:1Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

 

# ls -l
total 24
-rw-r--r-- 1 root root 4 Aug 10 11:28 dbex.lck <======
-rw-r--r-- 1 root root 38 Aug 10 11:28 db.lck <=======
drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 10:15 log
drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 10:15 seg0
-rw-r--r-- 1 sqoop2 sqoop2 853 Aug 10 10:15 service.properties
drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 13:49 tmp

 

To get past this issue do the following:

  1. Ensure that there are no connections to the database
  2. Address the lock files
    1. Option 1: chown sqoop2:sqoop2 *.lck
    2. Option 2: rm -rf *.lck
  3. Start Sqoop2

 

Hope this helps,

Markus Kemper - Cloudera Support

Announcements
New solutions