Blank Sqoop page in Hue 2.5.0

[#1]

Hi,

I recently wanted to use the Sqoop UI in Hue to import data from mysql to HDFS, but after clicking the Sqoop2 icon I just got a blank page (except the heading "Sqoop Jobs"). Yes, of course, there aren't any jobs to display, but I expected the wizard to be displayed for creating jobs...?!?!

Can I use Sqoop2 UI in Hue 2.5 at all, or do I need Hue 3 ?

thanks in advance...Gerd...

============
env.:  Hue 2.5.0 / sqoop2 / CDH4.4 / cluster managed by CM 4.6.2

[#2]

Indeed, you should get a wizard like in http://gethue.tumblr.com/post/63064228790/move-data-in-out-your-hadoop-cluster-with-the-sqoop

Your Sqoop2 server is probably not up?

You could check the request from Hue to Sqoop2 by checking the logs on hue_server:8888/logs?

[#3]

Hi Romain,

thanks for answering, the page you referred to is great. It's exactly the page I looked at before and thought the Hue version could cause the problem, since this page mentions Hue3.

Nevertheless, I checked the logs, and......gotcha, here's the error =>

[24/Oct/2013 11:07:21 -0800] exception    ERROR    Could not connect to sqoop server: 111 (Connection refused)

[24/Oct/2013 11:07:21 -0800] http_client  DEBUG    GET http://localhost:12000/sqoop/v1/job/all

[24/Oct/2013 11:07:21 -0800] access       INFO     10.147.x.y admin - "GET /sqoop/api/jobs/ HTTP/1.0"

I'm wondering why Hue tries to connect to localhost ?!?! This will fail "by design" since sqoop is running on a different host.

How can I modifiy that setting ?
If I open "http://hadoop-pg-2:12000" I get the response, "Apache Sqoop ROOT", thereby I assume Sqoop is running correctly.

UPDATE 1:

but somehow wired, requesting "http://hadoop-pg-2:12000/sqoop/v1/job/all" returns a 404 "The requested resource () is not available."

UPDATE 2:

I'm somehow confused  . The CM manager services overview tells me that service "sqoop" is up and running. I dived into /var/log/sqoop2 folder on the sqoop-server "hadoop-pg-2" and the file "sqoop-cmf-sqoop1-SQOOP_SERVER-hadoop-pg-2.cluster.log.out" shows =>

2013-10-24 20:50:47,127 INFO org.apache.sqoop.core.SqoopServer: Booting up Sqoop server
2013-10-24 20:50:47,136 INFO org.apache.sqoop.core.PropertiesConfigurationProvider: Starting config file poller thread
2013-10-24 20:50:47,138 INFO org.apache.sqoop.audit.AuditLoggerManager: Begin audit logger manager initialization
2013-10-24 20:50:47,139 INFO org.apache.sqoop.audit.AuditLoggerManager: Audit logger manager initialized: OK
2013-10-24 20:50:47,141 INFO org.apache.sqoop.repository.RepositoryManager: Setting system properties: {derby.stream.error.file=/var/log/sqoop2/derbyrepo.log}
2013-10-24 20:50:47,153 INFO org.apache.sqoop.repository.JdbcRepositoryContext: [repo-ctx] handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler, conn-url=jdbc:derby:/var/lib/sqoop2/repository/db;create=true, driver=org.apache.derby.jdbc.EmbeddedDriver, user=null, password=*****, jdbc-props={}, tx-isolation=READ_COMMITTED, max-conn=10
2013-10-24 20:50:47,603 INFO org.apache.sqoop.repository.derby.DerbyRepositoryHandler: DerbyRepositoryHandler initialized.
2013-10-24 20:50:47,605 INFO org.apache.sqoop.repository.JdbcRepositoryProvider: JdbcRepositoryProvider initialized
2013-10-24 20:50:47,605 INFO org.apache.sqoop.repository.JdbcRepositoryProvider: JdbcRepository initialized.
2013-10-24 20:50:47,605 INFO org.apache.sqoop.repository.RepositoryManager: Creating or upgrading on disk structures if necessary
2013-10-24 20:50:48,008 INFO org.apache.sqoop.repository.JdbcRepository: Creating repository schema objects
2013-10-24 20:50:48,722 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit
2013-10-24 20:50:48,750 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit
2013-10-24 20:50:48,750 INFO org.apache.sqoop.repository.RepositoryManager: Repository initialized: OK
2013-10-24 20:50:48,753 INFO org.apache.sqoop.connector.ConnectorManager: Connector config urls: [jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/sqoop2/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2-cdh4.4.0.jar!/sqoopconnector.properties]
2013-10-24 20:50:48,832 INFO org.apache.sqoop.connector.ConnectorHandler: Connector [org.apache.sqoop.connector.jdbc.GenericJdbcConnector] initialized.
2013-10-24 20:50:48,926 WARN org.apache.sqoop.repository.JdbcRepositoryTransaction: Marking transaction for rollback
2013-10-24 20:50:48,927 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction roll-back
2013-10-24 20:50:48,928 ERROR org.apache.sqoop.core.SqoopServer: Server startup failure
org.apache.sqoop.common.SqoopException: CONN_0007:Connector registration failed
at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:234)
at org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:195)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:51)
at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0026:Upgrade required but not allowed - Connector: generic-jdbc-connector
at org.apache.sqoop.repository.JdbcRepository$3.doIt(JdbcRepository.java:176)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
at org.apache.sqoop.repository.JdbcRepository.registerConnector(JdbcRepository.java:156)
at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:214)
... 27 more
2013-10-24 20:50:48,935 INFO org.apache.sqoop.core.SqoopServer: Shutting down Sqoop server

Something happened to the "generic-jdbc-connector", where does it come from ? A apt-get update && apt-get upgrade didn't update/-grade anything.

[#4]

Hum, at least you probably need to replace the hostname by the real Sqoop2 hostname in the Hue ini:

localhost --> hadoop-pg-2

around: https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L494

[#5]

I set the property accordingly in both files 

/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/etc/hue/hue.ini

/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/share/hue/desktop/conf/hue.ini

and restarted Hue as well as Sqoop, but the problem still exists.

I assume the error (while restarting sqoop service)

"

Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0026:Upgrade required but not allowed - Connector: generic-jdbc-connector

...

2013-10-25 13:54:11,672 INFO org.apache.sqoop.core.SqoopServer: Shutting down Sqoop server
2013-10-25 13:54:11,675 TRACE org.apache.sqoop.framework.JobManager: Begin submission engine manager destroy

"

on the sqoop server is causing the troubles.

Any hints regarding this ?!?!

[#6]

Does somebody have any hints for how to get to a working Sqoop wizard ?

[#7]

Hey There,

If you look closely, Sqoop is trying to upgrade its metadata. There should be a way to allow sqoop to automatically upgrade its metadata in the sqoop.properties file. The two properties you are looking for are:

- org.apache.sqoop.connector.autoupgrade

- org.apache.sqoop.framework.autoupgrade

-Abe

[#8]

Hi Abe,

thanks for your reply. I added the following properties to the "Sqoop Server Configuration Safety Valve for sqoop.properties" (after stopping service sqoop first):

org.apache.sqoop.repository.schema.immutable=false
org.apache.sqoop.connector.autoupgrade=true
org.apache.sqoop.framework.autoupgrade=true

The sqoop.properties file on the related server contains the entries, so they applied correctly.

The error in the sqoop-log disappeared, but the main problem still exists => after clicking the Sqoop icon I receive a blank page (except heading "Sqoop Jobs", no wizard.

The metadata upgrade worked fine, and I cannot detect any error message in one of the three log files

• sqoop-cmf-sqoop1-SQOOP_SERVER-hadoop-pg-2.cluster.log.out
• catalina.2013-11-02.log
• derbyrepo.log

This is the part of the sqoop-cmf-... log file of upgrading the connectors =>

2013-11-02 20:08:21,241 INFO org.apache.sqoop.repository.Repository: Upgrading metadata for connector: generic-jdbc-connector
2013-11-02 20:08:21,242 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-begin: 3, rollback: false
2013-11-02 20:08:21,393 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-commit: 3, rollback: false
2013-11-02 20:08:21,393 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-close: 2, rollback: false
2013-11-02 20:08:21,394 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-begin: 3, rollback: false
2013-11-02 20:08:21,494 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-commit: 3, rollback: false
2013-11-02 20:08:21,494 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-close: 2, rollback: false
2013-11-02 20:08:21,494 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-begin: 3, rollback: false
2013-11-02 20:08:21,858 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-commit: 3, rollback: false
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-close: 2, rollback: false
2013-11-02 20:08:21,859 INFO org.apache.sqoop.repository.Repository: Metadata upgrade finished for connector: generic-jdbc-connector
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-commit: 2, rollback: false
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-close: 1, rollback: false
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.connector.ConnectorManager: Registered connector: connector-generic-jdbc-connector:1:org.apache.sqoop.connector.jdbc.GenericJdbcConnector, Connection: Forms: form-connection:12:[input-connection.jdbcDriver:40:STRING:128, input-connection.connectionString:41:STRING:128, input-connection.username:42:STRING:40, input-connection.password:43:STRING:40, input-connection.jdbcProperties:44:MAP]Job type: IMPORTForms: form-table:13:[input-table.schemaName:45:STRING:50, input-table.tableName:46:STRING:50, input-table.sql:47:STRING:2000, input-table.columns:48:STRING:50, input-table.partitionColumn:49:STRING:50, input-table.partitionColumnNull:50:BOOLEAN, input-table.boundaryQuery:51:STRING:50]Job type: EXPORTForms: form-table:14:[input-table.schemaName:52:STRING:50, input-table.tableName:53:STRING:2000, input-table.sql:54:STRING:50, input-table.columns:55:STRING:50]
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-commit: 1, rollback: false
2013-11-02 20:08:21,859 DEBUG org.apache.sqoop.repository.JdbcRepositoryTransaction: Tx count-close: 0, rollback: false
2013-11-02 20:08:21,859 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit
2013-11-02 20:08:21,866 INFO org.apache.sqoop.connector.ConnectorManager: Connectors loaded: {generic-jdbc-connector={generic-jdbc-connector:org.apache.sqoop.connector.jdbc.GenericJdbcConnector:jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/sqoop2/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2-cdh4.4.0.jar!/sqoopconnector.properties}}

...

Some further ideas / suggestions ? Do you need some additional logs ?

br...Gerd...

[#9 · Accepted solution]

Hi again,

problem is solved, yiieeah !

Additionally to Abe's hint I detected that the sqoop server parameter on the hue.ini entry was again missing. After adding this value back everything's fine and the sqoop wizard launches.

Thanks to all who participated in my troubles

[#10]

just out of curiosity I tried it in the Cloudera quickstart vm.

Downloaded the latest 4.4 vmware image, logged into Hue as user "Cloudera" and clicked on the Sqoop icon.

Same result => blank page, no wizard to create jobs etc...