Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CM update Sqoop Service Environment Safety Valve i

avatar
Explorer

Problem desc:

I need to update the 2 properties in sqoop, but I am not able do it through sqoop.propeties files as CM will overide it, Its need to be done in the  Sqoop Service Environment Safety Valve in CM,

 

 

But when I did it in the sqoop2 service, in the advanced  configuration,to setup key/value pair for  "Sqoop Service Environment Safety Valve", below is the format.

 

org.apache.sqoop.connector.autoupgrade=true

org.apache.sqoop.framework.autoupgrade=true

 

I get the below error,

Could not parse: Sqoop Service Environment Safety Valve : Could not parse parameter 'sqoop_env_safety_valve'. Was expecting: valid variable name. Input: connector.autoupgrade=true

 

If i just give it like this it works

autoupgrade=true

 

but I need the whole string. I need help, what is format?

 

 

Thanks

1 ACCEPTED SOLUTION

avatar
There are multiple safety valves for sqoop, and you are editing the wrong one. See my first reply.

You need to edit the one that will alter sqoop.properties, NOT the one that will alter the environment. Read the name and description carefully.

View solution in original post

13 REPLIES 13

avatar

Hi,

 

Why do you want to enable auto upgrade? Sqoop has upgrade commands that will do this for you. Auto upgrade may have other side effects that could cause problems down the line.

 

If you really want to make this change, you need to modify the safety valve for sqoop.properties, not the environment safety valve.

 

Thanks,

Darren

avatar
Explorer

Thanks dlo for reply,

I started doing this because when I start the sqoop service in CM, I get the below error. So some one on the forum suggested that turning to true will help that, 

 

Connector config urls: [jar:file:/hdp2014poc/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/sqoop2/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2-cdh4.6.0.jar!/sqoopconnector.properties]
Connector [org.apache.sqoop.connector.jdbc.GenericJdbcConnector] initialized.
Marking transaction for rollback
Attempting transaction roll-back
Server startup failure
org.apache.sqoop.common.SqoopException: CONN_0007:Connector registration failed
at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:234)
at org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:195)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:51)
at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0026:Upgrade required but not allowed - Connector: generic-jdbc-connector
at org.apache.sqoop.repository.JdbcRepository$3.doIt(JdbcRepository.java:176)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
at org.apache.sqoop.repository.JdbcRepository.registerConnector(JdbcRepository.java:156)
at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:214)
... 27 more
Shutting down Sqoop server

avatar

When you get that message, it generally means you should run the upgrade command. CM has made this one easy for you, so no need to figure out how to do the upgrade manually.

avatar
Explorer

Thanks again,

but how do I run this upgrade command in CM.

Any tips would be helpful.

avatar

Click on Sqoop, then in the Actions dropdown menu on the right, you'll see an option to Upgrade Sqoop. If it's greyed out, that's because you need to stop Sqoop first.

 

Thanks,

DArren

avatar
Explorer

I am using CDH 4.6, 

I don't see the upgrade in action

these all the choices I have start, stop,restart,rename,enter maintenance mode, create group.

 

Let me know If I am missing something.

 

avatar
It looks like the sqoop upgrade action is only available in CM for CDH 5 clusters. Sorry for the confusion.

I'm not sure what triggered the need for a sqoop upgrade, but it seems like you'll need to use safety valves to configure sqoop.properties to perform this upgrade manually.

I'd be curious to know what you might have done to trigger a sqoop upgrade in CDH 4. CM currently assumes that's impossible.

avatar
Explorer

After the CM upgrade from 4.5 to 4.6, I get these errors, I am not able to start the sqoop2 service. So on the forum it is said to manually edit the sqoop.properties files.

 

I did the following

/etc/sqoop2
conf -> /etc/alternatives/sqoop2-conf
/etc/alternatives/sqoop2-conf -> /hdp2014poc/cloudera/parcel-repo/CDH-4.5.0-1.cdh4.5.0.p0.30/etc/sqoop2/conf.dist
sqoop.properties

 

org.apache.sqoop.connector.autoupgrade=true
org.apache.sqoop.framework.autoupgrade=true

 

even after updating the sqoop.properties, I still get the above error.

avatar

CM-managed sqoop2 doesn't read from that file. Edit the safety valve for sqoop.properties instead.