CM update Sqoop Service Environment Safety Valve i

[#1]

Problem desc:

I need to update the 2 properties in sqoop, but I am not able do it through sqoop.propeties files as CM will overide it, Its need to be done in the  Sqoop Service Environment Safety Valve in CM,

But when I did it in the sqoop2 service, in the advanced  configuration,to setup key/value pair for  "Sqoop Service Environment Safety Valve", below is the format.

org.apache.sqoop.connector.autoupgrade=true

org.apache.sqoop.framework.autoupgrade=true

I get the below error,

Could not parse: Sqoop Service Environment Safety Valve : Could not parse parameter 'sqoop_env_safety_valve'. Was expecting: valid variable name. Input: connector.autoupgrade=true

If i just give it like this it works

autoupgrade=true

but I need the whole string. I need help, what is format?

Thanks

[#2]

Hi,

Why do you want to enable auto upgrade? Sqoop has upgrade commands that will do this for you. Auto upgrade may have other side effects that could cause problems down the line.

If you really want to make this change, you need to modify the safety valve for sqoop.properties, not the environment safety valve.

Thanks,

Darren

[#3]

Thanks dlo for reply,

I started doing this because when I start the sqoop service in CM, I get the below error. So some one on the forum suggested that turning to true will help that, 

Connector config urls: [jar:file:/hdp2014poc/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/sqoop2/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2-cdh4.6.0.jar!/sqoopconnector.properties]

Connector [org.apache.sqoop.connector.jdbc.GenericJdbcConnector] initialized.

Marking transaction for rollback

Attempting transaction roll-back

Server startup failure org.apache.sqoop.common.SqoopException: CONN_0007:Connector registration failed at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:234) at org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:195) at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:51) at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0026:Upgrade required but not allowed - Connector: generic-jdbc-connector at org.apache.sqoop.repository.JdbcRepository$3.doIt(JdbcRepository.java:176) at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90) at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61) at org.apache.sqoop.repository.JdbcRepository.registerConnector(JdbcRepository.java:156) at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:214) ... 27 more

Shutting down Sqoop server

[#4]

When you get that message, it generally means you should run the upgrade command. CM has made this one easy for you, so no need to figure out how to do the upgrade manually.

[#5]

Thanks again,

but how do I run this upgrade command in CM.

Any tips would be helpful.

[#6]

Click on Sqoop, then in the Actions dropdown menu on the right, you'll see an option to Upgrade Sqoop. If it's greyed out, that's because you need to stop Sqoop first.

Thanks,

DArren

[#7]

I am using CDH 4.6, 

I don't see the upgrade in action

these all the choices I have start, stop,restart,rename,enter maintenance mode, create group.

Let me know If I am missing something.

[#8]

It looks like the sqoop upgrade action is only available in CM for CDH 5 clusters. Sorry for the confusion.

I'm not sure what triggered the need for a sqoop upgrade, but it seems like you'll need to use safety valves to configure sqoop.properties to perform this upgrade manually.

I'd be curious to know what you might have done to trigger a sqoop upgrade in CDH 4. CM currently assumes that's impossible.

[#9]

After the CM upgrade from 4.5 to 4.6, I get these errors, I am not able to start the sqoop2 service. So on the forum it is said to manually edit the sqoop.properties files.

I did the following

/etc/sqoop2
conf -> /etc/alternatives/sqoop2-conf
/etc/alternatives/sqoop2-conf -> /hdp2014poc/cloudera/parcel-repo/CDH-4.5.0-1.cdh4.5.0.p0.30/etc/sqoop2/conf.dist
sqoop.properties

org.apache.sqoop.connector.autoupgrade=true
org.apache.sqoop.framework.autoupgrade=true

even after updating the sqoop.properties, I still get the above error.

[#10]

CM-managed sqoop2 doesn't read from that file. Edit the safety valve for sqoop.properties instead.

[#11]

Thanks again,

but we are back to sqare one, I will repost the question again,

Problem desc:

I need to update the 2 properties in sqoop, but I am not able do it through sqoop.propeties files as CM will overide it, Its need to be done in the  Sqoop Service Environment Safety Valve in CM,

But when I did it in the sqoop2 service, in the advanced  configuration,to setup key/value pair for  "Sqoop Service Environment Safety Valve", below is the format.

org.apache.sqoop.connector.autoupgrade=true

org.apache.sqoop.framework.autoupgrade=true

I get the below error,

Could not parse: Sqoop Service Environment Safety Valve : Could not parse parameter 'sqoop_env_safety_valve'. Was expecting: valid variable name. Input: connector.autoupgrade=true

If i just give it like this it works

autoupgrade=true

but I need the whole string. I need help, what is format?

[#12 · Accepted solution]

There are multiple safety valves for sqoop, and you are editing the wrong one. See my first reply.

You need to edit the one that will alter sqoop.properties, NOT the one that will alter the environment. Read the name and description carefully.

[#13]

Thanks Darren,

Your are right there are couple of saftey valve, I was modifiying the wrong one.

I got it working.

Thanks again.

Praveen

[#14]

Glad you got it working!