Atlas on HDP 2.5.3 not staring anymore after enabling TLS

[#1]

After enabling TLS with the following properties in Atlas on HDP 2.5.3:

keystore.file=/etc/atlas/conf/keystore.jks
truststore.file=/etc/atlas/conf/truststore.jks
cert.stores.credential.provider.path=/etc/atlas/conf/stores.jceks

Atlas server doesn't start anymore.

Logs:

2017-01-17 15:35:46,681 DEBUG - [main:] ~ cert.stores.credential.provider.path = /etc/atlas/conf/stores.jceks (ApplicationProperties:102)
2017-01-17 15:35:46,682 DEBUG - [main:] ~ keystore.file = /etc/atlas/conf/keystore.jks (ApplicationProperties:102)
2017-01-17 15:35:46,682 DEBUG - [main:] ~ truststore.file = /etc/atlas/conf/truststore.jks (ApplicationProperties:102)
2017-01-17 15:35:46,684 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:168)
2017-01-17 15:35:46,695 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:181)
2017-01-17 15:35:46,716 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:220)
2017-01-17 15:35:46,889 DEBUG - [main:] ~ Setting hadoop.security.token.service.use_ip to true (SecurityUtil:116)
2017-01-17 15:35:46,898 DEBUG - [main:] ~ Failed to detect a valid hadoop home directory (Shell:477)
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
	at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:425)
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:396)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:473)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1443)
	at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:96)
	at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:80)
	at org.apache.atlas.security.InMemoryJAASConfiguration.initialize(InMemoryJAASConfiguration.java:312)
	at org.apache.atlas.security.InMemoryJAASConfiguration.<init>(InMemoryJAASConfiguration.java:216)
	at org.apache.atlas.security.InMemoryJAASConfiguration.init(InMemoryJAASConfiguration.java:184)
	at org.apache.atlas.security.InMemoryJAASConfiguration.init(InMemoryJAASConfiguration.java:172)
	at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:60)
	at org.apache.atlas.Atlas.main(Atlas.java:107)
2017-01-17 15:35:47,015 DEBUG - [main:] ~ setsid exited with exit code 0 (Shell:768)
2017-01-17 15:35:47,041 DEBUG - [main:] ~ Adding client: [KafkaClient{-1}]
	loginModule: [com.sun.security.auth.module.Krb5LoginModule]
	controlFlag: [LoginModuleControlFlag: required]
	Options:  [principal] => [atlas/nmara-hdp-m4.field.hortonworks.com@FIELD.HORTONWORKS.COM]
	Options:  [storeKey] => [true]
	Options:  [keyTab] => [/etc/security/keytabs/atlas.service.keytab]
	Options:  [useKeyTab] => [true]
	Options:  [serviceName] => [kafka]
 (InMemoryJAASConfiguration:334)
2017-01-17 15:35:47,041 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:347)
2017-01-17 15:35:47,042 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:190)
2017-01-17 15:35:47,042 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:177)
2017-01-17 15:35:47,044 INFO  - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (Atlas:134)
2017-01-17 15:35:47,044 INFO  - [main:] ~ 
########################################################################################
                               Atlas Server (STARTUP)


	project.name:	apache-atlas
	project.description:	Metadata Management and Data Governance Platform over Hadoop
	build.user:	jenkins
	build.epoch:	1480481030662
	project.version:	0.7.0.2.5.3.0-37
	build.version:	0.7.0.2.5.3.0-37-rf427fc5f5b82c6582d1520a279f523d1b1c874f6
	vc.revision:	f427fc5f5b82c6582d1520a279f523d1b1c874f6
	vc.source.url:	scm:git:git://git.apache.org/incubator-atlas.git/atlas-webapp
######################################################################################## (Atlas:202)
2017-01-17 15:35:47,045 INFO  - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:203)
2017-01-17 15:35:47,045 INFO  - [main:] ~ Server starting with TLS ? true on port 21443 (Atlas:204)
2017-01-17 15:35:47,045 INFO  - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:205)
2017-01-17 15:35:47,079 DEBUG - [main:] ~ Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog (log:176)
2017-01-17 15:35:47,088 INFO  - [main:] ~ Logging initialized @1508ms (log:186)
2017-01-17 15:35:47,107 DEBUG - [main:] ~ org.eclipse.jetty.server.Server@1b68b9a4 added {qtp878274034{STOPPED,8<=0<=200,i=0,q=0},AUTO} (ContainerLifeCycle:324)
2017-01-17 15:35:47,112 INFO  - [main:] ~ Attempting to retrieve password from configured credential provider path (SecureEmbeddedServer:118)
2017-01-17 15:35:47,195 INFO  - [pool-1-thread-1:] ~ ==> Shutdown of Atlas (Atlas:60)
2017-01-17 15:35:47,195 ERROR - [pool-1-thread-1:] ~ Failed to shutdown (Atlas:64)
java.lang.NullPointerException
	at org.apache.atlas.Atlas.shutdown(Atlas.java:73)
	at org.apache.atlas.Atlas.access$100(Atlas.java:42)
	at org.apache.atlas.Atlas$1.run(Atlas.java:62)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-01-17 15:35:47,196 INFO  - [pool-1-thread-1:] ~ <== Shutdown of Atlas (Atlas:66)
2017-01-17 15:35:47,197 DEBUG - [Thread-0:] ~ ShutdownHookManger complete shutdown. (ShutdownHookManager:84)

It shutdown just after attempting to get passwords from the jceks file.

[#2 · Accepted solution]

@Nicola Marangoni

From the logs, it looks like HADOOP_HOME is not set in your env. can you please set this and try restarting atlas?

java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

Also, please copy hdfs-site.xml to /etc/atlas/conf and make sure right permissions are given to all the files under /etc/atlas/conf. Restart Atlas now, this should resolve the issue.

[#3]

@Ayub Khan thanks for the help!

I put export HADOOP_HOME=/usr/hdp/current/hadoop-client in "atlas-env template" and the error about HADOOP_HOME has gone.

I also copied the hdfs-site.xml to /etc/atlas/conf and chown atlas:hadoop everything there

Atlas fails to start in any case. Previously i did:

/usr/hdp/current/atlas-server/bin/cputil.py

and entered /etc/atlas/conf/stores.jceks as file many times <password>.

Then:

sudo keytool -noprompt \
  -genkey -alias atlasssl -keyalg RSA -keysize 2048 -keypass <password> \
  -keystore /etc/atlas/conf/keystore.jks -storepass <password> \
  -dname "CN=Nicola Marangoni, OU=PS, O=Hortonworks, L=Munich, ST=BY, C=DE"
sudo cp /etc/atlas/conf/keystore.jks /etc/atlas/conf/truststore.jks
sudo chown atlas:hadoop /etc/atlas/conf/*
sudo chmod 400 /etc/atlas/conf/*.jks

Passwords are the same everywhere. Should I retry these last steps?

[#4]

@Nicola Marangoni

I think stores.jceks does not have right permissions, could you please repeat the last step for jceks file as well and try restarting?

From the logs, it looks like Atlas is trying to read the credential provider path which is set to /etc/atlas/conf/stores.jceks in your case and failing because of permissions issue. Also there will be .stores.jceks.crc(hidden file, there is a dot at the beginning) file which also should have same permissions.

It is also recommended to have different passwords for keystore.password & truststore.password/

[#5]

@Ayub Khan I recreated all mentioned files chown also the hidden file (it was still belonging to root) because chmod with * doesn't consider hidden files.

The problem persist. However, I noticed that after running /usr/hdp/current/atlas-server/bin/cputil.py, 3 passwords are asked:

Please enter the full path to the credential provider:/etc/atlas/conf/stores.jceks
Please enter the password value for keystore.password:
Please enter the password value for keystore.password again:
Please enter the password value for truststore.password:
Please enter the password value for truststore.password again:
Please enter the password value for password:
Please enter the password value for password again:

For what is the last password? is it the password for accessing stores.jceks itself?

[#6]

@Ayub Khan

I notice only now this other log entry in a separate file /var/log/atlas/atlas.20170118-091030.err:

Exception in thread "main" java.io.IOException: No CredentialProviderFactory for /etc/atlas/conf/stores.jceks in hadoop.security.credential.provider.path
        at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:66)
        at org.apache.atlas.web.service.SecureEmbeddedServer.getPassword(SecureEmbeddedServer.java:121)
        at org.apache.atlas.web.service.SecureEmbeddedServer.getConnector(SecureEmbeddedServer.java:69)
        at org.apache.atlas.web.service.EmbeddedServer.<init>(EmbeddedServer.java:45)
        at org.apache.atlas.web.service.SecureEmbeddedServer.<init>(SecureEmbeddedServer.java:60)
        at org.apache.atlas.web.service.EmbeddedServer.newServer(EmbeddedServer.java:60)
        at org.apache.atlas.Atlas.main(Atlas.java:117)

[#7]

@Ayub Khan it is sufficient to enter in jceks://file/etc/atlas/conf/stores.jceks the config, no need to execute cputil.py again.

Many thanks!

[#8]

@Nicola Marangoni Ahh that error cleared all clouds, now the issue is very clear.

To solve this, while giving the credential provider path in config as well as when using cpuutil.py, please give the jceks file in the below format. This should work.

"jceks://file/etc/atlas/conf/stores.jceks"

[#9]

That was the problem. Now it works! Thanks!

[#10]

@Nicola Marangoni

I think the property cert.stores.credential.provider.path should be set in below format

cert.stores.credential.provider.path=jceks://file//<path>/test2.jceks

I followed following steps to enable TLS in Atlas

• Properties in atlas-application.properties

# SSL config 
atlas.enableTLS=true 
client.auth.enabled=true
truststore.file=/home/nixon/ssl/atlas.keystore 
cert.stores.credential.provider.path=jceks://file//home/nixon/ssl/test2.jceks 
keystore.file=/home/nixon/ssl/atlas.keystore

• Step to generate atlas.keystore

keytool -genkey -alias serverkey -keypass <keypass> -keyalg RSA -sigalg SHA1withRSA -keystore atlas.keystore -storepass <keypass> -validity 3650 -dname "CN=Nicola Marangoni, OU=PS, O=Hortonworks, L=Munich, ST=BY, C=DE"

Steps to generate jceks file (password used should be same in keystore and jceks file)

cd ~/bin/
./cputil.py
Please enter the full path to the credential provider:jceks://file/home/nixon/ssl/test2.jceks
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Please enter the password value for keystore.password:<keypass>
Please enter the password value for keystore.password again:<keypass>
Please enter the password value for truststore.password:<keypass>
Please enter the password value for truststore.password again:<keypass>
Please enter the password value for password:<keypass>
Please enter the password value for password again:<keypass>

change the file permission of atlas.keystore & jceks files accordingly

Thanks

Nixon