Created 01-17-2017 04:02 PM
After enabling TLS with the following properties in Atlas on HDP 2.5.3:
keystore.file=/etc/atlas/conf/keystore.jks truststore.file=/etc/atlas/conf/truststore.jks cert.stores.credential.provider.path=/etc/atlas/conf/stores.jceks
Atlas server doesn't start anymore.
Logs:
2017-01-17 15:35:46,681 DEBUG - [main:] ~ cert.stores.credential.provider.path = /etc/atlas/conf/stores.jceks (ApplicationProperties:102) 2017-01-17 15:35:46,682 DEBUG - [main:] ~ keystore.file = /etc/atlas/conf/keystore.jks (ApplicationProperties:102) 2017-01-17 15:35:46,682 DEBUG - [main:] ~ truststore.file = /etc/atlas/conf/truststore.jks (ApplicationProperties:102) 2017-01-17 15:35:46,684 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:168) 2017-01-17 15:35:46,695 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:181) 2017-01-17 15:35:46,716 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:220) 2017-01-17 15:35:46,889 DEBUG - [main:] ~ Setting hadoop.security.token.service.use_ip to true (SecurityUtil:116) 2017-01-17 15:35:46,898 DEBUG - [main:] ~ Failed to detect a valid hadoop home directory (Shell:477) java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:425) at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:396) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:473) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1443) at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:96) at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:80) at org.apache.atlas.security.InMemoryJAASConfiguration.initialize(InMemoryJAASConfiguration.java:312) at org.apache.atlas.security.InMemoryJAASConfiguration.<init>(InMemoryJAASConfiguration.java:216) at org.apache.atlas.security.InMemoryJAASConfiguration.init(InMemoryJAASConfiguration.java:184) at org.apache.atlas.security.InMemoryJAASConfiguration.init(InMemoryJAASConfiguration.java:172) at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:60) at org.apache.atlas.Atlas.main(Atlas.java:107) 2017-01-17 15:35:47,015 DEBUG - [main:] ~ setsid exited with exit code 0 (Shell:768) 2017-01-17 15:35:47,041 DEBUG - [main:] ~ Adding client: [KafkaClient{-1}] loginModule: [com.sun.security.auth.module.Krb5LoginModule] controlFlag: [LoginModuleControlFlag: required] Options: [principal] => [atlas/nmara-hdp-m4.field.hortonworks.com@FIELD.HORTONWORKS.COM] Options: [storeKey] => [true] Options: [keyTab] => [/etc/security/keytabs/atlas.service.keytab] Options: [useKeyTab] => [true] Options: [serviceName] => [kafka] (InMemoryJAASConfiguration:334) 2017-01-17 15:35:47,041 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:347) 2017-01-17 15:35:47,042 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:190) 2017-01-17 15:35:47,042 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:177) 2017-01-17 15:35:47,044 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (Atlas:134) 2017-01-17 15:35:47,044 INFO - [main:] ~ ######################################################################################## Atlas Server (STARTUP) project.name: apache-atlas project.description: Metadata Management and Data Governance Platform over Hadoop build.user: jenkins build.epoch: 1480481030662 project.version: 0.7.0.2.5.3.0-37 build.version: 0.7.0.2.5.3.0-37-rf427fc5f5b82c6582d1520a279f523d1b1c874f6 vc.revision: f427fc5f5b82c6582d1520a279f523d1b1c874f6 vc.source.url: scm:git:git://git.apache.org/incubator-atlas.git/atlas-webapp ######################################################################################## (Atlas:202) 2017-01-17 15:35:47,045 INFO - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:203) 2017-01-17 15:35:47,045 INFO - [main:] ~ Server starting with TLS ? true on port 21443 (Atlas:204) 2017-01-17 15:35:47,045 INFO - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:205) 2017-01-17 15:35:47,079 DEBUG - [main:] ~ Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog (log:176) 2017-01-17 15:35:47,088 INFO - [main:] ~ Logging initialized @1508ms (log:186) 2017-01-17 15:35:47,107 DEBUG - [main:] ~ org.eclipse.jetty.server.Server@1b68b9a4 added {qtp878274034{STOPPED,8<=0<=200,i=0,q=0},AUTO} (ContainerLifeCycle:324) 2017-01-17 15:35:47,112 INFO - [main:] ~ Attempting to retrieve password from configured credential provider path (SecureEmbeddedServer:118) 2017-01-17 15:35:47,195 INFO - [pool-1-thread-1:] ~ ==> Shutdown of Atlas (Atlas:60) 2017-01-17 15:35:47,195 ERROR - [pool-1-thread-1:] ~ Failed to shutdown (Atlas:64) java.lang.NullPointerException at org.apache.atlas.Atlas.shutdown(Atlas.java:73) at org.apache.atlas.Atlas.access$100(Atlas.java:42) at org.apache.atlas.Atlas$1.run(Atlas.java:62) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-01-17 15:35:47,196 INFO - [pool-1-thread-1:] ~ <== Shutdown of Atlas (Atlas:66) 2017-01-17 15:35:47,197 DEBUG - [Thread-0:] ~ ShutdownHookManger complete shutdown. (ShutdownHookManager:84)
It shutdown just after attempting to get passwords from the jceks file.
Created 01-17-2017 05:35 PM
From the logs, it looks like HADOOP_HOME is not set in your env. can you please set this and try restarting atlas?
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
Also, please copy hdfs-site.xml to /etc/atlas/conf and make sure right permissions are given to all the files under /etc/atlas/conf. Restart Atlas now, this should resolve the issue.
Created 01-17-2017 05:35 PM
From the logs, it looks like HADOOP_HOME is not set in your env. can you please set this and try restarting atlas?
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
Also, please copy hdfs-site.xml to /etc/atlas/conf and make sure right permissions are given to all the files under /etc/atlas/conf. Restart Atlas now, this should resolve the issue.
Created 01-17-2017 06:33 PM
@Ayub Khan thanks for the help!
I put export HADOOP_HOME=/usr/hdp/current/hadoop-client in "atlas-env template" and the error about HADOOP_HOME has gone.
I also copied the hdfs-site.xml to /etc/atlas/conf and chown atlas:hadoop everything there
Atlas fails to start in any case. Previously i did:
/usr/hdp/current/atlas-server/bin/cputil.py
and entered /etc/atlas/conf/stores.jceks as file many times <password>.
Then:
sudo keytool -noprompt \ -genkey -alias atlasssl -keyalg RSA -keysize 2048 -keypass <password> \ -keystore /etc/atlas/conf/keystore.jks -storepass <password> \ -dname "CN=Nicola Marangoni, OU=PS, O=Hortonworks, L=Munich, ST=BY, C=DE" sudo cp /etc/atlas/conf/keystore.jks /etc/atlas/conf/truststore.jks sudo chown atlas:hadoop /etc/atlas/conf/* sudo chmod 400 /etc/atlas/conf/*.jks
Passwords are the same everywhere. Should I retry these last steps?
Created 01-18-2017 07:05 AM
I think stores.jceks does not have right permissions, could you please repeat the last step for jceks file as well and try restarting?
From the logs, it looks like Atlas is trying to read the credential provider path which is set to /etc/atlas/conf/stores.jceks in your case and failing because of permissions issue. Also there will be .stores.jceks.crc(hidden file, there is a dot at the beginning) file which also should have same permissions.
It is also recommended to have different passwords for keystore.password & truststore.password/
Created 01-18-2017 08:47 AM
@Ayub Khan I recreated all mentioned files chown also the hidden file (it was still belonging to root) because chmod with * doesn't consider hidden files.
The problem persist. However, I noticed that after running /usr/hdp/current/atlas-server/bin/cputil.py, 3 passwords are asked:
Please enter the full path to the credential provider:/etc/atlas/conf/stores.jceks Please enter the password value for keystore.password: Please enter the password value for keystore.password again: Please enter the password value for truststore.password: Please enter the password value for truststore.password again: Please enter the password value for password: Please enter the password value for password again:
For what is the last password? is it the password for accessing stores.jceks itself?
Created 01-18-2017 09:20 AM
I notice only now this other log entry in a separate file /var/log/atlas/atlas.20170118-091030.err:
Exception in thread "main" java.io.IOException: No CredentialProviderFactory for /etc/atlas/conf/stores.jceks in hadoop.security.credential.provider.path at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:66) at org.apache.atlas.web.service.SecureEmbeddedServer.getPassword(SecureEmbeddedServer.java:121) at org.apache.atlas.web.service.SecureEmbeddedServer.getConnector(SecureEmbeddedServer.java:69) at org.apache.atlas.web.service.EmbeddedServer.<init>(EmbeddedServer.java:45) at org.apache.atlas.web.service.SecureEmbeddedServer.<init>(SecureEmbeddedServer.java:60) at org.apache.atlas.web.service.EmbeddedServer.newServer(EmbeddedServer.java:60) at org.apache.atlas.Atlas.main(Atlas.java:117)
Created 01-18-2017 09:45 AM
@Ayub Khan it is sufficient to enter in jceks://file/etc/atlas/conf/stores.jceks the config, no need to execute cputil.py again.
Many thanks!
Created 01-18-2017 09:33 AM
@Nicola Marangoni Ahh that error cleared all clouds, now the issue is very clear.
To solve this, while giving the credential provider path in config as well as when using cpuutil.py, please give the jceks file in the below format. This should work.
"jceks://file/etc/atlas/conf/stores.jceks"
Created 01-18-2017 10:31 AM
That was the problem. Now it works! Thanks!
Created 01-18-2017 10:08 AM
I think the property cert.stores.credential.provider.path should be set in below format
cert.stores.credential.provider.path=jceks://file//<path>/test2.jceks
I followed following steps to enable TLS in Atlas
# SSL config atlas.enableTLS=true client.auth.enabled=true truststore.file=/home/nixon/ssl/atlas.keystore cert.stores.credential.provider.path=jceks://file//home/nixon/ssl/test2.jceks keystore.file=/home/nixon/ssl/atlas.keystore
keytool -genkey -alias serverkey -keypass <keypass> -keyalg RSA -sigalg SHA1withRSA -keystore atlas.keystore -storepass <keypass> -validity 3650 -dname "CN=Nicola Marangoni, OU=PS, O=Hortonworks, L=Munich, ST=BY, C=DE"
Steps to generate jceks file (password used should be same in keystore and jceks file)
cd ~/bin/ ./cputil.py Please enter the full path to the credential provider:jceks://file/home/nixon/ssl/test2.jceks 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Please enter the password value for keystore.password:<keypass> Please enter the password value for keystore.password again:<keypass> Please enter the password value for truststore.password:<keypass> Please enter the password value for truststore.password again:<keypass> Please enter the password value for password:<keypass> Please enter the password value for password again:<keypass>
change the file permission of atlas.keystore & jceks files accordingly
Thanks
Nixon