Member since
04-28-2017
3
Posts
0
Kudos Received
0
Solutions
05-04-2017
03:23 AM
@slachtermanWe have created an instance profile for the node, and not added credentials in core-site.xml. hadoop fs -ls s3a:// works and even selecting few rows from the external table (whose data is in s3) works, but I try to do aggregation function like : select max(updated_at) from s3_table; This query fails with the below mentioned error. Could you please help. Caused by: java.lang.RuntimeException: java.io.IOException: java.io.IOException: Cannot find password option fs.s3a.access.key
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:694)
at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:653)
at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:145)
at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
... 15 more
Caused by: java.io.IOException: java.io.IOException: Cannot find password option fs.s3a.access.key
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:382)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
... 26 more
Caused by: java.io.IOException: Cannot find password option fs.s3a.access.key
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:489)
at org.apache.hadoop.fs.s3a.S3AUtils.getPassword(S3AUtils.java:468)
at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:451)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:341)
at org.apache.hadoop.fs.s3a.S3ClientFactory$DefaultS3ClientFactory.createS3Client(S3ClientFactory.java:73)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:185)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:244)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:94)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:80)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:380)
... 27 more
Caused by: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1999)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1959)
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:484)
... 45 more
Caused by: java.io.IOException: No CredentialProviderFactory for jceks://file/usr/hdp/current/hive-server2-hive2/conf/conf.server/hive-site.jceks in hadoop.security.credential.provider.path
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:66)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1979)
... View more
04-29-2017
03:52 AM
We are using Ambari 2.5.0.3. Our LDAP Setup configs are as below:
Primary URL* {host:port} (host:3269): my.domain.com:3269
Secondary URL {host:port} :
Use SSL* [true/false] (true): true
User object class* (user): user
User name attribute* (sAMAccountName): sAMAccountName
Group object class* (group): group
Group name attribute* (memberof, ismemberof): memberof, ismemberof
Group member attribute* (member): member
Distinguished name attribute* (distinguishedName):distinguishedName
Base DN* : DC=my,DC=server,DC=com
Referral method [follow/ignore] (follow): follow
Bind anonymously* [true/false] (false): false
Handling behavior for username collisions [convert/skip] for LDAP sync* (skip): skip
Manager DN*: CN=bind_user,OU=Users,DC=my,DC=server,DC=com
Enter Manager Password* :
Re-enter password:
Do you want to provide custom TrustStore for Ambari [y/n] (y)?n
The TrustStore is already configured:
ssl.trustStore.type = jks
ssl.trustStore.path = /path/to/certs/ ssl.trustStore.password = xxxxxxx After this when I restart the ambari-server and sync-ldap, I get all the users. But when I try to login, I get "Invalid Username/Password error". I turned on the debug logging, and I see the below messages, which says Found DN with my user name and then says invalid user. Ranger is working fine with the same user from ldap: 28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] FilterChainProxy:337 - /api/v1/users/makaur10?fields=*,privileges/PrivilegeInfo/cluster_name,privileges/PrivilegeInfo/permission_name&_=1493430718712 at position 1 of 10 in additional filter chain; firing Filter: 'SecurityContextPersistenceFilter'
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] HttpSessionSecurityContextRepository:127 - No HttpSession currently exists
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] HttpSessionSecurityContextRepository:85 - No SecurityContext was available from the HttpSession: null. A new one will be created.
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] FilterChainProxy:337 - /api/v1/users/makaur10?fields=*,privileges/PrivilegeInfo/cluster_name,privileges/PrivilegeInfo/permission_name&_=1493430718712 at position 2 of 10 in additional filter chain; firing Filter: 'AmbariUserAuthorizationFilter'
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] FilterChainProxy:337 - /api/v1/users/makaur10?fields=*,privileges/PrivilegeInfo/cluster_name,privileges/PrivilegeInfo/permission_name&_=1493430718712 at position 3 of 10 in additional filter chain; firing Filter: 'AmbariDelegatingAuthenticationFilter'
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] AmbariDelegatingAuthenticationFilter:117 - Using authentication filter org.apache.ambari.server.security.authentication.AmbariBasicAuthenticationFilter since it applies
28 Apr 2017 20:51:57,635 DEBUG [ambari-client-thread-35] AmbariBasicAuthenticationFilter:161 - Basic Authentication Authorization header found for user 'makaur10'
28 Apr 2017 20:51:57,636 DEBUG [ambari-client-thread-35] ProviderManager:152 - Authentication attempt using org.apache.ambari.server.security.authorization.AmbariLocalUserProvider
28 Apr 2017 20:51:57,641 DEBUG [ambari-client-thread-35] ProviderManager:152 - Authentication attempt using org.apache.ambari.server.security.authorization.AmbariPamAuthenticationProvider
28 Apr 2017 20:51:57,641 DEBUG [ambari-client-thread-35] ProviderManager:152 - Authentication attempt using org.apache.ambari.server.security.authorization.AmbariLdapAuthenticationProvider
28 Apr 2017 20:51:57,642 DEBUG [ambari-client-thread-35] Configuration:3878 - Reading password from file /etc/ambari-server/conf/ldap-password.dat
28 Apr 2017 20:51:57,642 DEBUG [ambari-client-thread-35] AbstractContextSource:418 - AuthenticationSource not set - using default implementation
28 Apr 2017 20:51:57,642 DEBUG [ambari-client-thread-35] AbstractContextSource:441 - Not using LDAP pooling
28 Apr 2017 20:51:57,642 DEBUG [ambari-client-thread-35] AbstractContextSource:462 - Trying provider Urls: ldaps://my.server.com:3269/DC=my,DC=server,DC=com
28 Apr 2017 20:51:57,642 INFO [ambari-client-thread-35] FilterBasedLdapUserSearch:95 - SearchBase not set. Searches will be performed from the root: dc=my,dc=server,dc=com
28 Apr 2017 20:51:57,643 DEBUG [ambari-client-thread-35] LdapAuthenticationProvider:67 - Processing authentication request for user: makaur10
28 Apr 2017 20:51:57,643 DEBUG [ambari-client-thread-35] FilterBasedLdapUserSearch:115 - Searching for user 'makaur10', with user search [ searchFilter: '(&(sAMAccountName={0})(objectClass=person))', searchBase: '', scope: subtree, searchTimeLimit: 0, derefLinkFlag: false ]
28 Apr 2017 20:51:57,717 DEBUG [ambari-client-thread-35] AbstractContextSource:349 - Got Ldap context on server 'ldaps://my.server.com:3269/DC=my,DC=server,DC=com'
28 Apr 2017 20:51:57,719 DEBUG [ambari-client-thread-35] SpringSecurityLdapTemplate:337 - Searching for entry under DN 'dc=my,dc=server,dc=com', base = '', filter = '(&(sAMAccountName={0})(objectClass=person))'
28 Apr 2017 20:51:57,720 DEBUG [ambari-client-thread-35] SpringSecurityLdapTemplate:350 - Found DN: CN=makaur10,OU=Technology,OU=Users,OU=Corp
28 Apr 2017 20:51:57,780 DEBUG [ambari-client-thread-35] AbstractContextSource:349 - Got Ldap context on server 'ldaps://my.server.com:3269/DC=my,DC=server,DC=com'
28 Apr 2017 20:51:57,781 DEBUG [ambari-client-thread-35] Configuration:3878 - Reading password from file /etc/ambari-server/conf/ldap-password.dat
28 Apr 2017 20:51:57,842 DEBUG [ambari-client-thread-35] AbstractContextSource:349 - Got Ldap context on server 'ldaps://my.server.com:3269/DC=my,DC=server,DC=com'
28 Apr 2017 20:51:57,843 DEBUG [ambari-client-thread-35] DefaultAuthenticationEventPublisher:94 - No event was found for the exception org.apache.ambari.server.security.authorization.InvalidUsernamePasswordCombinationException
28 Apr 2017 20:51:57,843 DEBUG [ambari-client-thread-35] AmbariBasicAuthenticationFilter:185 - Authentication request for failed: org.apache.ambari.server.security.authorization.InvalidUsernamePasswordCombinationException: Unable to sign in. Invalid username/password combination.
28 Apr 2017 20:51:57,843 DEBUG [ambari-client-thread-35] HttpSessionSecurityContextRepository:269 - SecurityContext is empty or contents are anonymous - context will not be stored in HttpSession.
28 Apr 2017 20:51:57,844 DEBUG [ambari-client-thread-35] SecurityContextPersistenceFilter:97 - SecurityContextHolder now cleared, as request processing completed
... View more
Labels:
- Labels:
-
Apache Ambari
