Trying to view yarn app logs using know on a kerberized cluster. Getting 404 when clicking on logs, it is trying to go to /gateway/nodemanagerui. Is that a separate topology? ... View more

I am trying to configure accessing the yarn logs through Knox and I saw with the latest version of Knox that I need to add the NODEUI topology to make that work. I am trying the below topology but it does not seem to be working. <service> <role>NODEUI</role> <url>http://{{nm_host}}:{{nodeui_port}}</url> </service> This will also link to a single nodemanager, which is not very practical. Any recommendations to get this working please? ... View more

I am trying to view Yarn application logs using Knox on a kerberized cluster. Every time I click on the logs link it takes me to an invalid URL. I looked at the rewrite rules for yarnui and I see the below section: <rule dir="OUT" name="YARNUI/yarn/outbound/node/containerlogs"> <match pattern="{scheme}://{host}:{port}/node/containerlogs/{**}"/> <rewrite template="{gateway.scheme}://{gateway.host}:{gateway.port}/gateway/nodemanagerui/node/containerlogs/{**}?{scheme}?host={$hostmap(host)}?{port}"/> </rule> From the look of it, it is trying to redirect to another topology, not another service in the same topology. I am also unable to find any service definition for nodemanagerui, which confirms my suspicion about the additional topology. Anyone had any idea how to resolve this issue, or how to define this topology please? ... View more

I am trying to configure Ambari to use PAM and would like to assign a specific group to role during the configuration. I am performing the configuration inline and would like to add the option of adding a specific group to the service operator group while performing inline setup. My PAM setup command looks like this: ambari-server setup-pam --pam-config-file /etc/pam.d/login --pam-auto-create-groups true I have seen in some distribution that I can specify custom group mapping during the PAM configuration, like in this link: https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_pam_ambari.html While with HDP 2.6.2 and Ambari 2.5.2, I do not get any option to update the custom group mapping to roles. I added the property pam.group.cluster.admin=abcgroup to ambari.properties but that did not work also as it did show the group mapping in Ambari after restarting the ambari server. Does anyone know what I am doing wrong and what should I be doing instead? ... View more

Is there a way to make Hiveserver2 listen to 127.0.0.1 or localhost? I am installing a separate hiveserver2 instance to be used by an application running on an edge node and I want to restrict access to that hiveserver2 instance to only queries coming from that application. ... View more

Is there a way to ask Oozie to disable some actions and not make them available? I am looking to disable shell action and couple others please? ... View more

I am unable to start the Job History Server after kerberizing the cluster. I keep getting this exception everytime I try to start it. at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089) ... 6 more 2017-11-21 12:43:06,984 FATAL hs.JobHistoryServer (JobHistoryServer.java:launchJobHistoryServer(224)) - Error starting JobHistoryServer org.apache.hadoop.yarn.exceptions.YarnRuntimeException: History Server Failed to login at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:128) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:221) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:231) Caused by: java.io.IOException: Login failure for jhs/HOST@KRB_HOST from keytab /etc/security/keytabs/jhs.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1098) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:307) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.doSecureLogin(JobHistoryServer.java:175) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:126) ... 3 more Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089) ... 6 more ... View more

I am using blueprint to create HDP 2.6.2 cluster, with namenode HA and resourcemanager HA. It installs all the components but when trying to start the components, only HDFS and Zookeeper start and the others fail, like yarn, hive, etc.... when trying to list hdfs on the command line: hdfs dfs -ls / I get the following error: Incomplete HDFS URI, no host: hdfs://1234_test_cluster Anyone know what the issue could be? ... View more

I am trying to use HttpFS with PAM authentication. I was able to install it using this post: https://community.hortonworks.com/content/kbentry/804/httpfs-configure-and-run-with-hdp-224x.html Then I tried to configure PAM authentication using JPam. I copied the jpam-1.1.jar under /usr/hdp/current/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib I also modified /usr/hdp/current/hadoop-httpfs/webapps/webhdfs/META-INF/context.xml to be <Context><Realm className="org.apache.catalina.realm.JAASRealm" appName="jpamLogin"/></Context> everytime I try to access httpfs and provide the username:password it gives me this error org.apache.catalina.realm.JAASRealm authenticateSEVERE: Unexpected errorjavax.security.auth.login.LoginException: No LoginModules configured for jpamLoginat javax.security.auth.login.LoginContext.init(LoginContext.java:264)at javax.security.auth.login.LoginContext.<init>(LoginContext.java:417)at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:393)at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:334)at org.apache.catalina.authenticator.BasicAuthenticator.authenticate(BasicAuthenticator.java:181)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)at java.lang.Thread.run(Thread.java:748) UPDATE 2: I made some progress. I created the file /etc/hadoop-httpfs/tomcat-deployment/conf/jaas.config with the following content jpamLogin {net.sf.jpam.jaas.JpamLoginModule required serviceName="password-auth";}; then I updated httpfs-env.sh with export CATALINA_OPTS='-Djava.security.auth.login.config=/etc/hadoop-httpfs/tomcat-deployment/conf/jaas.config -Djava.library.path=/usr/hdp/current/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/' Restart httpfs and I get this exception now. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} javax.security.auth.login.LoginException: java.lang.UnsatisfiedLinkError: no jpam in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at net.sf.jpam.Pam.<clinit>(Pam.java:51) at net.sf.jpam.jaas.JpamLoginModule.createPam(JpamLoginModule.java:171) at net.sf.jpam.jaas.JpamLoginModule.login(JpamLoginModule.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:409) at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:334) at org.apache.catalina.authenticator.BasicAuthenticator.authenticate(BasicAuthenticator.java:181) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503) at java.lang.Thread.run(Thread.java:748) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:409) at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:334) at org.apache.catalina.authenticator.BasicAuthenticator.authenticate(BasicAuthenticator.java:181) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503) at java.lang.Thread.run(Thread.java:748) I appreciate your help ... View more

I would like to use a custom queue, for example queue1, as the default queue when any user submits a job to yarn and that user is not mapped to any existing queue. The default behavior for yarn would be to use the default queue for those sort of jobs, but I would like to instruct Yarn to use another queue as its default queue. Is there a way to do it please? ... View more

Is there a way to integrate Solr with LDAP. In other ways is there a way to configure Solr to use LDAP for user authentication? I am using Solr 6.2 ... View more

Is there a way to modify the Yarn application container logs URL to use HTTPS instead of HTTP? and also to modify how the URL is constructed to use custom location if that is possible? ... View more

I am trying to update the yarn scheduler queues through rest api and it does not seem to be working as I expect. I followed the example at the post: https://community.hortonworks.com/questions/33578/api-to-manage-yarn-capacity-queue.html but I keep getting the error 415 Unsupported Media Type. Here is the command I am using: curl -kv -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT -d@yarn_queues.json https://<FQDN>:<PORT>/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration ... View more