Dear experts, I am trying to copy a file from local filesystem to hdfs and getting below issue. Could you please let me know why this happens. I have not enabled HDFS ACLs yet. hdfs dfs -copyFromLocal -f sample.txt /data/test copyFromLocal: Permission denied: user=devuser, access=EXECUTE, inode="/data/test":hdfs:hdfs:d--------- Below are the details of the directories: and i clearly see devuser has access on the target directory d--------- - hdfs hdfs 0 2019-06-03 15:41 /data drwxr-xr-x - devuser hdfs 0 2019-07-05 14:42 /data/test Moreover it is weird that the error is throwing for EXECUTE permission where i am expecting WRITE/READ permission denied. Thanks, Chiranjeevi ... View more

Dear experts, I have a kerberized (using Active Directory) HDP cluster and an external Solr cloud(Not kerberized). I am now trying to configure Ranger audits to point to the external SolrCloud and not able to find any process. Could you please guide me regarding this ? I have followed the process from the below link to setup an external SolrCloud using Rangerscripts and confused how to proceed with kerberization. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/solr_ranger_configure_solrcloud.html It will be really helpful if i can get any documentation/links regarding this process. Thanks ... View more

Dear experts, We have installed Ranger with LDAP sync and after few days we have enabled Kerberos using AD. After which the usersync is getting failed with the below error, please suggest 21 Jun 2018 17:47:58 INFO UnixAuthenticationService [main] - Starting User Sync Service! 21 Jun 2018 17:47:58 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 21 Jun 2018 17:47:58 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 21 Jun 2018 17:47:58 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder created 21 Jun 2018 17:47:58 INFO UserGroupSyncConfig [UnixUserSyncThread] - Sleep Time Between Cycle can not be lower than [3600000] millisec. resetting to min value. 21 Jun 2018 17:47:58 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder 21 Jun 2018 17:47:58 WARN NativeCodeLoader [UnixUserSyncThread] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21 Jun 2018 17:47:59 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 21 Jun 2018 17:47:59 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder created 21 Jun 2018 17:47:59 INFO UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder initialization started 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder initialization completed with -- ldapUrl: ldap://XXXXX:389, ldapBindDn: CN=XXXX,OU=XXXX,DC=XXX,DC=XXX, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: DC=aero,DC=local, userSearchBase: [OU=XXX,DC=XX,DC=XXX], userSearchScope: 2, userObjectClass: user, userSearchFilter: , extendedUserSearchFilter: null, userNameAttribute: sAMAccountName, userSearchAttributes: [uSNChanged, sAMAccountName, modifytimestamp], userGroupNameAttributeSet: null, pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase: [XXXX], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: , extendedGroupSearchFilter: (&null(|(member={0})(member={1}))), extendedAllGroupsSearchFilter: null, groupMemberAttributeName: member, groupNameAttribute: name, groupSearchAttributes: [uSNChanged, name, member, modifytimestamp], groupUserMapSyncEnabled: true, groupSearchFirstEnabled: false, userSearchEnabled: false, ldapReferral: ignore 21 Jun 2018 17:47:59 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder updateSink started 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Performing user search first 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - extendedUserSearchFilter = (&(objectclass=user)(|(uSNChanged>=0)(modifyTimestamp>=19700101120000Z))) 21 Jun 2018 17:47:59 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - uSNChangedVal = 225739and currentDeltaSyncTime = 225739 21 Jun 2018 17:47:59 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://10.1.1.5:6080/service/users/default returned a response status of 401 Unauthorized at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.getMUser(LdapPolicyMgrUserGroupBuilder.java:672) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.access$500(LdapPolicyMgrUserGroupBuilder.java:73) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$6.run(LdapPolicyMgrUserGroupBuilder.java:645) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$6.run(LdapPolicyMgrUserGroupBuilder.java:641) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.addMUser(LdapPolicyMgrUserGroupBuilder.java:641) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.addOrUpdateUser(LdapPolicyMgrUserGroupBuilder.java:273) at org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder.getUsers(LdapDeltaUserGroupBuilder.java:468) at org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder.updateSink(LdapDeltaUserGroupBuilder.java:311) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:748) 21 Jun 2018 17:47:59 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user ... View more

Dear experts, We are enabling kerberos in our cluster with integrating it to Active Directory. The Kerberos has been enabled however during the service restarts, all the services are being failed with the below error, could you please assist on this ? /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-hdp@HADOOP.LOCAL;' returned 1. kinit: Preauthentication failed while getting initial credentials Zookeeper logs: 2018-06-17 14:02:52,217 - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started. 2018-06-17 14:02:52,218 - INFO [main:QuorumPeerMain@127] - Starting quorum peer 2018-06-17 14:02:52,229 - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed. 2018-06-17 14:02:52,814 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Because of this issues, it is not allowing us to disable the kerberos now. Kindly help on this. Thanks, Chiranjeevi ... View more

Dear experts, I am trying to create a HDP cluster using Cloudbreak 2.4.1 with Kerberos enabled. For this setup, i used Azure and i have installed MIT KDC server on the cloudbreak VM. In the create cluster page i have submitted all the necessary information about my KDC and started the creation. From the logs, i can see the VMs are provisioned and ambari server and agent have been started successfully, but after sometime the installation appears failed with the below error, could you please suggest where i am wrong ? Error message: /cbreak_cloudbreak_1 | FAILED: Failed to create the account for hdfs-hdpprodnew@CLZGBQSKXCTEVBZ1J2DNGVCSCA.TX.INTERNAL.CLOUDAPP.NET', status: 'ABORTED'] /cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation start failed: [component:'INSTALL_START', requestID: '1', context: 'Logical Request: Provision Cluster 'hdpprodnew' /cbreak_cloudbreak_1 | FAILED: Failed to create the account for hdfs-hdpprodnew@CLZGBQSKXCTEVBZ1J2DNGVCSCA.TX.INTERNAL.CLOUDAPP.NET', status: 'ABORTED'] cbd logs: name:hdpprodnew] [tracking:] Polling attempt 115. 2018-05-30 10:38:44,976 [reactorDispatcher-48] getRawResource:79 INFO c.s.a.c.AmbariClientUtils - [instance:29388674-8352-4092-9b17-495aa103ab97] [type:CLUSTER] [id:6] [name:hdpprodnew] [tracking:] AmbariClient getRawResource, resourceRequestMap: {path=https://104.211.222.94:9443/api/v1/clusters/hdpprodnew/requests/1, query={fields=Requests/*}} 2018-05-30 10:38:45,000 [reactorDispatcher-48] checkStatus:32 INFO c.s.c.s.c.f.AmbariOperationsStartCheckerTask - [instance:29388674-8352-4092-9b17-495aa103ab97] [type:CLUSTER] [id:6] [name:hdpprodnew] [tracking:] Ambari operation start: 'INSTALL_START', Progress: 0.0 2018-05-30 10:38:50,001 [reactorDispatcher-48] pollWithTimeout:32 INFO c.s.c.s.PollingService - [instance:29388674-8352-4092-9b17-495aa103ab97] [type:CLUSTER] [id:6] [name:hdpprodnew] I have chosen the base image instead of pre-warmed image as the blueprint which came with the Cloudbreak. Thanks ... View more

Dear experts, I have installed a 3 node HDP cluster on Azure. Due to some problem, all the VMs got restarted abruptly, after this i have manually restarted all the ambari agents and ambari server. I am able to see the agents and server is running fine but all of the services are in "Lost heartbeat"state. Could you please assist ? Below is the log file of a Ambari agent on master node: INFO 2018-05-29 05:57:21,051 Controller.py:311 - Building heartbeat message INFO 2018-05-29 05:57:21,053 Heartbeat.py:90 - Adding host info/state to heartbeat message. INFO 2018-05-29 05:57:21,149 logger.py:75 - Testing the JVM's JCE policy to see it if supports an unlimited key length. INFO 2018-05-29 05:57:21,149 logger.py:75 - Testing the JVM's JCE policy to see it if supports an unlimited key length. INFO 2018-05-29 05:57:21,995 Hardware.py:176 - Some mount points were ignored: /, /dev, /dev/shm, /, /mnt/resource INFO 2018-05-29 05:57:21,996 Controller.py:320 - Sending Heartbeat (id = 204) INFO 2018-05-29 05:57:22,039 Controller.py:333 - Heartbeat response received (id = 205) INFO 2018-05-29 05:57:22,040 Controller.py:342 - Heartbeat interval is 10 seconds INFO 2018-05-29 05:57:22,040 Controller.py:380 - Updating configurations from heartbeat INFO 2018-05-29 05:57:22,040 Controller.py:389 - Adding cancel/execution commands INFO 2018-05-29 05:57:22,040 Controller.py:475 - Waiting 9.9 for next heartbeat INFO 2018-05-29 05:57:31,941 Controller.py:482 - Wait for next heartbeat over WARNING 2018-05-29 05:57:46,099 base_alert.py:138 - [Alert][namenode_cpu] Unable to execute alert. [Alert][namenode_cpu] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,108 base_alert.py:138 - [Alert][datanode_health_summary] Unable to execute alert. [Alert][datanode_health_summary] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,119 base_alert.py:138 - [Alert][namenode_service_rpc_processing_latency_hourly] Unable to execute alert. Couldn't define hadoop_conf_dir: argument of type 'NoneType' is not iterable WARNING 2018-05-29 05:57:46,123 base_alert.py:138 - [Alert][namenode_client_rpc_queue_latency_hourly] Unable to execute alert. Couldn't define hadoop_conf_dir: argument of type 'NoneType' is not iterable WARNING 2018-05-29 05:57:46,136 base_alert.py:138 - [Alert][namenode_client_rpc_processing_latency_hourly] Unable to execute alert. Couldn't define hadoop_conf_dir: argument of type 'NoneType' is not iterable WARNING 2018-05-29 05:57:46,145 base_alert.py:138 - [Alert][namenode_directory_status] Unable to execute alert. [Alert][namenode_directory_status] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,267 base_alert.py:138 - [Alert][namenode_service_rpc_queue_latency_hourly] Unable to execute alert. Couldn't define hadoop_conf_dir: argument of type 'NoneType' is not iterable WARNING 2018-05-29 05:57:46,280 base_alert.py:138 - [Alert][yarn_resourcemanager_cpu] Unable to execute alert. [Alert][yarn_resourcemanager_cpu] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,282 base_alert.py:138 - [Alert][yarn_resourcemanager_rpc_latency] Unable to execute alert. [Alert][yarn_resourcemanager_rpc_latency] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,296 base_alert.py:138 - [Alert][smartsense_gateway_status] Unable to execute alert. [Alert][smartsense_gateway_status] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,298 logger.py:71 - Cannot find the stack name in the command. Stack tools cannot be loaded WARNING 2018-05-29 05:57:46,300 base_alert.py:138 - [Alert][smartsense_long_running_bundle] Unable to execute alert. [Alert][smartsense_long_running_bundle] Unable to extract JSON from JMX response WARNING 2018-05-29 05:57:46,298 logger.py:71 - Cannot find the stack name in the command. Stack tools cannot be loaded INFO 2018-05-29 05:57:46,303 logger.py:75 - call[('ambari-python-wrap', None, 'versions')] {} INFO 2018-05-29 05:57:46,303 logger.py:75 - call[('ambari-python-wrap', None, 'versions')] {} INFO 2018-05-29 05:57:46,712 logger.py:75 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist INFO 2018-05-29 05:57:46,712 logger.py:75 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist ERROR 2018-05-29 05:57:46,713 script_alert.py:123 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on hdpmaster'] ERROR 2018-05-29 05:57:46,713 script_alert.py:123 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on hdpmaster'] ... View more

Dear experts, I am running HDP 2.4 on AWS cluster. I am trying to decommission a datanode but it appears the status is stuck at "Decommissioning" for long time. If i try to delete the host the below error message is being displayed. Could you please help ? ... View more

Hi everyone, I am running HDP 2.5 sandbox in Azure cloud. After logging into ambari as admin and trying to add a service (Eg: Nifi) , i am getting the below error. Could you please help ? Error message: Error occured during stack advisor command invocation: Unable to delete directory /var/run/ambari-server/stack-recommendations/2. Below are few details of the sandbox: [root@sandbox ~]# sandbox-version Sandbox information: Created on: 25_10_2016_08_11_26 for Hadoop stack version: Hadoop 2.7.3.2.5.0.0-1245 Ambari Version: 2.4.0.0-1225 Ambari Hash: 59175b7aa1ddb74b85551c632e3ce42fed8f0c85 Ambari build: Release : 1225 Java version: 1.8.0_111 OS Version: CentOS release 6.8 (Final) [root@sandbox ~]# ps -ef | grep ambari-server root 1307 0 13 07:25 ? 00:06:46 /usr/lib/jvm/java/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer root 10677 10483 0 08:16 pts/1 00:00:00 grep ambari-server I have tried manually deleting the folders but no luck, [root@sandbox stack-recommendations]# ls -ltr total 24 drwxr-xr-x 1 root root 4096 Oct 20 03:54 2 drwxr-xr-x 1 root root 4096 Oct 25 03:54 3 drwxr-xr-x 1 root root 4096 Oct 26 06:29 1 drwxr-xr-x 1 root root 4096 Oct 30 08:29 4 drwxr-xr-x 1 root root 4096 Oct 30 08:29 5 drwxr-xr-x 1 root root 4096 Oct 30 08:29 6 [root@sandbox stack-recommendations]# pwd /var/run/ambari-server/stack-recommendations [root@sandbox stack-recommendations]# rmdir * rmdir: failed to remove `1': Invalid argument rmdir: failed to remove `2': Invalid argument rmdir: failed to remove `3': Invalid argument rmdir: failed to remove `4': Invalid argument rmdir: failed to remove `5': Invalid argument rmdir: failed to remove `6': Invalid argument ... View more