SYMPTOM: Access Audit logs show 6 hours behind from Central Timezone. This is related to https://issues.apache.org/jira/browse/RANGER-336 ERROR: Ranger Access Audit logs show 6 hours behind from Central Timezone. ROOT CAUSE: This is bug - https://issues.apache.org/jira/browse/RANGER-336 RESOLUTION: Create a file name "ranger-admin-env-javaopts.sh" with below entry in path "/usr/hdp/current/ranger-admin/conf/" - export JAVA_OPTS=" ${JAVA_OPTS} -Duser.timezone=UTC" Save the file and restart Ranger admin service. ... View more

SYMPTOM: Ambari agent not able to register with Ambari server. ERROR: Below is the error logs - ERROR 2016-12-19 10:17:54,387 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0) UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byte ERROR 2016-12-19 10:17:54,388 Controller.py:195 - Error:'utf8' codec can't decode byte 0xac in position 928: invalid start byte WARNING 2016-12-19 10:17:54,388 Controller.py:196 - Sleeping for 11 seconds and then trying again ERROR 2016-12-19 10:18:05,686 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0) UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byte ROOT CAUSE: This is bug - https://hortonworks.jira.com/browse/BUG-52919 RESOLUTION: Create /usr/lib/python2.6/site-packages/sitecustomize.py file with the below content, restart ambari-agent. import sys sys.setdefaultencoding('utf-8’) Restart ambari-agent. ... View more

SYMPTOM: While performing step for Namenode HA, on the step of 'hdfs namenode -initializeSharedEdits' it failed with below error - ERROR: [root@localhost conf]# sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits' 16/11/22 09:43:56 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = locahost/10.200.206.12 STARTUP_MSG: args = [-initializeSharedEdits] STARTUP_MSG: version = 2.7.1.2.3.0.0-2557 STARTUP_MSG: classpath = /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/commons-cli-1.2.jar:/us r/hdp/2.3.0.0-2557/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.0.0-2557/ . . . STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1; compiled by 'jenkins' on 2015-07-14T13:08Z STARTUP_MSG: java = 1.8.0_51 ************************************************************/ 16/11/22 09:43:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 16/11/22 09:43:56 INFO namenode.NameNode: createNameNode [-initializeSharedEdits] 16/11/22 09:43:56 ERROR namenode.NameNode: No shared edits directory configured for namespace null namenode null 16/11/22 09:43:56 INFO util.ExitUtil: Exiting with status 0 16/11/22 09:43:56 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost/10.200.206.12 ************************************************************/ ROOT CAUSE: While performing NN Ha we found that Step 5 "Configure Components" is getting executed in a second, which was issue suspected. It was not stopping services and performing desired steps mentioned in screenshot below - RESOLUTION: Suspected that Ambari server was holding cache in DB. Clearing the Ambari server cache using below steps resolved the issue - # log out from the ambari server # list persisted k-v(ui cache) $curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist # get the cached state: $curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist/CLUSTER_CURRENT_STATUS # reset/clean the cache $curl -u admin:admin -H 'X-Requested-By:ambari' -d '{"CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"DEFAULT\"}" }' -X POST 'http://localhost:8080/api/v1/persist' ... View more