@Ibrahim Jarrar You might be hitting the issue described in below HCC thread : https://community.hortonworks.com/questions/208893/hive-upgrade-tool-failed-at-hdp-3-upgrade.html . HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which seems to be fixed in HDP 2.6.2. So can you please try to upgrade to HDP-2.6.2/ HDP-2.6.5 first and then try the upgrade to HDP3. ... View more

@Zyann Have you following the steps mentioned in the below article to configure SPNEGO for NiFi? https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_registry-administration/content/kerberos_properties.html ... View more

@Victor L Looks like your NameNode is started but it is still in SafeMode hence it will not allow any mutable operations to be performed on HDFS. Thats the reason you are getting this SafeMode related error. In order to validate the same you can try creating a simple directory on HDFS and you might get the same error kind of # su - hdfs -c "hdfs dfs -mkdir /tmp/test" If you want to know if the NameNode is in SafeMode or not then you can run the following command: # su - hdfs -c "hdfs dfsadmin -safemode get" In order to tell NameNode to leave the safe mode you can run the following command: # su - hdfs -c "hdfs dfsadmin -safemode leave" If the NameNode still does not come out of SafeMode then you will need to try restarting the NameNode once and check the NameNode log for any error/warning. # su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh stop namenode" # su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode" # tail -f /var/log/hadoop/hdfs/hadoop-hdfs-namenod* . Please share the NN logs if it still does not start cleanly. . ... View more

@Daniel Zafar Can you please try to perform a "yum reinstall" on the mentioned package? # yum reinstall smartsense-hst -y . The above mentioned error can occur if the HST installation is corrupted or if it has missing files like: /usr/hdp/share/hst/hst-agent/lib/hst_agent/shell.py /usr/hdp/share/hst/hst-agent/lib/hst_agent/shell.pyc . Also please share the output of the following command to verify if you see the correct "sys.path" when you run the below command [python -m site] like: Example: # python -m site sys.path = [ '/root', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', ] USER_BASE: '/root/.local' (doesn't exist) USER_SITE: '/root/.local/lib/python2.7/site-packages' (doesn't exist) ENABLE_USER_SITE: True . ... View more

@Michael Bronson MapReduce uses RFA appender. Which is defined inside the "Advanced hdfs-log4j" Ambari UI --> HDFS --> Configs --> Advanced --> "Advanced hdfs-log4j" --> hdfs-log4j template (Text area) You will find something like this. By default you can see it like following: # # Rolling File Appender # log4j.appender.RFA=org.apache.log4j.RollingFileAppender log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file} # Logfile size and and 30-day backups log4j.appender.RFA.MaxFileSize={{hadoop_log_max_backup_size}}MB log4j.appender.RFA.MaxBackupIndex={{hadoop_log_number_of_backup_files}} log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n NOTE: However making changes to "RFA" appender will not be good because this appender is also being used by many other components. So if you want just want to limit the number of logfiles based on size for MapReduce hostory server only then you can try this : 1. Create a new Appender like "RFA_MAPRED" inside the somewhere just below the RFA appender which we posted above. Ambari UI --> HDFS --> Configs --> Advanced --> "Advanced hdfs-log4j" --> hdfs-log4j template (Text area) Add "RFA_MAPRED" as following: # # MapReduce Rolling File Appender (ADDED MANUALLY) # log4j.appender.RFA_MAPRED=org.apache.log4j.RollingFileAppender log4j.appender.RFA_MAPRED.File=${hadoop.log.dir}/${hadoop.log.file} log4j.appender.RFA_MAPRED.MaxFileSize=100MB log4j.appender.RFA_MAPRED.MaxBackupIndex=3 log4j.appender.RFA_MAPRED.layout=org.apache.log4j.PatternLayout log4j.appender.RFA_MAPRED.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n log4j.appender.RFA_MAPRED.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n 2. Now Save the config changes. Click Save Button. 3. Now we need to edit the "Advanced mapred-env" Ambari UI --> MapReduce --> Configs --> Advanced --> "Advanced mapred-env" --> mapred-env template Change the following line to use the "RFA_MAPRED" appender instead of default "RFA" (Single line change) Old value: export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA New value: export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA_MAPRED 4. Save and restart all required services to get this change reflected. 5. You can also verify that MapReduce is now using the correct Appender by running the following Command on Job History Server host. # ps -ef | grep JobHistoryServer . You should see -Dhadoop.root.logger=DEBUG,RFA_MAPRED in the output of the above command. ... View more