Cloudera Community
abhay_patil
user rank
user rank
Contributor

How to post HDP to CDP-DC migration using AMtoCM s...

by Contributor Contributor in Community Articles
‎10-05-2020 11:27 PM
‎10-05-2020 11:27 PM
In the CDP DC cluster, once you use the AMToCM migration script to migrate all configurations, you will not be able to fetch a yarn application logs from the command line when using the following command: $yarn logs -applicationId application_160120642 WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS. No class configured for IndexedFormat Can not find any log file matching the pattern: [ALL] for the application: application_1601206427676_0011 Can not find the logs for the application: application_1601206427676_0011 with the appOwner: hive This issue occurs because the yarn.log-aggregation.file-formats gets updated with the IndexedFormat configuration during the AMtoCM migration.   To resolve this issue, change it to the ifile and Tfile.  Once you change the configuration as mentioned in the above image, save and restart the yarn service, you will able to fetch the yarn logs using the command line.    Reference Log aggregation properties Log Aggregation File Controllers ... View more

Change spark history event log location.

by Contributor Contributor in Community Articles
‎07-26-2018 08:37 PM
‎07-26-2018 08:37 PM
Default location for spark event log history is hdfs:///spark-history (Spark) and hdfs:///spark2-history/ (spark2). This will help to debug spark-history page load issue or if you have huge count of event log files you can archive it by creating the new active location. Following are the steps to change this default location. 1. Create new directory on hdfs for e.g $hdfs dfs -mkdir /spark2-history_new $hdfs dfs -chown spark:hadoop /spark2-history_new 2. Login to Amabri==>Spark==>config. 3. Update following parameters with new path "hdfs:///spark2-history_new/ " as follow. 4. Save the configuration. 5. Restart Spark service to enable new changes. 6. Run spark job, your new event log file will get save in new location. Same you can view using the spark history UI. ... View more
Labels:

Re: How to remove all previous version HDP directo...

by Guru Guru in Community Articles
‎12-20-2018 06:10 PM
‎12-20-2018 06:10 PM
The Abouve Article doesnt work in ambari-2.7.3 due to a bug . Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py", line 119, in <module> RemovePreviousStacks().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py", line 49, in actionexecute self.remove_stack_version(structured_output, low_version) File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py", line 54, in remove_stack_version packages_to_remove = self.get_packages_to_remove(version) File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py", line 77, in get_packages_to_remove all_installed_packages = self.pkg_provider.all_installed_packages() AttributeError: 'YumManager' object has no attribute 'all_installed_packages' Please refer to article if you face the same bug : https://community.hortonworks.com/articles/230893/remove-old-stack-versions-script-doesnt-work-in-am.html ... View more
Contact Me
Online
Offline
Last Visited
‎11-05-2024 05:41 PM
Community Statistics
Member Since ‎04-19-2018 08:26 PM
Last Visited ‎11-05-2024 05:41 PM
Posts 13
Kudos received 1