Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

Hi,

mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

.Could anyone help me on this.
9 REPLIES 9

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

Super Mentor

@rmr1989

Please share the exact Ambari Version?

Also the exact API call that you made.

Also please check and share the ambari-server.log to see when you executed the API call .. was there any WARNING or ERROR logged?

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

Hi,

Ambari version is 2.2.2.2

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"HISTORYSERVER"}}] }' http://<ambarihost>:8080/api/v1/clusters/<clustername>/hosts?Hosts/host_name=<hostname>

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

HDP version is hdp2.4

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

While installing I didn't get any error, I am getting the below error when starting the history server from Ambari.

ClientConfigResourceProvider:416 - Execution of "ambari-python-wrap /var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py generate_configs /var/lib/ambari-server/data/tmp/MAPREDUCE2_CLIENT-configuration.json /var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package /var/lib/ambari-server/data/tmp/structured-out.json INFO /var/lib/ambari-server/data/tmp" returned 1.
java.util.concurrent.ExecutionException: Execution of "ambari-python-wrap /var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py generate_configs /var/lib/ambari-server/data/tmp/MAPREDUCE2_CLIENT-configuration.json /var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package /var/lib/ambari-server/data/tmp/structured-out.json INFO /var/lib/ambari-server/data/tmp" returned 1.

Caused by: java.lang.Throwable: Traceback (most recent call last):
File "/var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py", line 65, in <module>
MapReduce2Client().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 605, in generate_configs
import params
File "/var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/params.py", line 28, in <module>
from params_linux import *
File "/var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py", line 34, in <module>
import status_params
File "/var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py", line 47, in <module>
mapred_pid_dir = format("{mapred_pid_dir_prefix}/{mapred_user}")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 90, in format
return ConfigurationFormatter().format(format_string, args, **result)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 54, in format
result_protected = self.vformat(format_string, args, all_params)
File "/usr/lib64/python2.6/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
File "/usr/lib64/python2.6/string.py", line 582, in _vformat
result.append(self.format_field(obj, format_spec))
File "/usr/lib64/python2.6/string.py", line 599, in format_field
return format(value, format_spec)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 81, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'mapred-env' was not found in configurations dictionary!

... 97 more

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

Super Mentor

@rmr1989

I am not sure why it is missing. But as a temporary fix you can try to push the default values for the "mapred-env" with the help of "config.sh" utility as following:

1. Login to Ambari Server host using SSH

2. Copy the following content to a temp file like "/tmp/mapred-env.json"

"properties" : {
"content" : "\n# export JAVA_HOME=/home/y/libexec/jdk1.6.0/\n\nexport HADOOP_JOB_HISTORYSERVER_HEAPSIZE={{jobhistory_heapsize}}\n\nexport HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA\n\n#export HADOOP_JOB_HISTORYSERVER_OPTS=\n#export HADOOP_MAPRED_LOG_DIR=\"\" # Where log files are stored. $HADOOP_MAPRED_HOME/logs by default.\n#export HADOOP_JHS_LOGGER=INFO,RFA # Hadoop JobSummary logger.\n#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.\n#export HADOOP_MAPRED_IDENT_STRING= #A string representing this instance of hadoop. $USER by default\n#export HADOOP_MAPRED_NICENESS= #The scheduling priority for daemons. Defaults to 0.\nexport HADOOP_OPTS=\"-Dhdp.version=$HDP_VERSION $HADOOP_OPTS\"\nexport HADOOP_OPTS=\"-Djava.io.tmpdir={{hadoop_java_io_tmpdir}} $HADOOP_OPTS\"\nexport JAVA_LIBRARY_PATH=\"${JAVA_LIBRARY_PATH}:{{hadoop_java_io_tmpdir}}\"",
"jobhistory_heapsize" : "900",
"mapred_log_dir_prefix" : "/var/log/hadoop-mapreduce",
"mapred_pid_dir_prefix" : "/var/run/hadoop-mapreduce",
"mapred_user" : "mapred",
"mapred_user_nofile_limit" : "32768",
"mapred_user_nproc_limit" : "65536"
}


3. Now run the following command. Please make sure to replace the "ErieCluster" with your own cluster name. This command will push the mapred-env config to ambari.

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 -s set localhost ErieCluster mapred-env /tmp/mapred-env.json

.

Later based on your requierment you can change the values from the ambari UI.

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

Super Mentor

@rmr1989

GET Command : If you have any other working cluster of the same version then you can also get the default values for the "mapred-env" using the following command:

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080  get localhost ErieCluster mapred-env  /tmp/mapred-env.json

.

then you will see the values inside the file "/tmp/mapred-env.json" then in the same command you can use "set" to set the values back to the non working cluster.

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

Super Mentor

@rmr1989

As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.

Re: mapred-env is missing in Ambari when installing HistoryServer from Ambari REST API

New Contributor

Thanks..

It worked perfectly..

Don't have an account?
Coming from Hortonworks? Activate your account here