- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
What is the Path of hdfs.site.xml , core.xml ?
- Labels:
-
Cloudera Manager
-
HDFS
Created on ‎07-16-2014 07:05 AM - edited ‎09-16-2022 02:02 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello ,
I Installed HDFS using CM in my Linux system, I want to know the path of the hdfs.site.xml and hdfs.core.xml .
Thanks
Bala
Bala
Created ‎07-16-2014 12:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is important to understand the difference when using CM vs just CDH.
When using Cloudera Manager, configuration is stored in a central DB, upon startup of services on a target cluster node, cloudera manager passes, through the Agent on that host, the runtime configuration that should be used, and starts the processes, pointing to that runtime location.
This results in the actual services configuration being stored in a non-standard location of:
/var/run/cloudera-scm-agent/process/###-[service]-[SERVICE-ROLE]
The most recient 'instance" of a path is the current runtime config (use ls -ltr as root in the /var/run/cloudera-scm-agent/process path, the last being most current.
You can access the SAME information on a per service role instance basis, from the "process" tab. For example for hdfs
Cloudera Manager > Cluster > HDFS > Instances > (pick for example, the NameNode from the list)> Processes
You will see under "Configuration Files/Environment" a greater than (>) that you can click to expand and show all the current configs passed to the server, the same info in the path I describe above. This is handy as not all cluster administrators have root access to get to the indicated path.
The cloudera manager function of "Deploy client configuration" pushes the current configuration information, SPECIFIC TO CLIENT APPLICATIONS to the cluster hosts and defined gateway nodes, which end up in the default /etc/ locations you are used to from Hadoop and CDH documentation. Those locations will not have the complete configuration as used by the server, just values necessary for client applications (CLI, custom apps, etc) to use the cluster.
Todd
Created ‎07-16-2014 08:58 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Example:
/etc/hadoop/conf.cloudera.hdfs1/hdfs-site.xml
core-site.xml on the same path.
Created ‎07-16-2014 09:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The client configuration can be found under /etc/hadoop/conf but the
configuration used by various services can be different and is visible via
the CM web interface.
Created ‎07-16-2014 12:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is important to understand the difference when using CM vs just CDH.
When using Cloudera Manager, configuration is stored in a central DB, upon startup of services on a target cluster node, cloudera manager passes, through the Agent on that host, the runtime configuration that should be used, and starts the processes, pointing to that runtime location.
This results in the actual services configuration being stored in a non-standard location of:
/var/run/cloudera-scm-agent/process/###-[service]-[SERVICE-ROLE]
The most recient 'instance" of a path is the current runtime config (use ls -ltr as root in the /var/run/cloudera-scm-agent/process path, the last being most current.
You can access the SAME information on a per service role instance basis, from the "process" tab. For example for hdfs
Cloudera Manager > Cluster > HDFS > Instances > (pick for example, the NameNode from the list)> Processes
You will see under "Configuration Files/Environment" a greater than (>) that you can click to expand and show all the current configs passed to the server, the same info in the path I describe above. This is handy as not all cluster administrators have root access to get to the indicated path.
The cloudera manager function of "Deploy client configuration" pushes the current configuration information, SPECIFIC TO CLIENT APPLICATIONS to the cluster hosts and defined gateway nodes, which end up in the default /etc/ locations you are used to from Hadoop and CDH documentation. Those locations will not have the complete configuration as used by the server, just values necessary for client applications (CLI, custom apps, etc) to use the cluster.
Todd
Created ‎11-25-2014 08:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is an old post but I have a followup question to Tgrayson's excellent response.
These directories have the "client" configurations, is there a similar location where the server information is kept? I am trying to recreate and preserve the history of changes made to our cluster in order to baseline and, perhaps, roll-back.
Or is there a better way?
Thanks
Arthur
Created ‎11-25-2014 08:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Rollback is a function of the CM ui... You would never manually attempt rollback by manipulating anything in that path. The start/stop of the process would instantiate a new runtime config out of the SCM DB each time.
Created ‎11-25-2014 10:08 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When I said roll-back I was not referring to an automated one, obviously that is best handled by your system.
Created on ‎05-04-2018 08:02 AM - edited ‎05-04-2018 10:10 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Great Info.
Have one confusion.
Can we start/stop services manully which was insstaled using parcles?
If yes what are the configuration need to do?
