Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

which properties in hdfs-site.xml are ready by Impalad?

Highlighted

which properties in hdfs-site.xml are ready by Impalad?

Contributor

Hello, I have some question about setting up Impala and Datanode config.

 

Context

- using CDH 5.16.2

- Datanode

- CentOS 6.9

 

Question1:

Does `hdfs-site.xml` have to be copied to /etc/impala/conf/ and modified?

OR, can `hdfs-site.xml` be created with these config is enough like this?

 

<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>

<property>
    <name>dfs.domain.socket.path</name>
    <value>/var/lib/hadoop-hdfs/socket._PORT</value>
</property>

<property>
    <name>dfs.client.file-block-storage-locations.timeout.millis</name>
    <value>10000</value>
</property>

<property>
  <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
  <value>true</value>
</property> 

 

 

This is /etc/hadoop/conf/hdfs-site.xml

 

<configuration>
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.domain.socket.path</name>
    <value>/var/lib/hadoop-hdfs/socket._PORT</value>
  </property>
  <property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.client.read.shortcircuit.skip.checksum</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.client.file-block-storage-locations.timeout</name>
    <value>10000</value>
  </property>
  <property>
    <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.block.size</name>
    <value>134217728</value>
  </property>
  <property>
    <name>dfs.datanode.du.reserved</name>
    <value>1073741824</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data,/data5/hadoop/hdfs/data,/data6/hadoop/hdfs/data,/data7/hadoop/hdfs/data,/data8/hadoop/hdfs/data,/data9/hadoop/hdfs/data</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>/data0/hadoop/hdfs/name,/data1/hadoop/hdfs/name,/data2/hadoop/hdfs/name,/data3/hadoop/hdfs/name,/data4/hadoop/hdfs/name,/data5/hadoop/hdfs/name,/data6/hadoop/hdfs/name,/data7/hadoop/hdfs/name,/data8/hadoop/hdfs/name,/data9/hadoop/hdfs/name</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>/data0/hadoop/hdfs/secondary,/data1/hadoop/hdfs/secondary,/data2/hadoop/hdfs/secondary,/data3/hadoop/hdfs/secondary,/data4/hadoop/hdfs/secondary,/data5/hadoop/hdfs/secondary,/data6/hadoop/hdfs/secondary,/data7/hadoop/hdfs/secondary,/data8/hadoop/hdfs/secondary,/data9/hadoop/hdfs/secondary</value>
  </property>
  <property>
    <name>dfs.permissions.superusergroup</name>
    <value>hadoop</value>
  </property>
  <property>
      <name>dfs.client.use.datanode.hostname</name>
      <value>true</value>
  </property>
</configuration>

 

Question2:

Which value of properties in /etc/impala/conf/hdfs-site.xml will be read by Impala?

The reason of asking this question is that what if there is different value for `dfs.data.dir` property

like /etc/impala/conf/hdfs-site.xml has

 

...
  <property>
    <name>dfs.data.dir</name>
<value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data</value>
  </property>
...

 

 

but, /etc/hadoop/conf/hdfs-site.xml has

 

...
  <property>
    <name>dfs.data.dir</name>
<value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data,/data5/hadoop/hdfs/data,/data6/hadoop/hdfs/data,/data7/hadoop/hdfs/data,/data8/hadoop/hdfs/data,/data9/hadoop/hdfs/data</value>
  </property>
...

 

 

Can this make any performance issue?

 

Thank you for your comment/reply in advance

Don't have an account?
Coming from Hortonworks? Activate your account here