Support Questions
Find answers, ask questions, and share your expertise

connector hadoop to gpfs

Highlighted

connector hadoop to gpfs

Super Collaborator

Hi:

I need to connect the hadoop cluster into the GPFS Filesystem, and i read that i need to change this:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://mycluster</value>
</property>

actually i have this:

 fs.defaultFS=hdfs://hostname:8020

so, there is anithing to connect my cluster with gpfs???

<property>
  <name>fs.defaultFS</name>
  <value>gpfs://</value>
</property>

thanks

7 REPLIES 7
Highlighted

Re: connector hadoop to gpfs

I don't think it is that simple. I don't think that as of now gpfs provides an HDFS compatible layer you can simply connect to. This may come in the future. How GPFS worked until at least a year ago was that you needed a special set of client libraries that provide the connection to GPFS. ( i.e. you needed BigInsights ) So you could do a gpfs:// and then internally the client would do the necessary work to connect to gpfs instead.

Or in other words you need a GPFS compatible client with jars that support GPFS.

Not completely sure however. Things may have changed.

Highlighted

Re: connector hadoop to gpfs

Super Collaborator

Hi: Yes iam using BigInsights, but my question is, how ton send file from GPFS (IBM) to Cluster hadoop

Highlighted

Re: connector hadoop to gpfs

You have two clusters one BigInsights one HDP/Cloudera?

Distcp I would think from the BigInsights cluster. Or what do you mean with GPFS to Cluster hadoop.

Highlighted

Re: connector hadoop to gpfs

Super Collaborator

hi:

the manager on my HDP cluster is now conected to the GPFS Cluster, but I need to add some parameter into the core-site.xml like this:

My core-site config:
<configuration>
    <property>
    <name>hadoop.tmp.dir</name>
        <value>/tmp/hadoop</value>
    </property>
    <property>
    <name>fs.default.name</name>
        <value>gpfs:///</value>
    </property>
    <property>
    <name>fs.gpfs.impl</name>
        <value>org.apache.hadoop.fs.gpfs.GlobalParallelFileSystem</value>
    </property>
    <property>
    <name>gpfs.mount.dir</name>
        <value>/mnt/gpfs</value>
    </property>
</configuration>

But if i do this I will lost the hdfs point, so... any suggestions??

Highlighted

Re: connector hadoop to gpfs

Super Collaborator

Hi:

The GPFS Cluster is:

cluster GPFS de test.


 Node  Daemon node name  IP address    Admin node name  Designation
--------------------------------------------------------------------
   4   zlfor01.risa      10.1.232.12   zlfor01.risa     quorum-manager
   8   zlfor11.risa      10.1.232.74   zlfor11.risa     quorum-manager
   9   zlfor13.risa      10.1.232.78   zlfor13.risa     quorum-manager
  10   lnxbig05          10.1.246.19   lnxbig05


and the HDP Cluster is this:

		lnxbig01.cajarural.gcr
		lnxbig02.cajarural.gcr
		lnxbig03.cajarural.gcr
		lnxbig04.cajarural.gcr
		lnxbig05.cajarural.gcr
		lnxbig06.cajarural.gcr

So, i readed that i need to set this parameter into the core-xite.xml

gpfs.mount.dir 
fs.gpfs.impl 
fs.AbstractFileSystem.gpfs.impl 
gpfs.supergroup 


but i dont know if i need also set this value

<property>  
<name>fs.defaultFS</name>  
<value>gpfs://</value>
</property>

Please any help??

Highlighted

Re: connector hadoop to gpfs

Super Collaborator

Hi:

After all ill can send files fron gpfs to HDFS, but i cant see bouth in hdfs??? i mean, gpfs and hdfs bouth working with hadoop???

Re: connector hadoop to gpfs

New Contributor

@Robert Sancho Could you please explain the procedure and have you successfully moved the file from gpfs to hdfs?

,

@Robert Sancho : Could you please share the steps you followed to move file from gpfs to hdfs ? Can you see the files in hdfs and gpfs?