Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

access hadoop on virtual box from window

avatar
New Contributor

I am trying to run java code on window to access Hadoop 3.1.3 in virtualbox 6.1 with redhat 7.7.

the hadoop with all components working fine (NameNode,datanode,....).

 

19856 NameNode
20864 NodeManager
20274 SecondaryNameNode
20532 ResourceManager
20043 DataNode

 

 

I am trying to run java code on window to access Hadoop 3.1.3 in virtualbox 6.1 with redhat 7.7.

the hadoop with all components working fine (NameNode,datanode,....).

19856 NameNode
20864 NodeManager
20274 SecondaryNameNode
20532 ResourceManager
20043 DataNode

I am able to display  hadoop datanode and cluster from window after create port foreword VBox

Host IP 127.0.0.1 Gust IP 192.168.122.1

For the following ports:

9870
80888
9000
9866
9864
9869
9865
9867

 

Hadoop working fine and displayed from both host and gust

accessable from window on http://localhost:8088/cluster

accessable from Linux red hat on http://192.168.122.1:8088

vi4Fb

accessable from window browser on http://localhost:9870/

accessable from Linux red hat browser on http://192.168.122.1:9870

0k9cP

Linux /ect/hosts

######127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
####::1         localhost localhost.localdomain localhost6 localhost6.localdomain6q
192.168.122.1     master.hadoop      master

Virsualbox Network:

NAT

Java Code run Successfully on virtualbox (gust with hadoop) but cannot run successfully form windows (host)

import java.io.IOException;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class HadoopFileSystemCat {

    public static void main(String[] args) throws IOException {

        String uri = "hdfs://localhost:9000/TES";

        Configuration conf = new Configuration();

        conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
        FileSystem fs = FileSystem.get(URI.create(uri), conf);
        FSDataInputStream  in = null;
        try {
            in = fs.open(new Path(uri));
            System.out.println(fs.getFileBlockLocations(new Path(uri), 0, 0)[0]);
            System.out.println(fs.getFileBlockLocations(new Path(uri), 0, 0)[0].getNames()[0]);
           IOUtils.copyBytes(in, System.out, 4096, false);
        } finally {
            IOUtils.closeStream(in);
        }
    }
}

Code Output

0,10,localhost
192.168.122.1:9866

Exception Message

Exception in thread "main" org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-428104912-127.0.0.1-1586539199175:blk_1073741825_1001 file=/DDD
    at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:875)
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:858)
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:837)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:566)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:756)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:825)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
    at com.mycompany.hdfs_java_api.HadoopFileSystemCat.main(HadoopFileSystemCat.java:39)

actually block available as we can see through terminal

[muneeralnajdi@localhost hadoop]$ bin/hdfs dfs -ls  /
Found 1 items
-rw-r--r--   1 muneeralnajdi supergroup         10 2020-04-10 20:25 /DDD

Hadoop Configuration

Core-site.xsml

<configuration>
 <property>
<name>fs.default.name</name>
<value>hdfs://master.hadoop:9000</value>
</property>
</configuration>

yarn-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/muneeralnajdi/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/muneeralnajdi/hadoop_store/hdfs/datanode</value>
</property>
<property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>true</value>
    <description>Whether clients should use datanode hostnames when
      connecting to datanodes.
    </description>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>master.hadoop:9870</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>  
<value>file:/home/muneeralnajdi/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/muneeralnajdi/hadoop_store/hdfs/datanode</value>
</property>  
<property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>true</value> 
    <description>Whether clients should use datanode hostnames when
      connecting to datanodes. 
    </description>
</property> 
<property>
<name>dfs.namenode.http-address</name>
<value>master.hadoop:9870</value>
</property>
</configuration>

Could any one help me to fix this issue

I go through some answers

Access to file on Hadoop Virtualbox Cluster

But the java API still can not access the datanode block from windows

2 REPLIES 2

avatar
Community Manager

@muneeralnajdi Welcome to the community. While we appreciate the post, linking off to another forum to see the full context of the question will lessen your chances of receiving an answer here. Can you provide more details here so others will be able to understand the question and potentially assist?


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
New Contributor

@cjervis Thanks for comment and sorry for that as this is the first post.

I provide full details about the question.