Community Articles

VidyaSargur · ‎04-04-2024

In this article, I will demonstrate how to access the Ozone file system using Java API. I explain the steps to validate the Ozone cluster for volume, bucket, and key creation.

Let's begin and create a Java project with Maven build:

Create a Java project with the name AccessOFS using IntelliJ from File->New->Project.
The project structure will look like the below image:

Modify your pom.xml and add the dependencies for ozone-client. The pom.xml will be as follows.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.demo</groupId>
    <artifactId>AccessOFS</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.8.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.10</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdds-common</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-ozone-filesystem-hadoop3</artifactId>
            <version>1.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-ozone-filesystem-hadoop2</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdds-container-service</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdds-client</artifactId>
            <version>1.1.0</version>
        </dependency>
         <dependency>
            <groupId>org.apache.ozone</groupId>
            <artifactId>ozone-client</artifactId>
            <version>1.4.0</version>
        </dependency>
        <dependency>
            <groupId>com.jayway.jsonpath</groupId>
            <artifactId>json-path</artifactId>
            <version>2.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.10</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.10</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jul-to-slf4j</artifactId>
            <version>1.7.10</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.8</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>compile</scope>
        </dependency>
    </dependencies>
</project>

I have added the ozone-client dependencies below.

       <dependency>
            <groupId>org.apache.ozone</groupId>
            <artifactId>ozone-client</artifactId>
            <version>1.4.0</version>
        </dependency>

Create your AccessOzoneFS main class where we are trying to read the file from the local file system and store that in the ozone bucket:

The AccessOzoneFS class will be as follows:

package org.demo;

import org.apache.commons.io.FileUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.client.*;
import org.apache.hadoop.ozone.client.io.OzoneInputStream;
import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
import org.apache.hadoop.security.UserGroupInformation;

import java.io.File;

public class AccessOzoneFS {
    public static void main(String[] args) {
        try {
            OzoneConfiguration ozoneConfiguration = new OzoneConfiguration();
            String omLeaderAddress =  args[0];
            String omPrincipal =  args[1];
            String keytabPathLocal =  args[2];
            String volume =  args[3];
            String bucket =  args[4];
            String key =  args[5];
            String sourceFilePath =  args[6];
            Long dataSize =  Long.parseLong(args[7]);
            //set om leader node
            ozoneConfiguration.set("ozone.om.address", omLeaderAddress);
            // Setting kerberos authentication
            ozoneConfiguration.set("ozone.om.kerberos.principal.pattern", "*");
            ozoneConfiguration.set("ozone.security.enabled", "true");
            ozoneConfiguration.set("hadoop.rpc.protection", "privacy");
            ozoneConfiguration.set("hadoop.security.authentication", "kerberos");
            ozoneConfiguration.set("hadoop.security.authorization", "true");
            //Passing keytab for Authentication
            UserGroupInformation.setConfiguration(ozoneConfiguration);
            UserGroupInformation.loginUserFromKeytab(omPrincipal, keytabPathLocal);
            OzoneClient ozClient = OzoneClientFactory.getRpcClient(ozoneConfiguration);
            ObjectStore objectStore = ozClient.getObjectStore();
            // Let us create a volume to store buckets.
            objectStore.createVolume(volume);
            // Let us verify that the volume got created.
            OzoneVolume assets = objectStore.getVolume(volume);
            // Let us create a bucket called bucket.
            assets.createBucket(bucket);
            OzoneBucket video = assets.getBucket(bucket);
            // read data from the file, this is assumed to be a user provided function.
            byte[] videoData = FileUtils.readFileToByteArray(new File(sourceFilePath));
            // Create an output stream and write data.
            OzoneOutputStream videoStream = video.createKey(key, dataSize.longValue());
            videoStream.write(videoData);
            // Close the stream when it is done.
            videoStream.close();
            // We can use the same bucket to read the file that we just wrote, by creating an input Stream.
            // Let us allocate a byte array to hold the video first.
            byte[] data = new byte[(int) dataSize.longValue()];
            OzoneInputStream introStream = video.readKey(key);
            introStream.read(data);
            // Close the stream when it is done.
            introStream.close();
            // Close the client.
            ozClient.close();
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

Now we have the project setup in IntelliJ. Let's try a maven build from the directory where there is pom.xml.
```
AccessOFS % mvn clean install   
```
The build process will create a jar file under the project target directory which we will copy to the ozone cluster host under the /tmp directory.
```
target %  scp AccessOFS-1.0-SNAPSHOT.jar  user@ozoneclusterHost:/tmp
```
Create a file "demotest.mp4" of size 1 GB on the local file system under /tmp directory which you want to read and store under the ozone bucket.

Now, execute the jar file from the ozone cluster host by passing the values for the below parameters.

omLeaderAddress - OzoneManager leader address.
omPrincipal     - OzoneManager principal.
keytabPathLocal - OzoneManager keytab file with full path.
volume          - OzoneManager volume name you want to create.
bucket          - OzoneManager bucket name you want to create.
key             - OzoneManager key name you want to create.
sourceFilePath  - File name with full path you want to read.
dataSize        - Size of the chunk.

Use the below command and execute it from the ozone cluster host by passing suggested param values. (Please make sure your JDK directory is pointing correctly.

~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath  /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS omLeaderAddress omPrincipal keytabPathLocal volume bucket key sourceFilepath dataSize

Example:

~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath  /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS 10.140.204.133 om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE /var/run/cloudera-scm-agent/process/67-ozone-OZONE_MANAGER/ozone.keytab volume-sample bucket-sample key-sample /tmp/demotest.mp4 40749910

Now, let's validate volume, bucket, and key creation :

Validate volume creation?

ozone sh volume list o3://omLeaderHost:9862/

]# ozone sh volume list o3://10.140.204.133:9862/

{
  "metadata" : { },
  "name" : "s3v",
  "admin" : "om",
  "owner" : "om",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "usedNamespace" : 0,
  "creationTime" : "2024-04-02T08:21:12.980Z",
  "modificationTime" : "2024-04-02T08:21:12.980Z",
  "acls" : [ {
    "type" : "USER",
    "name" : "om",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  } ]
}
{
  "metadata" : { },
  "name" : "volume-sample",
  "admin" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
  "owner" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "usedNamespace" : 1,
  "creationTime" : "2024-04-04T05:28:52.580Z",
  "modificationTime" : "2024-04-04T05:28:52.580Z",
  "acls" : [ {
    "type" : "USER",
    "name" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  }, {
    "type" : "GROUP",
    "name" : "om",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  } ]
}

From the output, we noticed volume- "volume-sample" was created.

Validate bucket creation?

ozone sh bucket list o3://omLeaderHost:9862/volume-sample/

~]# ozone sh bucket list o3://10.140.204.133:9862/volume-sample/

{
  "metadata" : { },
  "volumeName" : "volume-sample",
  "name" : "bucket-sample",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 3221225472,
  "usedNamespace" : 1,
  "creationTime" : "2024-04-04T05:28:52.649Z",
  "modificationTime" : "2024-04-04T05:28:52.649Z",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1
}

From the output, we noticed bucket- "bucket-sample" was created.

Validate key creation?

ozone sh key list o3://omLeaderHost:9862/volume-sample/bucket-sample/

~]# ozone sh key list o3://10.140.204.133:9862/volume-sample/bucket-sample/

{
  "volumeName" : "volume-sample",
  "bucketName" : "bucket-sample",
  "name" : "key-sample",
  "dataSize" : 1073741824,
  "creationTime" : "2024-04-04T05:29:01.410Z",
  "modificationTime" : "2024-04-04T05:29:40.720Z",
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}

This way we can validate volume, bucket, and key creation using the AccessOFS jar which reads the file from the local file system and stores in the ozone bucket.

References

Java API Creating an Ozone client

Reference SourceCode
AccessOFS

Cloudera Community

Community Articles

How to access Ozone file system using Java API

Apache Ozone

Accessing AWS services using AWS Java SDK in Scala...

Access Kerberos cluster from JAVA using cached tic...

Best Practices: Linux File Systems for HDFS

Building a Custom Processor in Apache NiFi 1.5 for...

Use WASB as HDP 2.3.2 File System

Using the Schema Registry API

Using Spark to Virtually Integrate Hadoop with Ext...

How to use Ozone with AWS CLI

Change NiFi Flow Using Rest API - Part 1

Build and use Parquet-tools to read parquet files