How to access Ozone file system using Java API

[#1]

In this article, I will demonstrate how to access the Ozone file system using Java API. I  explain the steps to validate the Ozone cluster for volume, bucket, and key creation.

Let's begin and create a Java project with Maven build: 

1. Create a Java project with the name AccessOFS using IntelliJ from File->New->Project.
   The project structure will look like the below image:
   
2. Modify your pom.xml and add the dependencies for ozone-client. The pom.xml will be as follows.

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
       <groupId>org.demo</groupId>
       <artifactId>AccessOFS</artifactId>
       <version>1.0-SNAPSHOT</version>
       <build>
           <plugins>
               <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                   <artifactId>maven-compiler-plugin</artifactId>
                   <configuration>
                       <source>8</source>
                       <target>8</target>
                   </configuration>
               </plugin>
           </plugins>
       </build>
   
       <dependencies>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-hdfs</artifactId>
               <version>2.8.0</version>
           </dependency>
           <dependency>
               <groupId>org.apache.commons</groupId>
               <artifactId>commons-lang3</artifactId>
               <version>3.10</version>
           </dependency>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-common</artifactId>
               <version>3.1.1</version>
           </dependency>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-hdds-common</artifactId>
               <version>1.1.0</version>
           </dependency>
   
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-ozone-filesystem-hadoop3</artifactId>
               <version>1.1.0</version>
           </dependency>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-ozone-filesystem-hadoop2</artifactId>
               <version>1.1.0</version>
           </dependency>
   
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-hdds-container-service</artifactId>
               <version>1.1.0</version>
           </dependency>
   
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-hdds-client</artifactId>
               <version>1.1.0</version>
           </dependency>
            <dependency>
               <groupId>org.apache.ozone</groupId>
               <artifactId>ozone-client</artifactId>
               <version>1.4.0</version>
           </dependency>
           <dependency>
               <groupId>com.jayway.jsonpath</groupId>
               <artifactId>json-path</artifactId>
               <version>2.5.0</version>
           </dependency>
           <dependency>
               <groupId>org.slf4j</groupId>
               <artifactId>slf4j-api</artifactId>
               <version>1.7.10</version>
           </dependency>
           <dependency>
               <groupId>org.slf4j</groupId>
               <artifactId>slf4j-log4j12</artifactId>
               <version>1.7.10</version>
           </dependency>
           <dependency>
               <groupId>org.slf4j</groupId>
               <artifactId>jul-to-slf4j</artifactId>
               <version>1.7.10</version>
           </dependency>
           <dependency>
               <groupId>junit</groupId>
               <artifactId>junit</artifactId>
               <version>4.12</version>
           </dependency>
           <dependency>
               <groupId>junit</groupId>
               <artifactId>junit</artifactId>
               <version>4.8</version>
           </dependency>
           <dependency>
               <groupId>junit</groupId>
               <artifactId>junit</artifactId>
               <version>4.10</version>
               <scope>compile</scope>
           </dependency>
       </dependencies>
   </project>​

   I have added the ozone-client dependencies below.

          <dependency>
               <groupId>org.apache.ozone</groupId>
               <artifactId>ozone-client</artifactId>
               <version>1.4.0</version>
           </dependency>​

3. Create your AccessOzoneFS main class where we are trying to read the file from the local file system and store that in the ozone bucket:
   The AccessOzoneFS class will be as follows:

   package org.demo;
   
   import org.apache.commons.io.FileUtils;
   import org.apache.hadoop.hdds.conf.OzoneConfiguration;
   import org.apache.hadoop.ozone.client.*;
   import org.apache.hadoop.ozone.client.io.OzoneInputStream;
   import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
   import org.apache.hadoop.security.UserGroupInformation;
   
   import java.io.File;
   
   public class AccessOzoneFS {
       public static void main(String[] args) {
           try {
               OzoneConfiguration ozoneConfiguration = new OzoneConfiguration();
               String omLeaderAddress =  args[0];
               String omPrincipal =  args[1];
               String keytabPathLocal =  args[2];
               String volume =  args[3];
               String bucket =  args[4];
               String key =  args[5];
               String sourceFilePath =  args[6];
               Long dataSize =  Long.parseLong(args[7]);
               //set om leader node
               ozoneConfiguration.set("ozone.om.address", omLeaderAddress);
               // Setting kerberos authentication
               ozoneConfiguration.set("ozone.om.kerberos.principal.pattern", "*");
               ozoneConfiguration.set("ozone.security.enabled", "true");
               ozoneConfiguration.set("hadoop.rpc.protection", "privacy");
               ozoneConfiguration.set("hadoop.security.authentication", "kerberos");
               ozoneConfiguration.set("hadoop.security.authorization", "true");
               //Passing keytab for Authentication
               UserGroupInformation.setConfiguration(ozoneConfiguration);
               UserGroupInformation.loginUserFromKeytab(omPrincipal, keytabPathLocal);
               OzoneClient ozClient = OzoneClientFactory.getRpcClient(ozoneConfiguration);
               ObjectStore objectStore = ozClient.getObjectStore();
               // Let us create a volume to store buckets.
               objectStore.createVolume(volume);
               // Let us verify that the volume got created.
               OzoneVolume assets = objectStore.getVolume(volume);
               // Let us create a bucket called bucket.
               assets.createBucket(bucket);
               OzoneBucket video = assets.getBucket(bucket);
               // read data from the file, this is assumed to be a user provided function.
               byte[] videoData = FileUtils.readFileToByteArray(new File(sourceFilePath));
               // Create an output stream and write data.
               OzoneOutputStream videoStream = video.createKey(key, dataSize.longValue());
               videoStream.write(videoData);
               // Close the stream when it is done.
               videoStream.close();
               // We can use the same bucket to read the file that we just wrote, by creating an input Stream.
               // Let us allocate a byte array to hold the video first.
               byte[] data = new byte[(int) dataSize.longValue()];
               OzoneInputStream introStream = video.readKey(key);
               introStream.read(data);
               // Close the stream when it is done.
               introStream.close();
               // Close the client.
               ozClient.close();
           } catch (Exception ex) {
               ex.printStackTrace();
           }
       }
   }​

4. Now we have the project setup in IntelliJ. Let's try a maven build from the directory where there is pom.xml.

   AccessOFS % mvn clean install   ​

   The build process will create a jar file under the project target directory which we will copy to the ozone cluster host under the /tmp directory.

   target %  scp AccessOFS-1.0-SNAPSHOT.jar  user@ozoneclusterHost:/tmp​

5. Create a file "demotest.mp4" of size 1 GB on the local file system under /tmp directory which you want to read and store under the ozone bucket.
6. Now, execute the jar file from the ozone cluster host by passing the values for the below parameters.

   omLeaderAddress - OzoneManager leader address.
   omPrincipal     - OzoneManager principal.
   keytabPathLocal - OzoneManager keytab file with full path.
   volume          - OzoneManager volume name you want to create.
   bucket          - OzoneManager bucket name you want to create.
   key             - OzoneManager key name you want to create.
   sourceFilePath  - File name with full path you want to read.
   dataSize        - Size of the chunk.​

7. Use the below command and execute it from the ozone cluster host by passing suggested param values. (Please make sure your JDK directory is pointing correctly.

   ~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath  /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS omLeaderAddress omPrincipal keytabPathLocal volume bucket key sourceFilepath dataSize​

   Example:

   ~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath  /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS 10.140.204.133 om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE /var/run/cloudera-scm-agent/process/67-ozone-OZONE_MANAGER/ozone.keytab volume-sample bucket-sample key-sample /tmp/demotest.mp4 40749910​

Now, let's validate volume, bucket, and key creation :

•  Validate volume creation? 
• ozone sh volume list o3://omLeaderHost:9862/

  ]# ozone sh volume list o3://10.140.204.133:9862/
  
  {
    "metadata" : { },
    "name" : "s3v",
    "admin" : "om",
    "owner" : "om",
    "quotaInBytes" : -1,
    "quotaInNamespace" : -1,
    "usedNamespace" : 0,
    "creationTime" : "2024-04-02T08:21:12.980Z",
    "modificationTime" : "2024-04-02T08:21:12.980Z",
    "acls" : [ {
      "type" : "USER",
      "name" : "om",
      "aclScope" : "ACCESS",
      "aclList" : [ "ALL" ]
    } ]
  }
  {
    "metadata" : { },
    "name" : "volume-sample",
    "admin" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
    "owner" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
    "quotaInBytes" : -1,
    "quotaInNamespace" : -1,
    "usedNamespace" : 1,
    "creationTime" : "2024-04-04T05:28:52.580Z",
    "modificationTime" : "2024-04-04T05:28:52.580Z",
    "acls" : [ {
      "type" : "USER",
      "name" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
      "aclScope" : "ACCESS",
      "aclList" : [ "ALL" ]
    }, {
      "type" : "GROUP",
      "name" : "om",
      "aclScope" : "ACCESS",
      "aclList" : [ "ALL" ]
    } ]
  }​

From the output, we noticed volume- "volume-sample" was created.

•      Validate bucket creation?
•      ozone sh bucket list o3://omLeaderHost:9862/volume-sample/

  ~]# ozone sh bucket list o3://10.140.204.133:9862/volume-sample/
  
  {
    "metadata" : { },
    "volumeName" : "volume-sample",
    "name" : "bucket-sample",
    "storageType" : "DISK",
    "versioning" : false,
    "usedBytes" : 3221225472,
    "usedNamespace" : 1,
    "creationTime" : "2024-04-04T05:28:52.649Z",
    "modificationTime" : "2024-04-04T05:28:52.649Z",
    "quotaInBytes" : -1,
    "quotaInNamespace" : -1
  }​

From the output, we noticed bucket- "bucket-sample" was created.

•      Validate key creation?
•     ozone sh key list o3://omLeaderHost:9862/volume-sample/bucket-sample/

  ~]# ozone sh key list o3://10.140.204.133:9862/volume-sample/bucket-sample/
  
  {
    "volumeName" : "volume-sample",
    "bucketName" : "bucket-sample",
    "name" : "key-sample",
    "dataSize" : 1073741824,
    "creationTime" : "2024-04-04T05:29:01.410Z",
    "modificationTime" : "2024-04-04T05:29:40.720Z",
    "replicationType" : "RATIS",
    "replicationFactor" : 3
  }​

This way we can validate volume, bucket, and key creation using the AccessOFS jar which reads the file from the local file system and stores in the ozone bucket.

References 

Java API Creating an Ozone client

Reference SourceCode  
AccessOFS