Member since
02-12-2021
133
Posts
0
Kudos Received
0
Solutions
04-04-2024
05:50 AM
In this article, I will demonstrate how to access the Ozone file system using Java API. I explain the steps to validate the Ozone cluster for volume, bucket, and key creation.
Let's begin and create a Java project with Maven build:
Create a Java project with the name AccessOFS using IntelliJ from File->New->Project. The project structure will look like the below image:
Modify your pom.xml and add the dependencies for ozone-client. The pom.xml will be as follows. <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.demo</groupId>
<artifactId>AccessOFS</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.8.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.10</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdds-common</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-ozone-filesystem-hadoop3</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-ozone-filesystem-hadoop2</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdds-container-service</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdds-client</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.ozone</groupId>
<artifactId>ozone-client</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.10</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.10</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
<version>1.7.10</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project> I have added the ozone-client dependencies below. <dependency>
<groupId>org.apache.ozone</groupId>
<artifactId>ozone-client</artifactId>
<version>1.4.0</version>
</dependency>
Create your AccessOzoneFS main class where we are trying to read the file from the local file system and store that in the ozone bucket: The AccessOzoneFS class will be as follows: package org.demo;
import org.apache.commons.io.FileUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.client.*;
import org.apache.hadoop.ozone.client.io.OzoneInputStream;
import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
import org.apache.hadoop.security.UserGroupInformation;
import java.io.File;
public class AccessOzoneFS {
public static void main(String[] args) {
try {
OzoneConfiguration ozoneConfiguration = new OzoneConfiguration();
String omLeaderAddress = args[0];
String omPrincipal = args[1];
String keytabPathLocal = args[2];
String volume = args[3];
String bucket = args[4];
String key = args[5];
String sourceFilePath = args[6];
Long dataSize = Long.parseLong(args[7]);
//set om leader node
ozoneConfiguration.set("ozone.om.address", omLeaderAddress);
// Setting kerberos authentication
ozoneConfiguration.set("ozone.om.kerberos.principal.pattern", "*");
ozoneConfiguration.set("ozone.security.enabled", "true");
ozoneConfiguration.set("hadoop.rpc.protection", "privacy");
ozoneConfiguration.set("hadoop.security.authentication", "kerberos");
ozoneConfiguration.set("hadoop.security.authorization", "true");
//Passing keytab for Authentication
UserGroupInformation.setConfiguration(ozoneConfiguration);
UserGroupInformation.loginUserFromKeytab(omPrincipal, keytabPathLocal);
OzoneClient ozClient = OzoneClientFactory.getRpcClient(ozoneConfiguration);
ObjectStore objectStore = ozClient.getObjectStore();
// Let us create a volume to store buckets.
objectStore.createVolume(volume);
// Let us verify that the volume got created.
OzoneVolume assets = objectStore.getVolume(volume);
// Let us create a bucket called bucket.
assets.createBucket(bucket);
OzoneBucket video = assets.getBucket(bucket);
// read data from the file, this is assumed to be a user provided function.
byte[] videoData = FileUtils.readFileToByteArray(new File(sourceFilePath));
// Create an output stream and write data.
OzoneOutputStream videoStream = video.createKey(key, dataSize.longValue());
videoStream.write(videoData);
// Close the stream when it is done.
videoStream.close();
// We can use the same bucket to read the file that we just wrote, by creating an input Stream.
// Let us allocate a byte array to hold the video first.
byte[] data = new byte[(int) dataSize.longValue()];
OzoneInputStream introStream = video.readKey(key);
introStream.read(data);
// Close the stream when it is done.
introStream.close();
// Close the client.
ozClient.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
Now we have the project setup in IntelliJ. Let's try a maven build from the directory where there is pom.xml. AccessOFS % mvn clean install The build process will create a jar file under the project target directory which we will copy to the ozone cluster host under the /tmp directory. target % scp AccessOFS-1.0-SNAPSHOT.jar user@ozoneclusterHost:/tmp
Create a file "demotest.mp4" of size 1 GB on the local file system under /tmp directory which you want to read and store under the ozone bucket.
Now, execute the jar file from the ozone cluster host by passing the values for the below parameters. omLeaderAddress - OzoneManager leader address.
omPrincipal - OzoneManager principal.
keytabPathLocal - OzoneManager keytab file with full path.
volume - OzoneManager volume name you want to create.
bucket - OzoneManager bucket name you want to create.
key - OzoneManager key name you want to create.
sourceFilePath - File name with full path you want to read.
dataSize - Size of the chunk.
Use the below command and execute it from the ozone cluster host by passing suggested param values. (Please make sure your JDK directory is pointing correctly. ~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS omLeaderAddress omPrincipal keytabPathLocal volume bucket key sourceFilepath dataSize Example: ~]# /usr/java/jdk1.8.0_232-cloudera/bin/java -classpath /tmp/AccessOFS-1.0-SNAPSHOT.jar:`ozone classpath hadoop-ozone-ozone-manager` org.demo.AccessOzoneFS 10.140.204.133 om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE /var/run/cloudera-scm-agent/process/67-ozone-OZONE_MANAGER/ozone.keytab volume-sample bucket-sample key-sample /tmp/demotest.mp4 40749910
Now, let's validate volume, bucket, and key creation :
Validate volume creation?
ozone sh volume list o3://omLeaderHost:9862/ ]# ozone sh volume list o3://10.140.204.133:9862/
{
"metadata" : { },
"name" : "s3v",
"admin" : "om",
"owner" : "om",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"usedNamespace" : 0,
"creationTime" : "2024-04-02T08:21:12.980Z",
"modificationTime" : "2024-04-02T08:21:12.980Z",
"acls" : [ {
"type" : "USER",
"name" : "om",
"aclScope" : "ACCESS",
"aclList" : [ "ALL" ]
} ]
}
{
"metadata" : { },
"name" : "volume-sample",
"admin" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
"owner" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"usedNamespace" : 1,
"creationTime" : "2024-04-04T05:28:52.580Z",
"modificationTime" : "2024-04-04T05:28:52.580Z",
"acls" : [ {
"type" : "USER",
"name" : "om/ccycloud-2.ni-717x-yn.root.comops.site@ROOT.COMOPS.SITE",
"aclScope" : "ACCESS",
"aclList" : [ "ALL" ]
}, {
"type" : "GROUP",
"name" : "om",
"aclScope" : "ACCESS",
"aclList" : [ "ALL" ]
} ]
}
From the output, we noticed volume- "volume-sample" was created.
Validate bucket creation?
ozone sh bucket list o3://omLeaderHost:9862/volume-sample/ ~]# ozone sh bucket list o3://10.140.204.133:9862/volume-sample/
{
"metadata" : { },
"volumeName" : "volume-sample",
"name" : "bucket-sample",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 3221225472,
"usedNamespace" : 1,
"creationTime" : "2024-04-04T05:28:52.649Z",
"modificationTime" : "2024-04-04T05:28:52.649Z",
"quotaInBytes" : -1,
"quotaInNamespace" : -1
}
From the output, we noticed bucket- "bucket-sample" was created.
Validate key creation?
ozone sh key list o3://omLeaderHost:9862/volume-sample/bucket-sample/ ~]# ozone sh key list o3://10.140.204.133:9862/volume-sample/bucket-sample/
{
"volumeName" : "volume-sample",
"bucketName" : "bucket-sample",
"name" : "key-sample",
"dataSize" : 1073741824,
"creationTime" : "2024-04-04T05:29:01.410Z",
"modificationTime" : "2024-04-04T05:29:40.720Z",
"replicationType" : "RATIS",
"replicationFactor" : 3
}
This way we can validate volume, bucket, and key creation using the AccessOFS jar which reads the file from the local file system and stores in the ozone bucket.
References
Java API Creating an Ozone client
Reference SourceCode AccessOFS
... View more
Labels:
01-23-2024
07:10 AM
Hi @parimalpatil The RedactorAppender is mostly you can ignore it is nothing to do with real failure unless the stacktraces at bottom points something related to any ozone roles.
This Log4j Appender redacts log messages using redaction rules before delegating to other Appenders. You can share the complete failure log so that we can check and update you. The workaround is add jar file in classpath of roles where you see RedactorAppender error. We can add this through CM UI -> Configuration-> Search "role_env_safety_valve" for the role you are getting error. OZONE_CLASSPATH=$OZONE_CLASSPATH:/opt/cloudera/parcels/CDH/jars/logredactor-2.0.8.jar
... View more