What is HDFS Ozone?
Ozone is an Object store for Hadoop. It is a redundant, distributed object store built by leveraging primitives present in HDFS. Below are some key features of ozone:
- A Hadoop compatible file system called Ozone File system that allows programs like Hive or Spark to run against Ozone without any modifications.
- Ozone supports RPC and REST API for accessing the store.
- Built to support billions of keys in distributed environment.
- Ozone can run concurrently with HDFS.
Like many other object stores, Ozone has a notion of volume. Only Administrators can create Volumes. Users create buckets in the volumes. To store data inside a bucket, users create keys.
An ozone file system allows other Hadoop ecosystem applications like Hive and Spark to use ozone. Once a bucket is created, it is trivial to create an ozone file system.
A 10-thousand foot view of Ozone
- OzoneManager (Om) acts as namespace manager. All ozone entities like volumes, buckets and keys are managed by Om. Om talks to an independent block manager (Storage Container Manager, SCM) to get blocks and passes it on to the Ozone client.
- SCM: Storage Container Manager is the block and cluster manager for Ozone.
- Block: Blocks are similar to blocks in HDFS. They are replicated blocks of data.
These components map very closely to the existing HDFS NameNode and DataNodes. The most significant difference is the presence of a block manager, SCM.
Using Ozone
The easiest way to run ozone is to try it out using the docker. To build Ozone from source, please checkout the hadoop sources from github. Then checkout the ozone branch, HDFS-7240 and build it.
git checkout HDFS-7240
You can build ozone by running the following build command.
mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist -Phdsl -Dtar -DskipShade
skipShade is just to make compilation faster and not really required.
Running Ozone via Docker
This assumes that you have a running docker setup on the machine. Please run following commands to see ozone in action.
- Go to the directory where the docker compose files exist.
cd hadoop-dist/target/compose/ozone
- Start ozone.
docker-compose up -d
- Log into the datanode container
docker exec -it ozone_datanode_1 bash
- Run the ozone load generator
./bin/oz freon
Take a look at OzoneManager UI, to see all the requests made by Freon http://localhost:9874/
Congratulations! on your first ozone deployment. In the next part of this tutorial we will cover oz command shell and look at how to use ozone to store files.
