In this tutorial, we will learn how to create Apache Ozone volumes, buckets, and keys. After that, we will see how we can access Apache Ozone data in Apache Spark. Ozone Create the volume with the name vol1 in Apache Ozone. # ozone sh volume create /vol1 21/08/25 06:23:27 INFO rpc.RpcClient: Creating Volume: vol1, with root as owner. Create the bucket with the name bucket1 under vol1. # ozone sh bucket create /vol1/bucket1 21/08/25 06:24:09 INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false Create the employee.csv file to upload to Ozone. # vi /tmp/employee.csv id,name,age 1,Ranga,33 2,Nishanth,4 3,Raja,60 Upload the employee.csv file to Ozone # ozone sh key put /vol1/bucket1/employee.csv /tmp/employee.csv Add the fs.o3fs.impl property to core-site.xml Go to Cloudera Manager > HDFS > Configuration > search for core-site.xml > Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml <property> <name>fs.o3fs.impl</name> <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value> </property> Display the files created earlier using 'hdfs' command. Note: Before running the following command, update the om-host.example.com value. hdfs dfs -ls o3fs://bucket1.vol1.om-host.example.com/ Spark Launch spark-shell spark spark-shell Run the following command to print the employee.csv file content. Note: Update the omHost value. scala> val omHost="om.host.example.com" scala> val df=spark.read.option("header", "true").option("inferSchema", "true").csv(s"o3fs://bucket1.vol1.${omHost}/employee.csv") scala> df.show() +---+--------+---+ | id| name|age| +---+--------+---+ | 1| Ranga| 33| | 2|Nishanth| 4| | 3| Raja| 60| +---+--------+---+​ Kerberized environment Pre-requisites: Create a user and provide proper Ranger permissions to create Ozone volume and buckets, etc. kinit with the user Steps: Create Ozone volumes, buckets, and keys mentioned in Ozone section. Launch spark-shell Replace the KEY_TAB, PRINCIPAL, and om.host.example.com in spark-shell spark-shell \ --keytab ${KEY_TAB} \ --principal ${PRINCIPAL} \ --conf spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.om.host.example.com:9862 Note: In a Kerberized environment, mandatorily, we need to specify the spark.yarn.access.hadoopFileSystems configuration, otherwise, it will display the following error: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]​ Run the following command to print the employee.csv file content. Note: Update the omHost value. scala> val omHost="om.host.example.com" scala> val df=spark.read.option("header", "true").option("inferSchema", "true").csv(s"o3fs://bucket1.vol1.${omHost}/employee.csv") scala> df.show() +---+--------+---+ | id| name|age| +---+--------+---+ | 1| Ranga| 33| | 2|Nishanth| 4| | 3| Raja| 60| +---+--------+---+ scala> val age30DF = df.filter(df("age") > 30) scala> val outputPath = s"o3fs://bucket1.vol1.${omHost}/employee_age30.csv" scala> age30DF.write.option("header", "true").mode("overwrite").csv(outputPath) scala> val df2=spark.read.option("header", "true").option("inferSchema", "true").csv(outputPath) scala> df2.show() +---+-----+---+ | id| name|age| +---+-----+---+ | 1|Ranga| 33| | 3| Raja| 60| +---+-----+---+ Note: If you get the java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.ozone.OzoneFileSystem not found, add the /opt/cloudera/parcels/CDH/jars/hadoop-ozone-filesystem-hadoop3-*.jar to spark class path using --jars option. Thanks for reading this article. If you liked this article, you can give kudos. ... View more

In this article, we will learn how to register a Hive UDFs using Spark HiveWarehouseSession. Download and build the Spark Hive UDF example. git clone https://github.com/rangareddy/spark-hive-udf cd spark-hive-udf mvn clean package -DskipTests Copy the target/spark-hive-udf-1.0.0-SNAPSHOT.jar to the edge node. Login to edge node and upload the spark-hive-udf-1.0.0-SNAPSHOT.jar to HDFS location for example, /tmp. hdfs dfs -put ./brickhouse-0.7.1-SNAPSHOT.jar /tmp Launch the spark-shell with 'hwc' parameters. spark-shell \ --jars /opt/cloudera/parcels/CDH/jars/hive-warehouse-connector-assembly-*.jar \ --conf spark.sql.hive.hiveserver2.jdbc.url='jdbc:hive2://hiveserver2_host1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \ --conf spark.sql.hive.hwc.execution.mode=spark \ --conf spark.datasource.hive.warehouse.metastoreUri='thrift://metastore_host:9083' \ --conf spark.datasource.hive.warehouse.load.staging.dir='/tmp' \ --conf spark.datasource.hive.warehouse.user.name=hive \ --conf spark.datasource.hive.warehouse.password=hive \ --conf spark.datasource.hive.warehouse.smartExecution=false \ --conf spark.datasource.hive.warehouse.read.via.llap=false \ --conf spark.datasource.hive.warehouse.read.jdbc.mode=cluster \ --conf spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V2 \ --conf spark.security.credentials.hiveserver2.enabled=false \ --conf spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions Create the HiveWarehouseSession. import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession._ val hive = HiveWarehouseSession.session(spark).build() Execute the following statement to register a Hive UDF. hive.executeUpdate("CREATE FUNCTION uppercase AS 'com.ranga.spark.hive.udf.UpperCaseUDF' USING JAR 'hdfs:///tmp/spark-hive-udf-1.0.0-SNAPSHOT.jar'") Test the registered function, for example, uppercase. scala> val data1 = hive.executeQuery("select id, uppercase(name), age, salary from employee") scala> data1.show() +---+-----------------------+---+---------+ | id|default.uppercase(name)|age| salary| +---+-----------------------+---+---------+ | 1| RANGA| 32| 245000.3| | 2| NISHANTH| 2| 345000.1| | 3| RAJA| 32|245000.86| | 4| MANI| 14| 45000.0| +---+-----------------------+---+---------+ Thanks for reading this article. ... View more

In this article, we will learn to pass atlas-application.properties configuration file from a different location in spark-submit command. When Atlas service is enabled in CDP, and we run Spark application by default, atlas-application.properties file is picked from /etc/spark/conf.cloudera.spark_on_yarn/ directory. Let's test with SparkPi example:   spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 10   We can see the following output in the application log.   21/08/23 06:12:03 INFO atlas.ApplicationProperties: Looking for atlas-application.properties in classpath 21/08/23 06:12:03 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/etc/spark/conf.cloudera.spark_on_yarn/atlas-application.properties   If we want to pass the atlas-application.properties configuration file from a different location, for example /tmp directory, copy the atlas-application.properties from /etc/spark/conf.cloudera.spark_on_yarn to /tmp directory and pass it using -Datlas.conf=/tmp/ variable in spark-submit.   Let's test with same SparkPi example by adding --driver-java-options="-Datlas.conf=/tmp/" property to the spark-submit.   spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --driver-java-options="-Datlas.conf=/tmp/" /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 10   We can see the following output in the application log.   21/08/05 14:36:24 INFO atlas.ApplicationProperties: Looking for atlas-application.properties in classpath 21/08/05 14:36:24 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/tmp/atlas-application.properties   In order to run the same SparkPi example in cluster mode, we need to place the atlas-application.properties file in all nodes /tmp directory and run the Spark application as follows:   spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster \ --files /tmp/atlas-application.properties#atlas-application.properties --driver-java-options="-Datlas.conf=/tmp/" \ /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 10   or,    sudo -u spark spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster \ --files /tmp/atlas-application.properties --conf spark.driver.extraJavaOptions="-Datlas.conf=./" \ /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 10   We can see the following output:   21/08/23 06:12:07 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/data1/tmp/usercache/spark/appcache/application_1629693759177_0016/container_e74_1629693759177_0016_01_000001/./atlas-application.properties   ... View more