@Jeff Watson
MapReduce the compute engine is never used and is not required in any way for Phoenix Spark adapter. The PhoenixConfigurationUtil class just helps to define the configuration of the RDD by leveraging the existing Phoenix input/output format definition.
Below is a snippet from org.apache.phoenix.spark.PhoenixRDD. The variable phoenixConf is defined using PhoenixConfigurationUtil class. There is no distributed compute, just serialization definition like record start/end and columns for DataFrame. It's just a way to explain to Spark how to turn a row in target Phoenix table into an RDD record.
def getPhoenixConfiguration: Configuration = {
// This is just simply not serializable, so don't try, but clone it because
// PhoenixConfigurationUtil mutates it.
val config = HBaseConfiguration.create(conf)
PhoenixConfigurationUtil.setInputClass(config, classOf[PhoenixRecordWritable])
PhoenixConfigurationUtil.setInputTableName(config, table)
if(!columns.isEmpty) {
PhoenixConfigurationUtil.setSelectColumnNames(config, columns.toArray)
}
if(predicate.isDefined) {
PhoenixConfigurationUtil.setInputTableConditions(config, predicate.get)
}
....phoenixConf is then used to create a new "custom" Hadoop API RDD using a method of SparkContext. This is where Spark will distribute all of the target rows of the Phoenix table into an RDD.
val phoenixRDD = sc.newAPIHadoopRDD(phoenixConf, classOf[PhoenixInputFormat[PhoenixRecordWritable]], classOf[NullWritable], classOf[PhoenixRecordWritable])
You do not need MapReduce installed on the cluster, just the util JAR in the classpath (included in the maven package of the Phoenix Spark adapter). All compute is handled by Spark. This is no different than needing the CSV reader JAR from Databricks in order to create a Spark RDD from a CSV file (snippet from...).
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") // Use first line of all files as header
.option("inferSchema", "true") // Automatically infer data types
.load("cars.csv")
val selectedData = df.select("year", "model")
selectedData.write
.format("com.databricks.spark.csv")
.option("header", "true")
.save("newcars.csv")It's just that the package com.databricks.spark.csv from Databricks was only designed for Spark and thus has the word spark in the package name. Hbase/Phoenix originated at a time when MapReduce was the compute main engine for Hadoop. Hence the package name has the word mapreduce in it. It's not that the MapReduce is a dependency, it's just that Engineering teams rarely go back and change package names.
