Regarding the questions asked: > What characters are allowed in aliases? e.g. is "!" allowed? In regards to Avro aliases, it follows the name rules[1], essentially: 1) Must start with [A-Za-z_] 2) Subsequently contain only [A-Za-z0-9_] > Is there anyway to get the alias information once it's loaded to Dataframe? There's a "printSchema" > API that lets you print the schema names, but there's not a counterpart for printing the aliases. Is > it possible to get the mapping from name to aliases from DF? Per our CDH 5.7 documentation[2], the spark-avro library strips all doc, aliases and other fields[3] when they are loaded into Spark. To work around this issue, we recommend to use the original name rather than the aliased name of the field in the table, as the Avro aliases are stripped during loading into Spark. References: [1] http://avro.apache.org/docs/1.8.1/spec.html#names [2] https://www.cloudera.com/documentation/enterprise/5-7-x/topics/spark_avro.html [3] Avro to Spark SQL Conversion: The spark-avro library supports conversion for all Avro data types: boolean -> BooleanType int -> IntegerType long -> LongType float -> FloatType double -> DoubleType bytes -> BinaryType string -> StringType record -> StructType enum -> StringType array -> ArrayType map -> MapType fixed -> BinaryType The spark-avro library supports the following union types: union(int, long) -> LongType union(float, double) -> DoubleType union(any, null) -> any The spark-avro library does not support complex union types. All doc, aliases, and other fields are stripped when they are loaded into Spark. ... View more

Hi Msun, Regarding your questions: > What characters are allowed in aliases? e.g. is "!" allowed? In regards to Avro aliases, it follows the name rules[1], essentially: 1) Must start with [A-Za-z_] 2) Subsequently contain only [A-Za-z0-9_] > Is there anyway to get the alias information once it's loaded to Dataframe? There's a "printSchema" > API that lets you print the schema names, but there's not a counterpart for printing the aliases. Is > it possible to get the mapping from name to aliases from DF? Per our CDH 5.7 documentation[2], the spark-avro library strips all doc, aliases and other fields[3] when they are loaded into Spark. To work around this issue, we recommend to use the original name rather than the aliased name of the field in the table, as the Avro aliases are stripped during loading into Spark. References: [1] http://avro.apache.org/docs/1.8.1/spec.html#names [2] https://www.cloudera.com/documentation/enterprise/5-7-x/topics/spark_avro.html [3] Avro to Spark SQL Conversion: The spark-avro library supports conversion for all Avro data types: boolean -> BooleanType int -> IntegerType long -> LongType float -> FloatType double -> DoubleType bytes -> BinaryType string -> StringType record -> StructType enum -> StringType array -> ArrayType map -> MapType fixed -> BinaryType The spark-avro library supports the following union types: union(int, long) -> LongType union(float, double) -> DoubleType union(any, null) -> any The spark-avro library does not support complex union types.   All doc, aliases, and other fields are stripped when they are loaded into Spark.   ... View more

Hi Jiro,   Thanks for bringing this to our attention.  It appears there's a bug with the validation for that property, in which the lower bound should be 0 and not 1.  We'll get this fixed in a future release.  In the meantime, you can accomplish setting mapreduce.shuffle.max.connections to 0 by setting it in a specific Safety Valve in Cloudera Manager.  Please follow the directions listed below:   Login to CM with admin privileges if you haven't done so already. Navigate to the YARN configuration page (CM -> Clusters -> YARN -> Configuration) Naviate to Gateway Default Group -> Advanced -> MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml In the field marked Value, please enter in the following entry: <property> <name>mapreduce.shuffle.max.connections</name> <value>0</value> </property> 5. Save changes then Deploy client configuration     Kind Regards,   Anthony   ... View more

Hi TS,   Responses inline below:   > Having said that the JobHistory Server is specific to Map Reduce jobs run on YARN, where other type of jobs will be shown? That will depend on what kind of application is being submitted to the YARN framework.     We know that if a MR2 job is submitted, the job details will be available while the job is running within the Resource Manager Web UI (as this is part of the YARN framework); When the job is completed, the job details will be available via the Job History Server.   If a Spark-on-YARN job was is submitted, the job details will still be availabile while the job is running within the Resource Manager Web UI, however when the job completes, the job details will then be available on the Spark History Server, which is a separate role/service that is configured when Spark-on-YARN if setup as a service in Cloudera Manager (or when configuring it in CDH, per our installation guide).   > Besides MR and Spark jobs, what other types of jobs can we launch via YARN? MR and Spark jobs are what is currently supported, however this may change in the future, as the need arises.  YARN is application agnostic and is intentionally designed to allow developers to create applications to run on its distributed framework.   Additonal details regarding YARN applications are available here, from this link.     > Are jobs moved from /tmp/logs/<user-id>/logs folder to /user/history/done & /user/history/done_intermediate ones? > Are they created simultaneously? To best clarify the answer, listed below is a brief overview of the order of operations of a MR job in YARN: 1) MR job submitted to RM from client 2) Application folder is created in /tmp/logs/<user-id>/logs/application_xxxxxxxxxxxx_xxxx 3) MR job runs in YARN on the cluster 4) MR job completes, counters from job are reported on job client that submitted job 5) Counter information (.jhist file) and job_conf.xml files are written to /user/history/done_intermediate/<user>/job_xxxxxxxxxx_xxxx* 6) .jist file and job_conf.xml are then moved from /user/history/done_intermediate/<user>/ to /user/history/done 7) Container logs from each Node Manager is aggregated into /tmp/logs/<user-id>/logs/application_xxxxxxxxxxxx_xxxx     Hope ths helps!     ... View more