About ZachRoes

ZachRoes · ‎01-17-2017

When Hive stores a timestamp value into Parquet format, it converts local time into UTC time, and when it reads data out, it converts back to local time.

ZachRoes · ‎01-17-2017

Yes. Use a spark-hbase-connector.

ZachRoes · ‎01-17-2017

Spark properties control most application parameters and can be set by using a SparkConf object, or through Java system properties. Environment variables can be used to set per-machine settings, such as the IP address, through the conf/spark-env.sh script on each node. Logging can be configured through log4j.properties.

ZachRoes · ‎01-17-2017

Amazon Elastic MapReduce (EMR) builds proprietary versions of Apache Hadoop, Hive, and Pig optimized for running on Amazon Web Services. Amazon EMR provides a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (EC2) or Simple Storage Service (S3)

ZachRoes · ‎01-17-2017

With a `SparkSession`, applications can create DataFrames from an [existing `RDD`](#interoperating-with-rdds), from a Hive table, or from [Spark data sources](#data-sources). As an example, the following creates a DataFrame based on the content of a JSON file: {% include_example create_df scala/org/apache/spark/examples/sql/SparkSQLExample.scala %}

ZachRoes · ‎01-17-2017

In Impala, a table can be created by using the ‘CREATE Table’ command. Let us understand the general syntax of creating a table in Impala with the help of the commands shown on the screen. The ‘PARTITIONED BY’ clause partitions data files based on one or more specified columns values.

ZachRoes · ‎01-17-2017

A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used ("before" in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come "before" the messages attribute.)

ZachRoes · ‎01-17-2017

The input data set consists of three tables as shown with the following table creation statements in Impala SQL dialect

ZachRoes · ‎01-17-2017

448 [ GB ] hdfs : //aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/mybigtable 8 [ GB ] hdfs : //aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/anotherone 0 [ GB ] hdfs : //aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/tinyone

ZachRoes · ‎01-17-2017

A smoke test is scripted, either using a written set of tests or an automated test A Smoke test is designed to touch every part of the application in a cursory way. It’s shallow and wide. Smoke testing is conducted to ensure whether the most crucial functions of a program are working, but not bothering with finer details. (Such as build verification). Smoke testing is normal health check up to a build of an application before taking it to testing in depth

Online	Offline
Last Visited	‎02-12-2017 10:22 AM

Date Registered	‎01-17-2017 04:43 PM
Last Visited	‎02-12-2017 10:22 AM
Total Messages Posted	20
Total Kudos Received	3

Re: Fire sanity check manually?

Re: Impala Timestamp - non-zero padded issue

Re: can we match HBase partitions with Impala Part...

Re: Config log4j in Spark

Re: Debug Spark program in Eclipse Data in AWS

Re: [CDH 5.7] Can't create table in Hive compatibl...

Re: Partitioning in Impala

Re: Pyspark: Table Dataframe returning empty recor...

Re: can we match HBase partitions with Impala Part...

Re: Can we check size of Hive tables? If so - how?

Re: Fire sanity check manually?