@revan Apache Hive Strengths: The Apache Hive facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop, it provides: Tools to enable easy data extract/transform/load (ETL) A mechanism to impose structure on a variety of data formats Access to files stored either directly in Apache HDFS or in other data storage systems such as Apache HBase Query execution via MapReduce Hive defines a simple SQL-like query language, called QL, that enables users familiar with SQL to query the data. At the same time, this language also allows programmers who are familiar with the MapReduce framework to be able to plug in their custom mappers and reducers to perform more sophisticated analysis that may not be supported by the built-in capabilities of the language. QL can also be extended with custom scalar functions (UDF's), aggregations (UDAF's), and table functions (UDTF's). Indexing to provide acceleration, index type including compaction and Bitmap index as of 0.10. Different storage types such as plain text, RCFile, HBase, ORC, and others. Metadata storage in an RDBMS, significantly reducing the time to perform semantic checks during query execution. Operating on compressed data stored into the Hadoop ecosystem using algorithms including DEFLATE, BWT, snappy, etc. Built-in user defined functions (UDFs) to manipulate dates, strings, and other data-mining tools. Hive supports extending the UDF set to handle use-cases not supported by built-in functions. SQL-like queries (HiveQL), which are implicitly converted into MapReduce, or Spark jobs. Apache Spark Strengths: Spark SQL has multiple interesting features: it supports multiple file formats such as Parquet, Avro, Text, JSON, ORC it supports data stored in HDFS, Apache HBase, Cassandra and Amazon S3 it supports classical Hadoop codecs such as snappy, lzo, gzip it provides security through authentification via the use of a "shared secret" (spark.authenticate=true on YARN, or spark.authenticate.secret on all nodes if not YARN) encryption, Spark supports SSL for Akka and HTTP protocols it supports UDFs it supports concurrent queries and manages the allocation of memory to the jobs (it is possible to specify the storage of RDD like in-memory only, disk only or memory and disk it supports caching data in memory using a SchemaRDD columnar format (cacheTable(““))exposing ByteBuffer, it can also use memory-only caching exposing User object it supports nested structures When to use Spark or Hive- Hive is still a great choice when low latency/multiuser support is not a requirement, such as for batch processing/ETL. Hive-on-Spark will narrow the time windows needed for such processing, but not to an extent that makes Hive suitable for BI Spark SQL, lets Spark users selectively use SQL constructs when writing Spark pipelines. It is not intended to be a general-purpose SQL layer for interactive/exploratory analysis. However, Spark SQL reuses the Hive frontend and metastore, giving you full compatibility with existing Hive data, queries, and UDFs. Spark SQL includes a cost-based optimizer, columnar storage and code generation to make queries fast. At the same time, it scales to thousands of nodes and multi hour queries using the Spark engine, which provides full mid-query fault tolerance. The performance is biggest advantage of Spark SQL. ... View more

Try below- 1. stop the HDFS $HADOOP_HOME/sbin/stop-dfs.sh 2.Remove the temp folder. Check for the log to get the name dir 3. Set the name node and data node directories in hdfs-site.xml in your prefered location property> <name>dfs.namenode.name.dir</name> <value>file:/users/gangadharkadam/hadoopdata/hdfs/namenode</value> </property> <property> <name>dfs.datanode.name.dir</name> <value>file:/users/gangadharkadam/hadoopdata/hdfs/datanode</value> 4.Set the permissions on the new directories sudo chown gangadharkadam:staff /users/gangadharkadam/hadoopdata/hdfs/namenode sudo chmod 750 /users/gangadharkadam/hadoopdata/hdfs/namenode 5. Format the namnode hdfs dfs namenode -format 6. Start the HDFS again $HADOOP_HOME/sbin/start-dfs.sh 7. check the running daemons using jps -l Have good luck with your new HDFS 🙂 ... View more

Hive is trying to provide a value for the new column for those records where it did not exist, you need to specify a default for the new columns using 'avro.schema.literal table property. In the below example original tables has just one column. Age is added in the second version of schema. If the file that it’s reading is of a different schema, it will attempt to convert it using Avro schema resolution. The entire definition is in the avro.schema.literal property. ALTER TABLE test_avro SET TBLPROPERTIES ( 'avro.schema.literal'='{"name":"test_record", "type":"record", "fields": [ {"name":"full_name", "type":"string"}, {"name":"age", "type":"int", "default":999}]}'); Hope this helps. ... View more

--Register the jars REGISTER lib/parquet-pig-1.3.1.jar; REGISTER lib/parquet-column-1.3.1.jar; REGISTER lib/parquet-common-1.3.1.jar; REGISTER lib/parquet-format-2.0.0.jar; REGISTER lib/parquet-hadoop-1.3.1.jar; REGISTER lib/parquet-pig-1.3.1.jar; REGISTER lib/parquet-encoding-1.3.1.jar; --store in parquet format SET parquet.compression gzip or SNAPPY; STORE table INTO '/path/to/table' USING parquet.pig.ParquetStorer; -- options you might want to fiddle with SET parquet.page.size 1048576 -- default. this is your min read/write unit. SET parquet.block.size 134217728 -- default. your memory budget for buffering data SET parquet.compression lzo -- or you can use none, gzip, snappy STORE mydata into '/some/path' USING parquet.pig.ParquetStorer; --Reading mydata = LOAD '/some/path' USING parquet.pig.ParquetLoader AS (x: int, y int); ... View more

This message will pop up any time an application is requesting more resources from the cluster than the cluster can currently provide. What resources you might ask? Well Spark is only looking for two things: Cores and Ram. Cores represents the number of open executor slots that your cluster provides for execution. Ram refers to the amount of free Ram required on any worker running your application. Note for both of these resources the maximum value is not your System’s max, it is the max as set by the your Spark configuration. 1. Check out the current state of your cluster (and it’s free resources) at SparkMasterIP:7080 2.Make sure you have not started Spark Shell in 2 different terminals.The first Spark shell might consume all the available cores in the system leaving the second shell waiting for resources. Until the first spark shell is terminated and its resources are released, all other apps will display the above warning. The short term solution to this problem is to make sure you aren’t requesting more resources from your cluster than exist or to shut down any apps that are unnecessarily using resources. If you need to run multiple Spark apps simultaneously then you’ll need to adjust the amount of cores being used by each app. ... View more

is your hive Server is running in HTTP mode? Connection URL When HiveServer2 Is Running in HTTP Mode:- jdbc:hive2://<host>:<port>/<db>;transportMode=http;httpPath=<http_endpoint> where:- <http_endpoint> is the corresponding HTTP endpoint configured in hive-site.xml. Default value is cliservice. Default port for HTTP transport mode is 10001. ... View more

@Nilesh Shrimant There is an issue with renaming partitions in hive 0.13 which is fixed in hive 0.14 I guess. Possible workaround - Setting fs.hdfs.impl.disable.cache=false and fs.file.impl.disable.cache=false Refer https://issues.apache.org/jira/browse/HIVE-7623 for more details. ... View more