Try below- 1. stop the HDFS $HADOOP_HOME/sbin/stop-dfs.sh 2.Remove the temp folder. Check for the log to get the name dir 3. Set the name node and data node directories in hdfs-site.xml in your prefered location property> <name>dfs.namenode.name.dir</name> <value>file:/users/gangadharkadam/hadoopdata/hdfs/namenode</value> </property> <property> <name>dfs.datanode.name.dir</name> <value>file:/users/gangadharkadam/hadoopdata/hdfs/datanode</value> 4.Set the permissions on the new directories sudo chown gangadharkadam:staff /users/gangadharkadam/hadoopdata/hdfs/namenode sudo chmod 750 /users/gangadharkadam/hadoopdata/hdfs/namenode 5. Format the namnode hdfs dfs namenode -format 6. Start the HDFS again $HADOOP_HOME/sbin/start-dfs.sh 7. check the running daemons using jps -l Have good luck with your new HDFS 🙂 ... View more

Hive is trying to provide a value for the new column for those records where it did not exist, you need to specify a default for the new columns using 'avro.schema.literal table property. In the below example original tables has just one column. Age is added in the second version of schema. If the file that it’s reading is of a different schema, it will attempt to convert it using Avro schema resolution. The entire definition is in the avro.schema.literal property. ALTER TABLE test_avro SET TBLPROPERTIES ( 'avro.schema.literal'='{"name":"test_record", "type":"record", "fields": [ {"name":"full_name", "type":"string"}, {"name":"age", "type":"int", "default":999}]}'); Hope this helps. ... View more

--Register the jars REGISTER lib/parquet-pig-1.3.1.jar; REGISTER lib/parquet-column-1.3.1.jar; REGISTER lib/parquet-common-1.3.1.jar; REGISTER lib/parquet-format-2.0.0.jar; REGISTER lib/parquet-hadoop-1.3.1.jar; REGISTER lib/parquet-pig-1.3.1.jar; REGISTER lib/parquet-encoding-1.3.1.jar; --store in parquet format SET parquet.compression gzip or SNAPPY; STORE table INTO '/path/to/table' USING parquet.pig.ParquetStorer; -- options you might want to fiddle with SET parquet.page.size 1048576 -- default. this is your min read/write unit. SET parquet.block.size 134217728 -- default. your memory budget for buffering data SET parquet.compression lzo -- or you can use none, gzip, snappy STORE mydata into '/some/path' USING parquet.pig.ParquetStorer; --Reading mydata = LOAD '/some/path' USING parquet.pig.ParquetLoader AS (x: int, y int); ... View more

This message will pop up any time an application is requesting more resources from the cluster than the cluster can currently provide. What resources you might ask? Well Spark is only looking for two things: Cores and Ram. Cores represents the number of open executor slots that your cluster provides for execution. Ram refers to the amount of free Ram required on any worker running your application. Note for both of these resources the maximum value is not your System’s max, it is the max as set by the your Spark configuration. 1. Check out the current state of your cluster (and it’s free resources) at SparkMasterIP:7080 2.Make sure you have not started Spark Shell in 2 different terminals.The first Spark shell might consume all the available cores in the system leaving the second shell waiting for resources. Until the first spark shell is terminated and its resources are released, all other apps will display the above warning. The short term solution to this problem is to make sure you aren’t requesting more resources from your cluster than exist or to shut down any apps that are unnecessarily using resources. If you need to run multiple Spark apps simultaneously then you’ll need to adjust the amount of cores being used by each app. ... View more

is your hive Server is running in HTTP mode? Connection URL When HiveServer2 Is Running in HTTP Mode:- jdbc:hive2://<host>:<port>/<db>;transportMode=http;httpPath=<http_endpoint> where:- <http_endpoint> is the corresponding HTTP endpoint configured in hive-site.xml. Default value is cliservice. Default port for HTTP transport mode is 10001. ... View more