Maybe it's not the only issue, but you have to specify an alias for the subquery. Try doing this and let us know if you have other issues or the same error still remains... ... View more

You can simply use spark-submit, which is in the bin folder of your spark-client installation. Here you can find the documentation for it: http://spark.apache.org/docs/latest/submitting-applications.html ... View more

If you do just rm you're actually moving your data to the Trash. In order to remove the data from HDFS and free space, when you do the rm you have to put the flag -skipTrash. In order to delete the data from the trash, you can run: hdfs dfs -expunge ... View more

With Hive what you can do is reading those fields as string and validate them through regexp. Otherwise, if you are sure that you don't have NULL values, you can simply define your schema and if a int field is NULL, this means that it is not properly formatted. ... View more

That INFO only states that there were no TEZ session available and then a new one has to be created. But this is not an issue: the only problem is that you have a bit more time to start because it has to allocate the resources. The problem is more related to the Hive View which is not working properly showing you the results. Running the same query via beeline on command line you will see all the results. ... View more

I think that the best option for compiling scala Spark code is to use sbt,which is a tool for managing dependencies. You can do the same with Maven anyway, as you prefer. ... View more

You can simply use sqoop ... View more