Merge is not happening because you are writing with Spark, not through Hive, thus all these configurations don't apply. Here you might have two reasons causing the big amount of files: 1 - Spark has a default parallelism of 200 and it writes one file per partition, thus each Spark minibatch will write 200 files. This can be easily solved, especially if you are not writing a lot of data at each minibatch reducing the parallelism before writing using `coalesce` (eventually using 1 to write only 1 file per minibatch). 2 - Spark will anyway write (at least) one file per minibatch and this depends on the frequency you are scheduling them. In this case, the solution is to schedule periodically a CONCATENATE job (but be careful you might encounter HIVE-17280->HIVE-17403) or you can write your own application with your logic to do the concatenation. ... View more

That file is needed only for performance reason. It works like a cache. Otherwise, you have to upload the jars everytime an application starts. Your problem might be that you have a root folder in your tar.gz. In this case, if you list your files in the archive, you should see something like ./one.jar ./another.jar ... Instead, you should have no root folder, and listing the files should be: one.jar another.jar ... If this is the case, here you have some examples how to do it: https://stackoverflow.com/questions/939982/how-do-i-tar-a-directory-of-files-and-folders-without-including-the-directory-it. Hope this helps. ... View more

Here you have no problem in the log. It's only a warning. And this is because you have another Spark application running and then the port 4040 (which is the default port for the Spark Monitoring UI) is already in use. But then it will try on port 4041 and so on, until it finds a free port, as you can see in the following log, The only problem I can see is that you are specifying an invalid port number (100015 is not a valid port number, I guess you meant 10015). ... View more

I guess there are some errors in your DDL. The first one I can see is that location should be: array<struct<x: double, y: double>> Please try with this change and see whether it works or there are other problems. ... View more

If you read carefully the exception message, it says: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1] This is likely to be caused by some CR-LF characters inside your "msg" object. Hive interprets each line as a full JSON. If you JSON contains newlines, Hive cannot parse it. Thus you have to clean/reformat you data before being able to analyze it with Hive. ... View more

Hi, the best option is to get the ETL output to a structured DB for interactive reports. Spark has a high latency compared to them. You might also try to use SparkSQL caching the tables which have to be queried, but I'd not recommend this option. Spark Structured Streaming would be helpful for you for the ETL. I guess you are going to read in streaming the data from your source RDBMS and perform some transformation on the data and output it to a new RDBMS for your reporting purpose: this ETL application can be written using Spark Structured Streaming. Anyway, at the moment, Spark2 is still not supported in current HDP releases and Spark Structured Streaming is not perfect yet. So, if you have to start your project now, I would suggest you to write a simple SparkSQL application which you can run on Spark 1.6 and later on Spark 2 (when it will be supported) with very few changes. Thanks, Marco ... View more