Hi @Ed R, There is no such Hive syntax without a UDF, but there are some such UDFs available, and you can try to use them as is, or modify them if needed, like for example "to_map" described here. This is also assuming that your query is actually "select id1, id2, to_map(col1_key, col2_val) as result from <table> order by id1, id2" where <table> can either be a single table, or a union as you mentioned, and to_map is actaully a UDAF or User defined aggregate function. In this case you may want to decide how to handle cases where the same key points to different values. And finally, there is nothing misterious with Hive UDFs or UDAFs, you just implement some interfaces or overwrite some functions, and you have these examples to speed you up. As you go deeper into Hive, you will realize sooner or later that UDFs are the must. ... View more

No, not each job, you must be careful how do you, as I said "address Oozie". If you run Oozie commands from the command line you need to use the ozlb address as your Oozie server, or export OOZIE_URL=ozlb. And if you use Falcon, then when you define a logical cluster you need to use ozlb as your Oozie server address. ... View more

As a workaround, you can create an external table on /user/test1/csvfolder, and then insert records from this table into your internal table using "INSERT INTO TABLE tbl SELECT * FROM tbl_ext; However, creating external table requires write permission on your csvfolder, although the files there will be left intact. ... View more

First of all, you should not use "localhost" anywhere in your settings, but FQDNs. Let's call your Oozie HA servers oz1 and oz2, and your LB ozlb (these are all FQDNs). If you keep addressing Oozie by let's say oz1, then if oz1 goes down you lose Oozie functionality. That's the main reason to do HA, and the main reason to set OOZIE_BASE_URL=ozlb, because the LB can detect a failed Oozie server, and redirect the traffic to the other one. Of course, the LB is also doing load balancing, when both oz1 and oz2 are healthy (jobs on demand). HTH. ... View more

You can safely ignore this. Your app first tried to contact rm1 and found it to be in the Stand-By mode: "WARN ipc.Client: Failed to connect to server: str20/10.5.168.121:8032:". After that it failed over to rm2: "INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2". If rm1 is Active, there will be no such message. ... View more

Hi Leo, it seems that error can happen because of some miscommunication between the RM, an app's AM and one or more NMs. One scenario is described in YARN-3535, in particular in this post. How to fix it depends on what are you doing. If it's after an HDP upgrade you can try to clear affected NM recovery directories, and restart Yarn. If it happens on a particular app, then something might be wrong with app, like some configs missing, etc. ... View more

As the Jira says, stmk is rarely used, but the current version of Zookeeper, ver. 3.4.6 was added in HDP-2.x, so yes, 3.4.7 or higher which fixes the bug should be added soon. ... View more