Hi Cristian, the amount of memory that yarn can allocate is controlled by the setting "Memory allocated for all YARN containers on a node" under YARN. Set this to 3 GB, and that should give enough room for Tez to run (it needs 2.5 GB if you follow the settings above.). ... View more

Since you are in a sandbox, you need to reduce the amount of memory taken by each component so that they can fit (try running with 12 GB if you can). Reduce the memory footprint as follows: Tez container size = 1024 MB Map join, per Map memory = 256 MB Metastore heap = 512 MB Client heap = 512 MB Tez Client->tez.am.resource.memory.mb = 512 YARN will need to fit at least one Tez AM (512 MB) and a couple Tez containers (512MB *2). You can check how much memory is allocated to YARN on the YARN config page "Memory allocated for all YARN containers on a node". ... View more

@Pooja Sahu In the source file, the original '\001' character code has been replaced with the string representation "^A". One way to process the file is to convert it back to \001: CREATE EXTERNAL TABLE fix_raw (line string) ROW FORMAT DELIMITED LOCATION '/user/pooja/fix/'; CREATE TABLE fix_map (tag MAP<STRING, STRING>) STORED AS ORC; INSERT INTO TABLE fix_map SELECT str_to_map( replace(line, '^A', '\001'), '\001', '=') tag from fix_raw; -- query SELECT tag[49] FROM fix_map; ... View more

@Pooja Sahu Can you share a couple rows of the data set ? ... View more

In the second scenario, is it possible to copy the raw encrypted files from the first to the second cluster ? ... View more

The CREATE EXTERNAL TABLE statement must match the format on disk. If the files are in a self-describing format like parquet, you should not need to specify any table properties to read them (remove the TBLPROPERTIES line). If you want to convert to a new format, including a different compression algorithm, you will need to create a new table. ... View more