This is a known issue and has been fixed in CM 6.2. Here is the relevant item in the release notes.   Cheers, Lars ... View more

This sounds like a result of the drastically increased link latency between your two "racks". While within a single rack you should see latencies less than a millisecond, US-EU latencies will be around 150ms, depending on where in the US and EU your machines are located. Bandwidth between your locations is likely also much lower than between the racks.   Impala currently does not do any rack-aware scheduling of I/O and data exchanges. In addition it is not optimized for high variance in link latencies and throughput. HDFS itself to my knowledge also makes no optimizations for such a case.   Frankly, I don't think you will see good performance in such a scenario. If you want to increase data availability, you could explore replicating the data between your locations while running queries in only one at a time. If you want to increase service availability, you can look into using a load balancer and switching from one cluster to the other in case of failure. ... View more

Thank you Chris for providing more information. It looks like it crashed in the code that writes Parquet files (HdfsParquetTableWriter::ColumnWriter<impala::StringValue>::ProcessValue). However, your query should not write any data: " SELECT a.topLevelField, b.priceFromNestedField FROM db.table a LEFT JOIN a.nestedField b"   I also noticed that the stack looks like it has been overwritten by something.   I don't recall any recent issues in that method and will have a look at the code to see if I can spot anything obvious.   In the meantime, can you double check that this query caused the crash and no other query was running?   Thanks, Lars ... View more

Let's see what the hs_err_pid file contains next.   Additionally, would you be willing to share the Minidump or a core dump with us in private? Please be aware that Minidumps contain process memory of each thread's stack, and core dumps contain all of the process's memory. Let me know if you'd like to do that and I'll share a private upload link with you.   Alternatively you can follow these instructions to resolve the minidump yourself and share the contained stack traces: https://cwiki.apache.org/confluence/display/IMPALA/Debugging+Impala+Minidumps ... View more

Hi,   Can you post the ends of the INFO and ERROR logs? Can you also post the content of the hs_err_pid<pid>.log file?   Thanks, Lars ... View more

For Python it makes a difference whether output gets printed to the terminal (which in this case likely supports unicode) or output is redirected to a file (which means it needs to be encoded in ASCII).   This post on StackOverflow seems to describe the issue well. I linked the post in the JIRA for future reference.   Cheers, Lars ... View more

Hi GeKas,   I'm not sure I understood your question. In general, writing to stdout should respect the local language settings of your shell:   $ echo $LANG en_US.UTF-8   Writing to a file however does not need to respect these, so it's behavior may be different. ... View more

This looks like IMPALA-2717 to me. The Jira has a patch attached to it, but no-one ever seems to have pushed a code review for this. Unfortunately there's no targeted release for this issue. Contributions are always welcome, let me know if you want to give it a shot.   Cheers, Lars ... View more

Can you share a query profile? That could give insights into where Impala is spending the time. ... View more

Hi hrishi1dypim,   Have you restarted all Impala roles including statestored and catalogd after the upgrade?   Cheers, Lars ... View more

Hi yehudaks,   How long does your processing step take? I.e., what do you mean by "takes a lot of time"?   Can you share a query profile?   Cheers, Lars ... View more

I just had a look, but I couldn't spot an obvious problem. The HDFS scanner fragments read around 15 MB/s, which seems reasonable to me, given how computationally intensive Parquet decoding is. There also doesn't seem to be any considerable skew. Each of your 5 nodes reads ~ 100GB of data in 134s, so the overall throughput is around 764 MB/s.   I suggest to have a look at the perf improvements around Parquet files in CDH 5.12 that I mentioned in an earlier reply. ... View more

The files shouldn't be too many. Impala processes files in parallel locally, too, so you should see a higher utilization on each node. Can you post a profile of one of the slow queries? ... View more

I'd try to reduce the file size to 256MB and make sure that the block size is at least that large, too. That way you should end up with 32GB * 4 = 128 files per partition. That should allow you to exploit parallelism across all your nodes. You can also try 512MB per file and see if that improves things, but I suspect it won't.   Btw, we're currently working on improving the ETL performance. You may want to look at the "SORT BY" clause that is included in Impala 2.9 and how it allows you to write data in a way that allows Impala to skip row groups much more effectively. You can find more information in the umbrella JIRA: https://issues.apache.org/jira/browse/IMPALA-2522 ... View more

Hi Shannon,   Impala does not split up Parquet files over several readers when reading them. Instead, only one daemon will be assigned for each file and will read the whole file. Therefore it is recommended to have only one block per file. Otherwise some of the blocks can be on remote nodes and remote reads will slow down your queries. See this page for more information: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_perf_cookbook.html   Cheers, Lars ... View more

@adi91 - How did you set --mem_limit? What value did you pass to it? What did http://hostname:25000/memz?detailed=true say after applying --mem_limit to the command line options? Did your value show up there? ... View more

After more investigation I found that this is already documented as a Known Issue in CM: Known Issues and Workarounds in Cloudera Manager 5   For Impala I opened IMPALA-5631 to explain the problem and possible solutions in the docs. ... View more

@mbigelow - Thank you for keeping the JIRA updated - I'm glad you found the solution through support. It looks like you are hitting a bug in CM and we are working on fixing it. I will reach out to our documentation team to point out this issue in the docs and the release notes of 5.11.1. I'm sorry for the troubles this has caused you. ... View more

Steps to generate SSL certificates are generally independent of Impala and well documented. One such place where you can find more information is here: https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs ... View more

Hi, You would do something like 'select * from A a left outer join B b...'. You can find more documentation on the topic here: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_joins.html Cheers, Lars ... View more

num_nodes=1 forces Impala to execute the query on a single node (machine), which will then only write a single parquet file per partition. ... View more

Hi imad87,   Your question looks related to Solr, so I think it may fit better into the "Search" community: http://community.cloudera.com/t5/Cloudera-Search-Apache-SolrCloud/bd-p/Search   Cheers, Lars ... View more

Hi Alon,   Have you tried the  inc_stats_size_limit_bytes command line flag as suggested by Tim? It is supported on CDH5.10.0. Here's the full help text from impalad:   -inc_stats_size_limit_bytes (Maximum size of incremental stats the catalog is allowed to serialize per table. This limit is set as a safety check, to prevent the JVM from hitting a maximum array limit of 1GB (or OOM) while building the thrift objects to send to impalads. By default, it's set to 200MB) type: int64 default: 209715200     This should allow you to increase the limit you are hitting.   Cheers, Lars ... View more

Thank you for catching this Tim!   The "SORTBY()" hint was added in IMPALA-4163, which was not included in Impala 2.8.0. It is currently being reworked into a SQL clause (IMPALA-4166), so I cannot make promises as to which release will contain this feature. My apologies for the confusion. I will make sure the documentation gets updated. ... View more

Hi imad87,   What is the purpose of specifying " ROW FORMAT DELIMITED" without a delimiter character?   On first glance it looks like your data file contains the substring "\N" (the \ character followed by the N character) to delimit lines, instead of the "\n" character (ASCII 0xA). Can you double check the file in a hex editor?   Cheers, Lars ... View more

Unfortunately I don't know how to speed up your particular query. You may want to have a look at Impala's query hints, especially the section about "Hints for join queries" on this page: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_hints.html   If that doesn't get you anywhere, maybe someone else here has an idea.   Cheers, Lars ... View more