Hi @Plop564   > If an error like that occurs, does the corresponding Impalad could crash ? If an Impalad crash owing to an UDF, restarting it will be enough to go back to good health ? Yes and yes.   >What about the Impalad isolation ? Again if a segmentation fault, memory leaks or race conditions occurs, does other cloudera services instances can be affected ? (HDFS, Hive, ...) It won't affect services outside of Impala - the crash is isolated to the Impala process   > Could you please quickly summarize the associated risks with buggy C++ UDF ? You already mentioned the possibilities of crashes, memory leaks and memory corruption. The other thing to keep in mind is that the UDF runs within the Impala process so essentially has the same permissions as the "impala" user. A malicious UDF could exploit this. This is why we recommend reviewing UDF code. It sounds like you're already following that best practice. ... View more

Yes, Close() will be called if the query fails.   Keep in mind that the same UDF can be in use from multiple threads at the same time, so any cleanup logic needs to be thread-safe and not clean up things that other threads might be using. ... View more

Hi Julien,   I wanted to clarify the question a bit to understand what you're trying to achieve. Impala really has two different concepts.   "Fragments" are a way of breaking down the query plan into units that can be executed in a distributed manner. You can see these in query plans with explain_level >= 2. They show up as sections of the plan with a heading like "F00: PLAN FRAGMENT". There are only two modes here. The default is to produce a distributed plan, which is broken up into fragments. The alternative, when the option num_nodes is set to 1, is to produce a single-node plan with only a single fragment.   The other concept is "fragment instances", which is the number of instances of each plan fragment that are run by the query. By default you generally get 0 or 1 fragments per impala daemon, depending on whether there is any data to scan, but we will do the scanning of data in a multi-threaded way. We have a new mode, under development, where you get multiple fragments per Impala daemon, controlled by the mt_dop query option. This only works for some queries, without inserts or joins and can sometimes consume a lot more resources. mt_dop can increase throughput of queries if the bottleneck is outside of the scan, e.g. in an aggregation.   ... View more

Hi Majuyell,   I think the problem is your use of unix_timestamp(), which returns a value in seconds precision.   You probably want to convert directly to timestamp. It looks like your timestamps are in a regular format so you can probably cast directly.   > select cast("2005-05-04 11:12:54.297" as timestamp); +----------------------------------------------+ | cast('2005-05-04 11:12:54.297' as timestamp) | +----------------------------------------------+ | 2005-05-04 11:12:54.297000000 | +----------------------------------------------+ Otherwise you can use to_timestamp() with a timestamp format string:   > select to_timestamp("2005-05-04 11:12:54.297", "yyyy-MM-dd HH:mm:ss.SSS"); +--------------------------------------------------------------------+ | to_timestamp('2005-05-04 11:12:54.297', 'yyyy-mm-dd hh:mm:ss.sss') | +--------------------------------------------------------------------+ | 2005-05-04 11:12:54.297000000 | +--------------------------------------------------------------------+ Then you can use from_timestamp to format it in your desired way:   > select from_timestamp(to_timestamp("2005-05-04 11:12:54.297", "yyyy-MM-dd HH:mm:ss.SSS"), "yyyy-MM-dd HH:mm:ss.SSS"); +---------------------------------------------------------------------------------------------------------------+ | from_timestamp(to_timestamp('2005-05-04 11:12:54.297', 'yyyy-mm-dd hh:mm:ss.sss'), 'yyyy-mm-dd hh:mm:ss.sss') | +---------------------------------------------------------------------------------------------------------------+ | 2005-05-04 11:12:54.297 | +---------------------------------------------------------------------------------------------------------------+   ... View more

"Exprs" tracks memory used by any SQL expressions evaluated in that node. E.g. if you have a regexp_replace() function operating on large strings, it will use more memory than a simple expression like "1 + 2".   The main use case is to help track down UDFs that uses a lot of memory, e.g. if they have a memory leak. ... View more

I just saw this in my email backlog. Yes this is the expected behaviour. Your UDF may be called again on the same thread or a different thread. After you call SetError() the query will fail but the error will take some time to propagate. ... View more