About COE

COE · ‎07-19-2022

For the workflows that were created on Hue, you can follow the next steps to export/import the workflows: >To export: Go to Documents> Select the workflow(s)> Select the button with three dots and select the Export option >To import: Go to Documents> Select the button with three dots and select the Import option.

COE · ‎06-26-2022

As upsert is not SQL-92 syntax, the Impala JDBC driver does not currently support it. However, we are currently testing a newer version of the JDBC driver and hope to support upsert in the near future. For the time being, please break these statements into a combination of insert and update statements.

COE · ‎06-23-2022

For a permanent fix, you need to install the python version 2.7.5.

COE · ‎06-22-2022

Yes, you need to switch to venv everytime or you need to reinstall python with the 2.7.5 version.

COE · ‎06-22-2022

You can try the below steps in edge node B virtualenv venv -p python2 cd venv source bin/activate (venv) impala-shell

COE · ‎06-22-2022

Hello Syed, Yes, the impala is compatible with Python version 2.7.5.

COE · ‎06-13-2022

Oozie Email Action supports attachments of HDFS files. Because standard output files are written as local files, you need to copy them to HDFS first, then specify the path in the <attachment> tag of the email action. <workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1"> ... <action name="[NODE-NAME]"> <email xmlns="uri:oozie:email-action:0.2"> <to>[COMMA-SEPARATED-TO-ADDRESSES]</to> <cc>[COMMA-SEPARATED-CC-ADDRESSES]</cc>  <subject>[SUBJECT]</subject> <body>[BODY]</body> <content_type>[CONTENT-TYPE]</content_type>  <attachment>[COMMA-SEPARATED-HDFS-FILE-PATHS]</attachment>  </email> <ok to="[NODE-NAME]"/> <error to="[NODE-NAME]"/> </action> ... </workflow-app>

COE · ‎06-09-2022

Hello Team, We have tested the java code internally and it worked fine for us. ClouderaJDBC version:- 2.6.27.1032 Java code:- ======== import java.sql.*; import java.math.*; public class test{ public static void main(String args[]){ try{ Class.forName("com.cloudera.impala.jdbc41.Driver"); Connection con=DriverManager.getConnection("jdbc:impala://<<hostname>> :21050;UseNativeQuery=1"); String sql = "upsert into user_info(id, name, address, email, insert_time) values (?,?,?,?,?)"; PreparedStatement statement = con.prepareStatement(sql); statement.setInt(1, 102); statement.setString(2, "Peter"); statement.setString(3, "New York"); statement.setString(4, "John@xyz.com"); statement.setTimestamp(5, java.sql.Timestamp.valueOf(java.time.LocalDateTime.now())); statement.addBatch(); statement.executeBatch(); statement.close(); con.close(); } catch(Exception e){ System.out.println(e); } } } Please let us know if it helps.

COE · ‎06-09-2022

Please elaborate a little more on the issue:- Also please share the steps you are performing and share the table DDL for the same.

COE · ‎11-25-2021

From where the query is submitted[Jdbc/ODBC, Impala-shell, Hue]. If it is from JDBC/ODBC then the query is generated to figure out column names. You can disable the feature that generates it with the PreparedMetadataLimitZero flag - see [page 90] https://www.cloudera.com/documentation/other/connectors/impala-jdbc/latest/Cloudera-JDBC-Driver-for-...

COE · ‎10-14-2021

Hello, Please try the below command and let us know:- $ sqoop import --connect jdbc:mysql://localhost/employees --username hive --password hive --table departments --hcatalog-database default --hcatalog-table my_table_orc --create-hcatalog-table --hcatalog-storage-stanza --hcatalog-storage-stanza "stored as orc tblproperties (\"transactional\"=\"false\")" If it doesn’t work then the workaround is the two step process. 1. Create the ORC table in hive with the keyword external and set transactional to false 2. Then use the sqoop command to load the data into the orc table.

COE · ‎10-14-2021

Hello, Please try to run the sqoop command as below and let us know how it goes:- $ sqoop import --connect jdbc:mysql://localhost/employees --username hive --password hive --table departments --hcatalog-database default --hcatalog-table my_table_orc --create-hcatalog-table --hcatalog-storage-stanza "stored as orcfile"

COE · ‎10-14-2021

Hello, The partition clause in the drop partition expect CONSTANT VALUE on the right-hand side and the functions inside the drop partition clause are not supported The correct syntax would be:- ALTER TABLE audit_logs DROP PARTITION (evt_date<‘some constant value ' )

COE · ‎10-13-2021

Hello, Please try with the below connection URL and let us know how it goes jdbc:impala://nightly57-3.gce.cloudera.com:21050/default;AuthMech=1;SSL=1;KrbRealm=GCE.CLOUDERA.COM;KrbHostFQDN=nightly57-4.gce.cloudera.com;KrbServiceName=impala;SSLTrustStore=/etc/cdep-ssl-conf/CA_STANDARD/truststore.jks

COE · ‎10-12-2021

Hello, There is no such way to kill the query in one go. You need to either do it by one of the following methods:- You can kill the query from the Impala Daemon web UI of the Impala Daemon coordinating the query. or you can try killing from the browser. https://<query_coordinator_server_name>:25000/close_session?session_id=<session_id>

COE · ‎10-11-2021

Hello, You can kill the query from the Impala Daemon web UI of the Impala Daemon coordinating the query. or you can try killing from the browser. https://<query_coordinator_server_name>:25000/close_session?session_id=<session_id> Please let us know if it helps

COE · ‎10-04-2021

Hello, Are you trying to connect impala from spark via JDBC? if yes, we don't support this feature yet. please refer to the below document. https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_621_unsupported_features.html#spark

COE · ‎10-04-2021

Hello, Please use the either of the methods and let us know if it helps:- Step 1:- Pass the --hs2-url correctly sqoop import --connect <<Connection string>> --username <<username>> --password <<password>> --table “ta”ble_name --target-dir “<<“directory path>> --delete-target-dir --hive-import --hs2-url "jdbc:hive2://hs2host:10000/default;principal=hive/hs2host@DOMAIN.COM;ssl=true;sslTrustStore=/etc/cdep-ssl-conf/CA_STANDARD/truststore.jks;trustStorePassword=password" --hs2-user username --hs2-keytab "/path/to/sqooptestkeytab" If you are launching the sqoop job from CLI, you don’t need to pass the hs2-user and hs2 key tab. You can remove the parameters and run the sqoop command after doing the kinit from CLI or you can pass in the sqoop command itself like above. Step 2:- If you don’t want to add the hs2_url Add the following 2 properties in the sqoop configuration and restart sqoop 1. Sqoop Client Advanced Configuration Snippet (Safety Valve) for sqoop-conf/sqoop-site.xml : Name: sqoop.beeline.env.preserve Value: HADOOP_CLIENT_OPTS 2. Sqoop Client Advanced Configuration Snippet (Safety Valve) for sqoop-conf/sqoop-env.sh: export HADOOP_CLIENT_OPTS="-Djline.terminal=jline.UnsupportedTerminal"

COE · ‎10-04-2021

Hello, Please find the sample impala oozie action and check if it helps. 1 Create a file impala_invalidate.sh export PYTHON_EGG_CACHE=/tmp/impala_eggs impalacmd="impala-shell -k -s impala --quiet --impalad=host-10-17-100-110.coe.cloudera.com:21000" ${impalacmd} --query="show databases;" > /tmp/impala_databases 2 Create the workflow for the shell action <workflow-app name="Impala_redirect" xmlns="uri workflow:0.5"> <start to="shell-11a3"/> <kill name="Kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <action name="shell-11a3"> <shell xmlns="uri shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <exec>impala_invalidate.sh</exec> <file>/user/systest/lib/impala_invalidate.sh#impala_invalidate.sh</file> <file>/user/systest/lib/impala.keytab#impala.keytab</file> <capture-output/> </shell> <ok to="End"/> <error to="Kill"/> </action> <end name="End"/> </workflow-app>

COE · ‎09-23-2021

Hello Can you try giving space between -- & import as it is a sub tool name. sqoop job --create job1 -- import --connect jdbc:oracle:thin:xxxxx --username xxxxx --password xxxxxxxx --as-parquetfile --target-dir /user/cdh/testgb/raw/adminunits --fields-terminated-by ',' --lines-terminated-by '\n' --table develop.adminunits --m 1 --hive-import --hive-database metdev_testgb --hive-table ADMINUNITS

COE · ‎09-22-2021

1. For total memory configured you can check (impala daemon memory * a total number of demons ) , these values should be displayed on top of Impala admission control as well that this much if memory is allocated to the impala. 2. You can check other memory metrics from the cluster utilization report, please note that how much memory is consumed per pool feature is not currently captured in impala metrics. a) Max Allocated Peak Allocation Time – The time when Impala reserved the maximum amount of memory for queries. Click the drop-down list next to the date and time and select View Impala Queries Running at the Time to see details about the queries. Max Allocated – The maximum memory that was reserved by Impala for executing queries. If the percentage is high, consider increasing the number of hosts in the cluster. Utilized at the Time – The amount of memory used by Impala for running queries at the time when maximum memory was reserved. Click View Time Series Chart to view a chart of peak memory allocations. Histogram of Allocated Memory at Peak Allocation Time – Distribution of memory reserved per Impala daemon for executing queries at the time Impala reserved the maximum memory. If some Impala daemons have reserved memory close to the configured limit, consider adding more physical memory to the hosts. b) Max Utilized Peak Usage Time – The time when Impala used the maximum amount of memory for queries. Click the drop-down list next to the date and time and select View Impala Queries Running at the Time to see details about the queries. Max Utilized – The maximum memory that was used by Impala for executing queries. If the percentage is high, consider increasing the number of hosts in the cluster. Reserved at the Time – The amount of memory reserved by Impala at the time when it was using the maximum memory for executing queries. Click View Time Series Chart to view a chart of peak memory utilization. Histogram of Utilized Memory at Peak Usage Time – Distribution of memory used per Impala daemon for executing queries at the time Impala used the maximum memory. If some Impala daemons are using memory close to the configured limit, consider adding more physical memory to the hosts. [1] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cluster_util_custom.html#concept_jp4_4bh_hx [2] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_metrics_impala_daemon.html [3] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_metrics_impala_daemon_resource_pool.html

Online	Offline
Last Visited	‎02-01-2023 02:52 AM

Date Registered	‎07-07-2020 10:27 AM
Last Visited	‎02-01-2023 02:52 AM
Total Messages Posted	62
Total Kudos Received	4

Re: If i have to send a mail via oozie containing ...

Re: impala jdbc doesn't work for preparestatement ...

Re: Impala Queries

Re: Find the % of the used memory in impala

Re: Accessing Oozie Workflows/Schedules Created by...

Re: impala jdbc insert chinese string have a probl...

Re: Impala shell error

Re: Impala shell error

Re: Impala shell error

Re: Impala shell error

Re: If i have to send a mail via oozie containing ...

Re: impala jdbc doesn't work for preparestatement ...

Re: impala jdbc doesn't work for preparestatement ...

Re: Impala query with limit 0

Re: Sqoop from Postgre to Hive and convert to ORC ...

Re: Sqoop from Postgre to Hive and convert to ORC ...

Re: Hive drop partition is not working

Re: Unable to make a jdbc connection with impala

Re: Impala Queries

Re: Impala Queries

Re: some impala built-in function can't work in sp...

Re: How do you pass on Hiveserver2 URL in Sqoop co...

Re: Impala-shell in oozie doesn't work

Re: cannot create sqoop job in terminal

Re: Find the % of the used memory in impala