Member since
09-19-2020
45
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
360 | 07-13-2021 12:09 AM |
05-18-2022
01:11 AM
Hello Team, could you please advise how update with table alias works? update cbs_cubes.TB_JDV_CBS_NEW set a.SUB_SERVICE_CODE_V=b.SUB_SERVICE_CODE_V from cbs_cubes.update1_sub_service_code b where a.SUB_SERVICE_CODE_V=b.SUB_SERVICE_CODE_V [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: 'cbs_cubes.TB_JDV_CBS_NEW' is not a valid table alias or reference. ), Query: update cbs_cubes.TB_JDV_CBS_NEW set a.SUB_SERVICE_CODE_V=b.SUB_SERVICE_CODE_V from cbs_cubes.update1_sub_service_code b where a.SUB_SERVICE_CODE_V=b.SUB_SERVICE_CODE_V. Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
04-25-2022
12:52 AM
Hello Team, can you please advise why connection to Kafka fails? Is the issue with keytab? Keytab is working fine on Linux machine. sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="D:\services.kerberos.keytab" principal="services/rb-hadoop-06.mtg.local@INNOV.LOCAL"; org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540) at org.apache.kafka.clients.admin.Admin.create(Admin.java:134) at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$2(PluginsAwareKafkaAdmin.scala:17) at io.conduktor.plugins.PluginsResources$$anon$1.executeWithPlugins(PluginsResources.scala:74) at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$1(PluginsAwareKafkaAdmin.scala:17) at scala.util.Try$.apply(Try.scala:210) at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.create(PluginsAwareKafkaAdmin.scala:17) at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin.create(PluginsAwareKafkaAdmin.scala) at io.conduktor.kafka.KafkaClientMaker.makeAdminClient(KafkaClientMaker.kt:103) at io.conduktor.kafka.KafkaClientMaker.checkConnectivity(KafkaClientMaker.kt:79) at io.conduktor.views.clusterconfiguration.KafkaCluster$kafkaClusterConfigTab$1$2$1$11$1$2$1$1.invokeSuspend(KafkaCluster.kt:334) at io.conduktor.views.clusterconfiguration.KafkaCluster$kafkaClusterConfigTab$1$2$1$11$1$2$1$1.invoke(KafkaCluster.kt) at io.conduktor.views.clusterconfiguration.KafkaCluster$kafkaClusterConfigTab$1$2$1$11$1$2$1$1.invoke(KafkaCluster.kt) at io.conduktor.JavaFxExtensionsKt$launchWithProgressCancelable$job$1.invokeSuspend(JavaFxExtensions.kt:353) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: null (68) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:513) ... 19 more Caused by: javax.security.auth.login.LoginException: null (68) at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source) at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source) at java.base/javax.security.auth.login.LoginContext.invoke(Unknown Source) at java.base/javax.security.auth.login.LoginContext$4.run(Unknown Source) at java.base/javax.security.auth.login.LoginContext$4.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at java.base/javax.security.auth.login.LoginContext.invokePriv(Unknown Source) at java.base/javax.security.auth.login.LoginContext.login(Unknown Source) at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60) at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103) at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62) at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170) ... 23 more Caused by: KrbException: null (68) at java.security.jgss/sun.security.krb5.KrbAsRep.<init>(Unknown Source) at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.send(Unknown Source) at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(Unknown Source) ... 36 more Caused by: KrbException: Identifier doesn't match expected value (906) at java.security.jgss/sun.security.krb5.internal.KDCRep.init(Unknown Source) at java.security.jgss/sun.security.krb5.internal.ASRep.init(Unknown Source) at java.security.jgss/sun.security.krb5.internal.ASRep.<init>(Unknown Source) ... 39 more
... View more
Labels:
- Labels:
-
Apache Kafka
03-28-2022
08:16 AM
Dear Team, kindly advise on why connection from our CDC tools to Kudu is not working? What does KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) mean? at java.lang.Thread.run(Thread.java:750) com.webaction.common.exc.ConnectionException: Couldn't establish connection with Kudu ( RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051 ) at com.striim.proc.Connection.KuduWriterConnection.connect(KuduWriterConnection.java:83) at com.striim.proc.Connection.KuduWriterConnection.<init>(KuduWriterConnection.java:44) at com.striim.proc.Connection.KuduWriterConnection.getConnection(KuduWriterConnection.java:100) at com.striim.proc.KuduWriter.initializeKuduClient(KuduWriter.java:224) at com.striim.proc.KuduWriter.initWriter(KuduWriter.java:132) at com.webaction.utils.writers.common.RetriableBaseDataStoreWriter.init(RetriableBaseDataStoreWriter.java:78) at com.webaction.runtime.components.Target.start(Target.java:377) at com.webaction.runtime.components.Flow.start(Flow.java:431) at com.webaction.runtime.components.Flow.start(Flow.java:375) at com.webaction.runtime.components.Flow.start(Flow.java:335) at com.webaction.runtime.components.Flow$2.run(Flow.java:1612) at java.lang.Thread.run(Thread.java:750) com.webaction.common.exc.ConnectionException: Couldn't establish connection with Kudu ( RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051 ) at com.striim.proc.Connection.KuduWriterConnection.connect(KuduWriterConnection.java:83) at com.striim.proc.Connection.KuduWriterConnection.<init>(KuduWriterConnection.java:44) at com.striim.proc.Connection.KuduWriterConnection.getConnection(KuduWriterConnection.java:100) at com.striim.proc.KuduWriter.initializeKuduClient(KuduWriter.java:224) at com.striim.proc.KuduWriter.initWriter(KuduWriter.java:132) at com.webaction.utils.writers.common.RetriableBaseDataStoreWriter.init(RetriableBaseDataStoreWriter.java:78) at com.webaction.runtime.components.Target.start(Target.java:377) at com.webaction.runtime.components.Flow.start(Flow.java:431) at com.webaction.runtime.components.Flow.start(Flow.java:375) at com.webaction.runtime.components.Flow.start(Flow.java:335) at com.webaction.runtime.components.Flow$2.run(Flow.java:1612) at java.lang.Thread.run(Thread.java:750) com.striim.proc.exception.KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) at com.striim.proc.CheckPoint.CheckpointTableImpl.checkForCheckpointTable(CheckpointTableImpl.java:133) at com.striim.proc.CheckPoint.CheckpointTableImpl.<init>(CheckpointTableImpl.java:44) at com.striim.proc.KuduWriter.initWriter(KuduWriter.java:152) at com.webaction.utils.writers.common.RetriableBaseDataStoreWriter.init(RetriableBaseDataStoreWriter.java:78) at com.webaction.runtime.components.Target.start(Target.java:377) at com.webaction.runtime.components.Flow.start(Flow.java:431) at com.webaction.runtime.components.Flow.start(Flow.java:375) at com.webaction.runtime.components.Flow.start(Flow.java:335) at com.webaction.runtime.components.Flow$2.run(Flow.java:1612) at java.lang.Thread.run(Thread.java:750) Command RESUME failed so application CDC.CDC_CBS_CUST_CBS_NP is put in HALTED state com.striim.proc.exception.KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) at com.striim.proc.CheckPoint.CheckpointTableImpl.checkForCheckpointTable(CheckpointTableImpl.java:133) at com.striim.proc.CheckPoint.CheckpointTableImpl.<init>(CheckpointTableImpl.java:44) at com.striim.proc.KuduWriter.initWriter(KuduWriter.java:152) at com.webaction.utils.writers.common.RetriableBaseDataStoreWriter.init(RetriableBaseDataStoreWriter.java:78) at com.webaction.runtime.components.Target.start(Target.java:377) at com.webaction.runtime.components.Flow.start(Flow.java:431) at com.webaction.runtime.components.Flow.start(Flow.java:375) at com.webaction.runtime.components.Flow.start(Flow.java:335) at com.webaction.runtime.components.Flow$2.run(Flow.java:1612) at java.lang.Thread.run(Thread.java:750) com.striim.proc.exception.KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) at com.striim.proc.CheckPoint.CheckpointTableImpl.checkForCheckpointTable(CheckpointTableImpl.java:133) at com.striim.proc.CheckPoint.CheckpointTableImpl.<init>(CheckpointTableImpl.java:44) at com.striim.proc.KuduWriter.initWriter(KuduWriter.java:152) at com.webaction.utils.writers.common.RetriableBaseDataStoreWriter.init(RetriableBaseDataStoreWriter.java:78) at com.webaction.runtime.components.Target.start(Target.java:377) at com.webaction.runtime.components.Flow.start(Flow.java:431) at com.webaction.runtime.components.Flow.start(Flow.java:375) at com.webaction.runtime.components.Flow.start(Flow.java:335) at com.webaction.runtime.components.Flow$2.run(Flow.java:1612) Thanks, Roshan
... View more
Labels:
10-20-2021
07:21 AM
Hello Team, we are having some performance issue when sending data to Kafka using JSON as format. Performs is very slow. Performance is faster when using DSV parser. How can we fine tune JSON formatter? Regards, Roshan
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Zookeeper
09-14-2021
03:38 AM
Dear Team, can you please advise why varchar is not acceptable when replicating from Oracle to Kudu though I can create table with varchar on Kudu? Error from our CDC tool. Error while writing batch on table impala::cbs.cb_ad_x Reason account_type_v isn't [Type: string], it's varchar. Cause: account_type_v isn't [Type: string], it's varchar CREATE TABLE cbs.cb_address_xm ( addressid varchar(65535) , account_link_code_n BIGINT , account_type_v varchar(65535) , address_type_n INT , account_type varchar(65535) , address_format varchar(65535) , address_type INT , building varchar(65535) , city varchar(65535) , city_desc varchar(65535) , country varchar(65535) , country_desc varchar(65535) , district varchar(65535) , district_desc varchar(65535) , floor varchar(65535) , landmark varchar(65535) , postal_code varchar(65535) , po_code varchar(65535) , street varchar(65535) , street_desc varchar(65535) , sub_locality_code varchar(65535) , sub_locality_desc varchar(65535) , op_insert_date TIMESTAMP , op_update_date TIMESTAMP , state_desc varchar(65535) , PRIMARY KEY (addressid) ) PARTITION BY HASH (addressid) PARTITIONS 16 STORED AS KUDU ; Regards, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
08-12-2021
10:45 PM
Hello Team, I have the following file to load on Kudu. 1829;BN=0;UNIT=VOLUME_ALL;IN=0;TC=0;TCC=0;CT=;FU=1000001;CU=54274;FB=61701;FL=ugw9811_3828500385_360_27153 0=5126742111750858;U=23059268534;SI=6;SG=1;SR=7;SN=BROWSING;SC=BROWSING;BS=60342256;BR=2581143;TU=2021-04-27 14:02:47;TF=2021-04-27 00:00:00;TA=2021-04-27 14:02:47;TB=2021-04-27 00:00:00;TE=2021-04-27 14:02:47;TS=1619517767;D=16292;R=151;E=0;UDR_cu=0;UDR_fb=BROWSING;DCM=0;UP=Prepaid;ST=BROWSING;MSISDN=23059268534;APN=orange;SGSN=196.192.13.113;GGSN=196.192.13.113;IMSI=617010014925066;BU1=23292;BN=0;UNIT=VOLUME_ALL;IN=0;TC=0;TCC=62923399;CT=;FU=1000000;CU=3586;FB=61701;FL=ugw9811_3828490275_312_8799 0=5126752111750858;U=23059268534;SI=6;SG=1;SR=7;SN=BROWSING;SC=BROWSING;BS=0;BR=0;TU=2021-04-27 14:02:47;TF=2021-04-27 00:00:00;TA=2021-04-27 14:02:47;TB=2021-04-27 00:00:00;TE=2021-04-27 14:02:47;TS=1619517767;D=16292;R=151;E=0;UDR_cu=0;UDR_fb=BROWSING;DCM=0;UP=Prepaid;ST=BROWSING;MSISDN=23059268534;APN=orange;SGSN=196.192.13.113;GGSN=196.192.13.113;IMSI=617010014925066;BU1=21829;BN=0;UNIT=VOLUME_ALL;IN=0;TC=0;TCC=0;CT=;FU=1000001;CU=3586;FB=61701;FL=ugw9811_3828490275_312_8799 How can I proceed using Spark SQL? Table structure on Kudu: CREATE EXTERNAL TABLE cdr.mobile_datadbs ( id BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, msisdn STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, serviceid INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, servicegroup INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, servicerev INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, servicename STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, serviceclass STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, downlink INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, uplink INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, storedtime TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, firstaccesstime TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, lastaccesstime TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, servicebegintime TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, sessionendtime TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, cdrcreatedtime BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, duration BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, hitsperreq BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, errors1 BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, udrcu INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, udrfb STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, status1 STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, userprofile STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, servicetype STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, subsmsisdn STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, apn STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, sgsnaddress STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, ggsnaddress STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, imsi STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, bonusunit STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, bn INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, unit STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, instatus INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, totalcost BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, totalcharge BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, terminationcause BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, firstrequestedurl STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, cellidinfo STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, idpname STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, failureslist STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, PRIMARY KEY (id) ) PARTITION BY HASH (id) PARTITIONS 16 STORED AS KUDU TBLPROPERTIES ('external.table.purge'='TRUE', 'kudu.master_addresses'='rb-hadoop-03.mtg.local,rb-hadoop-04.mtg.local,rb-hadoop-05.mtg.local') Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
-
Apache Spark
08-12-2021
09:48 PM
Hi, yes I would like to re-use your Oracle functions and use Zeppelin as your notebook. we are using impala as SQL engine. How do I translate that SQL function to Oracle dialect? Regards, Roshan
... View more
08-10-2021
05:16 AM
Hello Team, can you please advise why Kudu returns 0 rows? count of base table is same. Oracle - returns 21000 rows: select BankOrgProfile.* from AR.HZ_ORGANIZATION_PROFILES BankOrgProfile where SYSDATE between TRUNC(BankOrgProfile.effective_start_date) and NVL(TRUNC(BankOrgProfile.effective_end_date), SYSDATE+1) ; Kudu (Impala) - returns 0 rows: select BankOrgProfile.* from oracle_financial.HZ_ORGANIZATION_PROFILES BankOrgProfile where current_date() between cast(BankOrgProfile.effective_end_date as date) and nvl(cast(BankOrgProfile.effective_end_date as date), adddate(current_date(),1)) ; Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
08-07-2021
08:44 AM
1 Kudo
Dear Team, is there a way I can convert Oracle functions to Zeplin (Kudu) equivalent? Below query calls function "apps.xxmt_getvendor_emailadd (ai.vendor_site_id)" SELECT sysdate, apps.xxmtg_get_org_name (aip.org_id) operating_unit, ai.creation_date inv_creation_date, ai.invoice_date, ai.invoice_received_date Inv_RecvDte, aba.batch_name, ai.invoice_num, ai.description invoice_desc, ai.invoice_amount, aps.payment_priority, aps.due_date, ac.currency_code, ac.creation_date pmt_creation_date, apps.X_MTG_get_user_name(ac.created_by) created_by, ac.status_lookup_code, ac.check_date document_date, ac.check_number, aip.amount pmt_doc_amount, pd.payment_document_name, ac.bank_account_name, ac.VENDOR_NAME Supplier_Name, pov.segment1 SupplierNum, b.bank_name Supplier_Bank, to_char(b.bank_number) Supplier_Banknum, br.bank_branch_name Supplier_BankBranch, iby_ac.bank_account_num Supplier_BankAccount, apps.xxmt_getvendor_emailadd (ai.vendor_site_id) supplier_email,trunc(ac.CREATION_DATE) Payment_Creation_Date, ... Oracle function <apps.xxmt_getvendor_emailadd> content: create or replace Function xxmt_getvendor_emailadd(P_Vendor_Site_Id In Number) Return Varchar2 Is LvEmailaddress Varchar2(240):=NULL; Begin Select email_address Into LvEmailaddress From ap_supplier_sites_all Where vendor_site_id=P_Vendor_Site_Id; Return LvEmailaddress; Exception When Others Then LvEmailaddress:=''; Return LvEmailaddress; End xxmt_getvendor_emailadd; Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
-
Apache Zeppelin
07-13-2021
12:09 AM
On Oracle below works select concat(nvl(cast(glb.budget_version_id as varchar2(10)),0), cast(glc.segment1 as varchar2(10))) as balid from gl.gl_balances glb, gl.gl_code_combinations glc, gl.gl_ledgers glg ; Impala
... View more
07-13-2021
12:00 AM
Hello Team, can you please advise how to replace nulls by a default value? select concat(cast(glb.budget_version_id as string)," ", cast(glc.segment1 as string)) as balid from oracle_financial.gl_balances glb, oracle_financial.gl_code_combinations glc, oracle_financial.gl_ledgers glg ; If I add a null to a value, null will be returned. I tried NVL it is not working. Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
-
Apache Zeppelin
07-02-2021
04:16 AM
Dear Team, how can I mask last 5 digits in field below in Kudu? on Ranger? Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
-
Apache Ranger
07-01-2021
04:30 AM
Hello Team, how can we extract XML values from the column below (type string) from Impala (Kudu) Regards, Roshan
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
06-28-2021
10:41 PM
Hi, is there a way we can change the datatype from int to double for a column on Impala? Table size is around 3 billion records. I do not plan to drop and recreate it. ALTER TABLE cbs.gprs_home_cdrs CHANGE percentage_val_n percentage_val_n double; AnalysisException: Cannot change the type of a Kudu column using an ALTER TABLE CHANGE COLUMN statement: (INT vs DOUBLE) impalad version 3.4.0-SNAPSHOT RELEASE (build 134517e42b7b6085e758195465f956f431e0e575) Built on Sat Dec 12 11:15:02 UTC 2020 Version: Cloudera Enterprise 7.1.3 (#4999720 built by jenkins on 20200805-1701 git: fa596184790377f07ba80e9cd4da8b875237939c) Java VM Name: OpenJDK 64-Bit Server VM Java Version: 11.0.10 Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
-
Apache Kudu
06-27-2021
03:51 AM
What does the code below do? val conf = new SparkConf().setMaster("local").setAppName("testApp") val sc= SparkContext.getOrCreate(conf) Reference: https://www.educba.com/spark-rdd-operations/
... View more
06-27-2021
12:41 AM
Hi, I run the following command and I am getting error below: sqoop import --connect "jdbc:oracle:thin:@10.215.227.*:1521:cxx_stby" --username cbs_view --password ccccc --query "select TMP_ACCOUNT_CODE_N,decode(EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'),'.',null,EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'))||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@SUB_LOCALITY_DESC') ||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@CITY_DESC') "ADDRESS" from tmp_address_xml@cbsstandby where $CONDITIONS" -m 4 --split-by object_type --hive-import --target-dir '/devsh_loudacre' --hive-table test_oracle.address_tab --verbose SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/06/27 07:39:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.0.1.0-187 21/06/27 07:39:14 DEBUG tool.BaseSqoopTool: Enabled debug logging. 21/06/27 07:39:14 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/06/27 07:39:14 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 21/06/27 07:39:14 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 21/06/27 07:39:14 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory 21/06/27 07:39:14 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.DefaultManagerFactory 21/06/27 07:39:14 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory 21/06/27 07:39:14 DEBUG oracle.OraOopUtilities: Enabled OraOop debug logging. 21/06/27 07:39:14 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop! 21/06/27 07:39:14 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled. 21/06/27 07:39:14 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.DefaultManagerFactory 21/06/27 07:39:14 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:thin:@10.215.227.22:1521 21/06/27 07:39:14 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache. 21/06/27 07:39:14 INFO manager.SqlManager: Using default fetchSize of 1000 21/06/27 07:39:14 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@3bf7ca37 21/06/27 07:39:14 INFO tool.CodeGenTool: Beginning code generation 21/06/27 07:39:14 ERROR tool.ImportTool: Import failed: java.io.IOException: Query [select TMP_ACCOUNT_CODE_N,decode(EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'),'.',null,EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'))||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@SUB_LOCALITY_DESC') ||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@CITY_DESC') ADDRESS from tmp_address_xml@cbsstandby where ] must contain '$CONDITIONS' in WHERE clause. at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:333) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1879) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1672) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:516) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:656) at org.apache.sqoop.Sqoop.run(Sqoop.java:150) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:186) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:240) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:249) at org.apache.sqoop.Sqoop.main(Sqoop.java:258) Kindly advise Thanks, Roshan
... View more
06-26-2021
07:18 AM
Hi @aakulov thanks for the update. Can you please advise how can I schedule this scoop job so that it will update the hive table with incremental changes(CDC)? for example, suppose the XML fields are updated on Oracle, how can I schedule scoop job to replicate the incremental changes on Hive and Kudu? Regards, Roshan
... View more
06-26-2021
04:05 AM
Thanks for the update. scala> val myRDD=spark.read.textFile("/devsh_loudacre/frostroad.txt") myRDD: org.apache.spark.sql.Dataset[String] = [value: string] why does myRDD.parallelize not working for above? scala> val myRDD1=sc.parallelize(myRDD) <console>:26: error: type mismatch; found : org.apache.spark.sql.Dataset[String] required: Seq[?] Error occurred in an application involving default arguments. val myRDD1=sc.parallelize(myRDD) Does the above mean a dataset has been created? what is the difference between the above and below? val myRDD2=sc.textFile("/devsh_loudacre/frostroad.txt") can I add the .parallelize function with the above command? Thanks, Roshan
... View more
06-25-2021
08:41 AM
I managed to fix it but getting error below because of XML types [root@sandbox-hdp lib]# sqoop job --exec myjob7 Warning: /usr/hdp/3.0.1.0-187/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/06/25 15:32:55 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.0.1.0-187 21/06/25 15:32:55 INFO manager.SqlManager: Using default fetchSize of 1000 Enter password: 21/06/25 15:33:09 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled. 21/06/25 15:33:09 INFO manager.SqlManager: Using default fetchSize of 1000 21/06/25 15:33:09 INFO tool.CodeGenTool: Beginning code generation 21/06/25 15:33:10 INFO manager.OracleManager: Time zone has been set to GMT 21/06/25 15:33:10 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM cb_account_master t WHERE 1=0 Exception in thread "main" java.lang.NoClassDefFoundError: oracle/xdb/XMLType at oracle.jdbc.oracore.OracleTypeADT.applyTDSpatches(OracleTypeADT.java:1081) at oracle.jdbc.oracore.OracleTypeADT.parseTDSrec(OracleTypeADT.java:1002) at oracle.jdbc.oracore.OracleTypeADT.parseTDS(OracleTypeADT.java:936) at oracle.jdbc.oracore.OracleTypeADT.init(OracleTypeADT.java:489) at oracle.jdbc.oracore.OracleTypeADT.init(OracleTypeADT.java:470) at oracle.sql.TypeDescriptor.getTypeDescriptor(TypeDescriptor.java:981) at oracle.jdbc.driver.NamedTypeAccessor.otypeFromName(NamedTypeAccessor.java:78) at oracle.jdbc.driver.TypeAccessor.initMetadata(TypeAccessor.java:71) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:833) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:897) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1034) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3867) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1502) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:777) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:328) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1879) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1672) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:516) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:656) at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:248) at org.apache.sqoop.tool.JobTool.run(JobTool.java:303) at org.apache.sqoop.Sqoop.run(Sqoop.java:150) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:186) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:240) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:249) at org.apache.sqoop.Sqoop.main(Sqoop.java:258) Caused by: java.lang.ClassNotFoundException: oracle.xdb.XMLType at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 33 more [root@sandbox-hdp lib]# Kindly advise. Thanks, Roshan
... View more
06-25-2021
07:47 AM
Hi,
I am using Cloudera Sandbox for Hortonwork. Can anyone help me with Sqoop? I am trying to do an Oracle JDBC connection?
[root@sandbox-hdp lib]# sqoop list-databases --connect jdbc:oracle:thin:@10.124.0.70:1523/BI
Warning: /usr/hdp/3.0.1.0-187/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/06/25 14:32:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.0.1.0-187
21/06/25 14:32:10 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
21/06/25 14:32:10 INFO manager.SqlManager: Using default fetchSize of 1000
21/06/25 14:32:10 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:287)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.OracleManager.listDatabases(OracleManager.java:702)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:150)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:186)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:240)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:249)
at org.apache.sqoop.Sqoop.main(Sqoop.java:258)
Thanks,
Roshan
... View more
Labels:
06-25-2021
06:53 AM
Which methods do you think would be most appropriate to use? I was thinking of using beehive to read the table/run query in real time and save it in parquet. The load from parqet to Kudu? or using Sqoop to read Oracle tables into HDFS. Then from HDFS to Kudu.
... View more
06-25-2021
04:35 AM
Hello Team, can you please advise if there is an equivalent of the function on Kudu below to extract address and location from a table with XML data type? select TMP_ACCOUNT_CODE_N,decode(EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'),'.',null,EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@STREET_DESC'))||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@SUB_LOCALITY_DESC') ||' '||EXTRACTVALUE (address_x, '//ADDRESS_DTLS/@CITY_DESC') New_installation_address from tmp_address_xml@cbsstandby where address_type_n = 4 Regards, Roshan
... View more
- Tags:
- kudu
Labels:
- Labels:
-
Apache Kudu
06-24-2021
07:05 AM
Hello Team, I am working the tutorial on RDD. I am having some difficulties understanding some commands. Can you please advise what steps 3-8 do? . Encode the Schema in a string val schemaString = "name age" 4. Generate the schema based on the string of schema val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, nullable = true))
val schema = StructType(fields) 5. Convert records of the RDD (people) to Rows val rowRDD = peopleRDD.map(_.split(",")).map(attributes => Row(attributes(0), attributes(1).trim)) 6. Apply the schema to the RDD val peopleDF = spark.createDataFrame(rowRDD, schema) 6. Creates a temporary view using the DataFrame peopleDF.createOrReplaceTempView("people") 7. SQL can be run over a temporary view created using DataFrames val results = spark.sql("SELECT name FROM people") 8.The results of SQL queries are DataFrames and support all the normal RDD operations. The columns of a row in the result can be accessed by field index or by field name results.map(attributes => "Name: " + attributes(0)).show() https://www.cloudera.com/tutorials/dataframe-and-dataset-examples-in-spark-repl.html Programmatically Specifying Schema What does the code below do? val ds = Seq(1, 2, 3).toDS() val ds = Seq(Person("Andy", 32)).toDS() Section Section DataSet API is clear. If we need to map the JSON file to a class we use the as(class name). So to map a file to a class we use the ".as[Classname]"? what does this command do? val ds = Seq(1, 2, 3).toDS() Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Spark
06-24-2021
06:43 AM
Hi @RangaReddy thanks a lot for sharing the link. It will help me a lot. Can you please advise why we have to include df (data frame name) before each column? df.select(df("name"), df("age") + 1).show() I noticed in groupBy() there is no df. Grateful if you can clarify this. Thanks, Roshan
... View more
06-24-2021
03:08 AM
Hi,
can you please advise why source path is not recognized on Windows?
SyntaxError: invalid syntax >>> spark-submit /D:/Spark/devsh/exercises/yarn/wordcount.py /devsh_loudacre/kb File "<stdin>", line 1 spark-submit /D:/Spark/devsh/exercises/yarn/wordcount.py /devsh_loudacre/kb
Thanks,
Roshan
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
06-23-2021
08:04 AM
I have been working with Oracle databases, in what way is DataFrames and DataSets similar to Oracle? Are they similar to views?
... View more
06-23-2021
07:56 AM
Hello Everyone, can you please tell me the difference between DataFrames and DataSets (with examples)? The explanations is still unclear http://spark.apache.org/docs/2.4.0/sql-programming-guide.html Thanks, Roshan
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Impala
-
Apache Spark