I am trying to run an insert statement in hive but it returns me this error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask The same insert runs perfect through impala. What exactly is this error ? ... View more

Hi,   Does anyone know when CDP Certified Data Developer will be released?   Thanks in advance. ... View more

I am trying to run a script in oozie and every time I receive the below error regarding impala.dbapi. The module is inserted correctly in the script.     Stdoutput Traceback (most recent call last): Stdoutput File "/tmp/sorting_table.py", line 8, in <module> Stdoutput from impala.dbapi import connect Stdoutput ImportError: No module named impala.dbapi Exit code of the Shell command 1 <<< Invocation of Shell command completed <<< java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410) at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: org.apache.oozie.action.hadoop.LauncherMainException at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:76) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104) at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:63)     Script import libraries: from pyspark import SparkContext from pyspark.sql import SparkSession from datetime import datetime,timedelta import ssl from impala.dbapi import connect import thrift_sasl import os   ... View more

Hi,   After some actions in a cluster,  oozie  in  hue  hasn't got shell action in the ACTIONS tab. It has only  HIVE SCRIPT ,HIVESERVER2 SCRIPT, SUB WORKFLOW, SSH, FS, EMAIL, STREAMING, GENERIC, KILL . Is there any way to run my shell script now?   My oozie version is 5.1.0-cdh6.3.4 ... View more

Hi,   I have one primary sqoop job which fetches data from a table_a. Now for ha purposes I want to create in case of first's sqoop failure, a second sqoop job which will fetch data from the backup table_b. So in order to achieve this I want to take the --last-value from first sqoop and parse it to the second one. Is there any option to share the last-value between two sqoop jobs ?    Thanks in advance   ... View more

Hi,  I want to ask is there any way to make oozie run a WF only if the previous execution has finished? ... View more

I am fetching data with sqoop from a MySQL table. Although some times the fetching process is happening when the table is in no readable mode(because it does its daily backup proccess). For this I have created the below: # ! / bin / bash --status query if --variable from status query is true then sqoop import... else wait 3 sqoop import... fi I want to ask, how can I store the table's status which fetched with sqoop in a variable in oder to have the if condition? ... View more

Hi,   I have a sqoop jopb in order to transafer data from MySQL to hive with incremental import. Until now it is working fine but I want to ask how can I tell the job to rerun if it will fail? Is that possible? I saw this article: https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#validation but i am not sure. Can someon give an example? ... View more

Hi,   After restarting impala I ran some queries and two of them threw an error for to_timestamp function that I have in my view. If I rerun these queries everything runs smoothly. Why is this happening? It is like impala can not read the functions in the first-run of the queries.     Thanks in advance ... View more

Hi,   I am running a query and despite the fact that it reached the limit in (8 mins) and it seems FINISHED inside the query details in CM, it stops at (19 mins). Whys it this happening? I saw the below information in query details:   First row fetched: 7.9m (75ms) Last row fetched: 8.0m (7.21s) Released admission control resources: 19.5m (11.6m) Unregister query: 19.5m (103ms) ... View more

Hi,   I have create a UDF in impala but every time I am restarting it the UDF is gone. Is there anyway to fix this issue ?      c reate function my_udf(String,String,String) returns boolean location 'udfs-1.0-SNAPSHOT-jar-with-dependencies.jar' SYMBOL='MyUDF ; ... View more

I am trying to learn some basic things in sqoop and I want to insert some data from a mysql table  into hive. This Mysql table takes data every 5 mins. I found how to create sqoop job in order to connect and run the query but I can not understand how the sqoop will know the last-value from the primary key column in order to extract the newer data every time.    https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports   For example in the below sqoop command do I have to put the last value or the sqoop can understand it from its own? The check-column must be the primary key column?   sqoop job --create <JOBS NAME>\ --import \ --connect "jdbc:<PATH>" \ --username <USERNAME> \ --password <PASSWORD> \ --target-dir <DIR> \ --table <MYSQL TABLE>\ --hive-import \ --hive-table <HIVE TABLE>\ --fields-terminated-by , \ --escaped-by \\ \ --split-by <COLUMN TO BE SPLITED IN MAPPERS> \ --num-mappers -5 \ --incremental append \ --check-column \ --last-value     ... View more

Hi,   I am new in Kafka and I have some questions. I want to create a pipeline which extract data from Mysql to kafka. Although after a lot of search I found something about CDH which is when something in the source table changes to send the updated record to Kafka. I do not want such a thing. I only want to extract data according to primary key.  Is there any course to do in order to learn or can anybody provide any tips?   Thanks in advance. ... View more

I am trying to run kafka-server-start.bat but I get the below error:   ERROR Disk error while writing recovery offsets checkpoint in directory C:\kafka_2.13-3.0.0\data\kafka.   Zookeeper runs in the background successfully   How can I fix this? ... View more

I have some tables in   hive   and I want to find the size of each table through the metastore   (MySQL). I am trying the below but it returns paratition_params, notification_log, sds etc. My tables are stored in   TABLES.TBLS   but when I am running the below query with   from information_schema.TABLES.TBLS   it returns this: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.TBLS ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC LIMIT 0, 200' at line 5 SELECT TABLE_NAME AS ` Table `, ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024 ) AS `Size (MB)` FROM information_schema.TABLES ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC ; ... View more

I have some tables in impala and in the create tables statement they have sort by( timestamp column) but when I query them they are returning the data not sorted. Why is this happening?   For example  2019-09-18 08:44:10 2020-08-05 13:15:48 2020-08-05 13:24:00 2020-10-15 18:29:34 2020-09-09 09:35:04 ... View more

Hi,   I have a table in Impala and it had some files in its location in hdfs. All of a sudden it can not read the files. I tried to invalidate and recover the partitions but nothing happened. I also moved the files from this folder into a new's folder path but it couldn't read them too.   What can I do ?   UPDATE Tried to repair the table in hive set hive.msck.repair.batch.size=1; set hive.msck.path.validation=ignore; msck REPAIR TABLE db.table but received this error : Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask    ... View more

I have a table and when I am trying to query it through imapala it throws me this error: File 'hdfs:/path/table/1_data.0.parq' has an incompatible Parquet schema for column 'db.table.parameter_11'. Column type: STRING, Parquet schema: optional int64 amount [i:10 d:1 r:0] But when I query it through hive it returns results correctly. How can I fix it? ... View more