Member since
04-03-2017
164
Posts
8
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1005 | 03-09-2021 10:47 PM | |
2180 | 12-10-2018 10:59 AM | |
4455 | 12-02-2018 08:55 PM | |
5193 | 11-28-2018 10:38 AM |
03-09-2021
10:47 PM
1 Kudo
Hi, I am able to replicate this in my cluster.. But I tested in CDH 6. Shell output:- [root@host-10-17-102-176 hive]# locale LANG=en_US.UTF-8 LC_CTYPE=UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= Oozie Launcher, capturing output data: ======================= LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= To fix this kindly make the below configuration change. Access the CM and navigate to the Yarn Configuration > Containers Environment Variable (yarn.nodemanager.admin-env) --> And append these properties "LC_ALL=en_US.UTF-8,LANG=en_US.UTF-8" to this config. Restart the affected services to make the changes permanent. Post this kindly re run the oozie job and check the output. In my cluster it shows like this post making the change. Oozie Launcher, capturing output data: ======================= LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 Nitish
... View more
03-09-2021
10:19 PM
1 Kudo
Hi, What's the CDH version you are using currently on which you are seeing this issue? Can you share the workflow.xml and the script that you are running? Also kindly share the oozie launcher logs. Regards Nitish
... View more
04-11-2020
03:19 AM
Hi,
NOTE:- Parquet is hard coded to write the temporary data in /tmp even though the target directory is different.
Kindly check /tmp for intermediate data, you will see it there.
Regards
... View more
11-10-2019
10:56 PM
Hi, Can you please share the sqoop command that you are running? Regards Nitish
... View more
11-07-2019
09:10 PM
Hi, To solve these kind of issue there are 2 options. 1. Either clean the special characters at the DB level. 2. Kindly use any binary format import in sqoop like avro and parquet. In these format the line won't be splitted like it does in text format. Regards Nitish
... View more
11-07-2019
09:04 PM
HI, I see that you are passing -m3 but there is no split by in the sqoop command. Can you please add that or if not then can you please try to run with -m1 It if fails again then please upload the full application logs. Regards Nitish
... View more
11-07-2019
08:59 PM
Hi, You can use this tag " --temporary-rootdir" to make sure that the temporary data goes into this folder. Example:- [example] sqoop import --target-dir /<hdfs path>/<import_data_dir> --temporary-rootdir /<dfs path>/<tmp_dir> ... Regards Nitish
... View more
09-30-2019
08:44 PM
Hi, Can you please share the exact error that you are seeing? Also kindly share the exact line so we can test the issue in house. Regards Nitish
... View more
09-30-2019
07:13 PM
Hi, Can you please share the oozie launcher logs? I am worried the --hadoop-classpath you are passing is not correct. Regrads Nitish
... View more
09-30-2019
07:10 PM
Hi, Can you please confirm how you are running the oozie job? Is it from command line or HUE? If it is command line then kindly do the kinit prior running the oozie job so that the oozie will use its own keytab and provide user the delegation token. Regards Nitish
... View more
09-30-2019
07:06 PM
Hi, Sqoop is a client and not a Service. You can check HDFS audit logs which will tell what all files have been used. Regards Nitish
... View more
09-30-2019
07:04 PM
Hi, Spark is a tool used for processing the data and Sqoop is used to import/export. NOTE:- Both the tools have different behaviours and are used accordingly. Hope able helps. Regards Nitish
... View more
09-30-2019
07:00 PM
Hi, You can try like below example. Data in source ########### [root@host-10-17-103-77 ~]# sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table test_partition(id int,name varchar(30),par varchar(20))" [root@host-10-17-103-77 ~]# sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into test_partition values(1,'nitish','par1')" [root@host-10-17-103-77 ~]# sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into test_partition values(2,'mohit','par2')" [root@host-10-17-103-77 ~]# sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into test_partition values(3,'mohit123','par3')" Table in Hive:- ############ 0: jdbc:hive2://host-10-17-103-79.coe.clouder> create external table test_sqoop_par(id int,name string) partitioned by (par string) row format delimited fields terminated by '\t' location '/user/systest/test_sqoop_par'; NOTE:- partition column as string is only supported. Sqoop command:- ############## [root@host-10-17-103-77 ~]# sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --hcatalog-database default --hcatalog-table test_sqoop_par --table TEST_PARTITION -m1 --hive-partition-value par Data in Hive:- ########## 0: jdbc:hive2://host-10-17-103-79.coe.clouder> show partitions test_sqoop_par; +------------+ | partition | +------------+ | par=par1 | | par=par2 | | par=par3 | +------------+ [root@host-10-17-103-77 ~]# hadoop fs -ls /user/systest/test_sqoop_par Found 3 items drwxr-xr-x - systest supergroup 0 2019-09-30 18:49 /user/systest/test_sqoop_par/par=par1 drwxr-xr-x - systest supergroup 0 2019-09-30 18:49 /user/systest/test_sqoop_par/par=par2 drwxr-xr-x - systest supergroup 0 2019-09-30 18:49 /user/systest/test_sqoop_par/par=par3 Hope above helps. Regards Nitish
... View more
08-26-2019
07:49 AM
Hi, Not sure at this point what is the issue on that host as I am not able to debug the host related issue. Have you configured Sqoop gateway on both the hosts? Regards Nitish
... View more
08-26-2019
12:15 AM
This means that from the master node the job is failed to be created but when you are running the job from slave node then it is getting created. Am I right? If yes then I would request you to check the difference b/w the hosts. Also check if the sqoop gateway has been deployed? If yes then on what node. There is some setup issue in your cluster which is causing this. Regards Nitish
... View more
08-26-2019
12:08 AM
Hi, I would request you to configure Sqoop gateways on that hosts and install metastore. KB:- https://my.cloudera.com/knowledge/Creating-and-Executing-Sqoop-Jobs-Remotely-|-Sqoop1-Shared-Metastore-Database?id=70942 Regards Nitish
... View more
08-25-2019
11:20 PM
Also, If you run the same job from other node then it fails with the same message? Can you please confirm? Regards Nitish
... View more
08-25-2019
11:15 PM
Hi, Did you install the sqoop metastore? Any configuration changes you made in Sqoop configurations via CM? Regards Nitish
... View more
08-25-2019
10:46 PM
Hi, Can you please install Sqoop gateway on that node. We always recommend to install sqoop gateways on that host on which you are running the Sqoop commands. Link:- https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_mc_sqoop1_client.html Also can you please confirm if you are able to run the normal sqoop import commands from that host? Can you run :- sqoop job --list and share the output of this? You have enough disk space on that host ? Because Sqoop needs to create the local directories as a metastore? Regards Nitish
... View more
08-25-2019
10:23 PM
Hi Akhila, This is the output of what command? Can you please run the below commands from the host and share the output. ## ls -ltr /var/lib/sqoop ## ls -ltra /home/<username> Regards Nitish
... View more
08-25-2019
09:21 PM
Hi, Can you please share the full stderr log file so that can give more light on the error? Is this the first time you are installing oozie server? Regards Nitish
... View more
08-25-2019
09:20 PM
Hi, You will see the captured output in "Stdout" logs. Please let us know if you are not able to see in that section as well. Regards Nitish
... View more
08-25-2019
09:18 PM
Hi, Generally we recommend to use spark action as it is available. If there is something that spark action can't do then you can have a java action which will run your spark job. Regards Nitish
... View more
08-25-2019
09:17 PM
Hi, For this I would request you to place the keytab file in HDFS and just reference the name in the <spark-opts> Example <spark-opts> --principal <abc> --keytab <abc.keytab> </spark-opts> <file> <path of HDFS keytab></file> NOTE:- Do add the <file> tag which will be pointing to the location of keytab on HDFS. This will localize the keytab file and will use in the oozie spark action. Kindly try the above and let us know how it goes. Regards Nitish
... View more
08-25-2019
09:13 PM
Hi, Hope you are doing good !!! Can you please confirm how you are creating the workflows? Is it from HUE? Can you share the screenshot with us so that we can check what steps you are performing? Regards Nitish
... View more
08-25-2019
09:11 PM
Hi, It looks like the spark job that is started is getting failed. Can you please share the logs for the Spark job. The application ID is mentioned in the snapshot for the spark as it gets initiated but after that it fails. ## yarn logs -applicationId <application ID> -appOwner <owner name> Kindly copy the logs in notepad and attach to the case. This will give more leads on this. Regards Nitish Regards Nitish
... View more
08-25-2019
09:07 PM
Hi, Can you share the sqoop command? Also does this same sqoop command used to work fine in C5? Regards Nitish
... View more
08-25-2019
09:05 PM
Hi, Can you share the output of below command from the host on which you are running the Sqoop job? ## ls -ltr /var/lib/sqoop ## ll -aR /home/<username> This will help us to check the permission on these folders. Regards Nitish
... View more
07-24-2019
11:10 PM
Hi, You can use couple of parameters for strings columns and non string column. ## string columns:- --null-string ## non string columns:- --null-non-string Link:- https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_null_string_handling Above is the link will help you to handle these characters. Regards Nitish
... View more