Member since
03-23-2015
1288
Posts
114
Kudos Received
98
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1519 | 06-11-2020 02:45 PM | |
2753 | 05-01-2020 12:23 AM | |
1429 | 04-21-2020 03:38 PM | |
1503 | 04-14-2020 12:26 AM | |
1126 | 02-27-2020 05:51 PM |
06-22-2020
03:17 PM
Hi @ram76, Any reason why the mv is slow? Are they on different mount points? We did see issues around 2-3 minutes delay, but unfortunately that I am not aware of a workaround. You should focus on finding out why the "move" command took this long. In my test, it took 2 minutes to copy: time cp -r /opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p3573.3750 /opt/cloudera/parcel-cache/5.13.3
real 1m59.830s
user 0m0.126s
sys 0m5.621s But my "mv' was instant: time mv /opt/cloudera/parcel-cache/5.13.3 /opt/cloudera/parcels/5.13.3
real 0m0.138s
user 0m0.000s
sys 0m0.002s Can you please share the output of : df -h So we can see the disk break down? Thanks Eric
... View more
06-21-2020
04:02 PM
@Daddy, This is a known issue in CM when either unpacking or moving of parcel directory is slow. When CM unpacks the parcel file, it performs below steps: 1. untar the parcel file under /opt/cloudera/parcel-cache 2. then "move" the parcel files to /opt/cloudera/parcels If the I/O is slow on the FS that holds directory /opt/cloudera/parcel-cache, the untar command will be slow. If the directories /opt/cloudera/parcel-cache and /opt/cloudera/parcels are mounted on different mount points, then the "move" command will become "copy" and then "delete", so operation will be much slower than the actual "move" which only involves rename. Since CM uses different threads to perform operations, it can happen that the checking happens before the untar operation finishes, hence you hit the issue. Can you please check if above is the issue in your case? So you can perform the same steps I mentioned: 1. manually untar parcel under /opt/cloudera/parcel-cache 2. run "mv" command to move directory from /opt/cloudera/parcel-cache to /opt/cloudera/parcels and time it If either of the above is slow, then you have found your issue. Cheers Eric
... View more
06-12-2020
05:27 PM
Hi Heri, Glad that it helped and thanks for the info. Cheers Eric
... View more
06-11-2020
02:45 PM
1 Kudo
Sorry, can you try below instead? select max(id) as id from NodeName a where $CONDITIONS BTW, do you really just want to import single MAX value into HDFS?
... View more
06-10-2020
03:13 PM
Hi Heri, As mentioned in the error: if using free form query import (consider adding clause AS if you're using column transformation) You max(id) aggregate function does not have "AS" clause, please change your query to below and try again: select max(id) as max_id from NodeName a where $CONDITIONS Cheers Eric
... View more
05-01-2020
12:23 AM
2 Kudos
This is old thread, but I have found a workaround, so would like to share here. Assuming I have a table with with a few partitions: SHOW PARTITIONS partitioned_table; +------------+--+ | partition | +------------+--+ | p=1 | | p=2 | | p=3 | | p=4 | | p=5 | +------------+--+ 1. create a macro: CREATE TEMPORARY MACRO partition_value() '1'; 2. create view using the macro: CREATE VIEW view_test AS SELECT * FROM partitioned_table WHERE p = partition_value(); 3. query the view: SELECT * FROM view_test; 4. if you want to update the value returned by the Macro, you need to DROP and CREATE it again: DROP TEMPORARY MACRO partition_value; CREATE TEMPORARY MACRO partition_value() '4'; 5. If you exit the session, also need to create it again in the next login, as Macro will be destroyed after session ends.
... View more
04-27-2020
03:58 PM
1 Kudo
@KokaN , Only 7 is supported in CDH 5.16.x, please see below doc: https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#c516_supported_os Cheers Eric
... View more
04-26-2020
12:40 AM
Please also share the output of below command on the Ranger host: ls -al /opt/cloudera/parcels/ Just to make sure the CDH version is linked properly. Cheers Eric
... View more
04-26-2020
12:30 AM
@Dombai_Gabor , Can you please log onto Ranger's host, go to directory /opt/cloudera/parcels/CDH/lib and then run command: grep -rni 'RangerRESTUtils' * Also under directory /opt/cloudera/parcels/CDH/jars and run the same command, just to check if file ranger-plugins-common-2.0.0.7.x.x.x-xxx.jar exist under your CDP installation. Please share the full output. Cheers Eric
... View more
04-23-2020
08:47 PM
Hi Karthik, Thanks for sharing the workaround. But since we have not tested MySQL in CDP DC 7.0, I am not sure if this is expected or not. I still suggest that you move to PostgreSQL 10 in case you will face other issues down the track. Cheers Eric
... View more
04-21-2020
03:38 PM
1 Kudo
Hi @SwasBigData , As Ferenc pointed out, In CDP, HiveServer2 and HiveMetaStore are separated into different services. The Hive Service now only contains HMS, and HiveServer2 is included in the Hive on Tez service. So to install Hive on Tez, you have to install Hive Service first, which will setup HMS. Can you please share more details on what error you got when trying to setup Hive on Tez? Thanks Eric
... View more
04-20-2020
06:35 PM
@karthikkadajji I just learnt that CDP DC 7.0 does not support MariaDB yet, only PostgreSQL 10 Are you able to either upgrade to 7.1 or change to Postgres 10 to see if that can help to resolve your issue? Cheers Eric
... View more
04-20-2020
04:33 PM
@RedOne, Thanks for the update. PostgreSQL 9 should be OK, though we recommend 10 in CDP 7, please see below: https://docs.cloudera.com/cdpdc/7.0/release-guide/topics/cdpdc-database-requirements.html I guess this is an external user? Are all external users affected? How is Ranger UserSync affected? Have you checked the UserSync log to see any obvious errors? Cheers Eric
... View more
04-19-2020
06:44 PM
@RedOne, Can you please share what backend DB you used for Ranger and its version? How many users are affected? Thanks Eric
... View more
04-14-2020
05:24 PM
@karthikkadajji , Can you please share the content of install.properties file for review, after redacting sensitive info? You can also compare the file under previous xxxx-ranger-RANGER_ADMIN-SetupRangerCommand directory and see if indeed db_password was missing. Also, have you tried to install on a different host and see if that can help? Have you checked the Ranger server log and see if there are other errors reported over there? Is Ranger web UI working? When you say unable to login into UI, what error did you see? Cheers
... View more
04-14-2020
12:26 AM
This is an old thread, but I will put in my findings recently. It is a limitation/restriction from Teradata side that data with larger than 64KB are required to use special API to be streamed into Teradata. Currently Sqoop does not make use of this API, so it does not support injecting data larger than 64KB into Teradata. Improvement JIRA has been requested, but not resolved at this stage. For the time being, have to either reduce the data or use another DB. I have tested using MySQL's CLOB has no issues. Cheers Eric
... View more
03-22-2020
03:12 PM
@ChineduLB , Did you mean that you got the same error while trying to export the sample data you provided earlier? Have you tried to update your driver in case it might be old? Cheers Eric
... View more
03-18-2020
04:04 PM
@ChineduLB Can you please share below info: - the DDL of the table in Teradata - some sample data from the file in HDFS It looks to me some issues in the data, I suggest you also narrow down the issue by slowly reducing the number of columns from the file and see which one is actually causing the issue and which rows. Cheers Eric
... View more
03-18-2020
04:01 PM
@Manus , Can you please share the error from Beeline and HS2 log? Without them, it is difficult to see what is going on. Cheers Eric
... View more
03-17-2020
07:12 PM
Hi Choon Kiat, Thanks for the confirmation. I have checked further, "Unable to obtain Principal Name for authentication" can happen when the JCE jars are not up to date on the client machine and not able to use the encryption key provided by Kerberos KDC. Can you please download the latest/appropriate JCE jars and place them in directory $JAVA_HOME/jre/lib/security. The files are: US_export_policy.jar local_policy.jar These files are dependent on the Java version. For Java8, it can be found here: https://www.oracle.com/java/technologies/javase-jce8-downloads.html Please give it a try and see how it goes. Regards Eric
... View more
03-17-2020
03:25 PM
Hi Gabor, Can you share the krb5.conf file on the HMS host for review? Cheers Eric
... View more
03-17-2020
03:19 PM
Hi Your JDBC connection string: jdbc:hive2://{host}:{port}/{database};AuthMech=1;KrbRealm=Domain;KrbHostFQDN={server};KrbServiceName=hive;KrbAuthType=2;LogLevel=6;LogPath=c:\ProgramData\MIT\Kerberos5\log.log Is "Domain" the real string or a placeholder? From your krb5, your Realm should be "AIU.XXXXXX", so I want to confirm. Cheers Eric
... View more
03-16-2020
04:00 AM
@TanChoonKiat Your JDBC Connection string is wrong, assuming that you are using Cloudera Hive JDBC Driver: jdbc:hive2://xx.xx.xxx.xx:10001/default;AuthMech=1;principal=hive/domain@domain;KrbHostFQDN=10.11.121.21;KrbServiceName=hive;KrbAuthType=2;LogLevel=6;LogPath=c:\ProgramData\MIT\Kerberos5\log.log 1. No need for "principal" 2. Can you please check what's the full principals from your KDC server? KrbHostFQDN should be normally the domain name, not IP. 3. you are missing KrbRealm, which should be the part after "@" from your principal Please try to correct those and try again. Full JDBC doc can be found here: https://docs.cloudera.com/documentation/other/connectors/hive-jdbc/latest/Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf Cheers Eric
... View more
03-15-2020
09:13 PM
@TanChoonKiat Can you please share your JDBC connection string? Cheers Eric
... View more
03-15-2020
05:02 PM
@aleesmin , Couple of questions: 1. I am not familiar with golden gate , but are those inserts and updates run inside transactions? 2. have you checked if the duplicate data in the same resulted text file in HDFS or on different files? 3. I see that you used "--direct", have you tried non-direct mode? Cheers Eric
... View more
03-11-2020
04:15 PM
Please use the query ID hive_20200308155555_9a444e04-9523-4443-b312-64a94e5a5138 and search in HS2 log to find out the exact reason for the failure. Looks like beeline did not capture the full error from server side. Cheers Eric
... View more
03-04-2020
02:42 PM
Hi @Dombai_Gabor, Can you check the keytab file under /var/run/cloudera-scm-agent/process/*HIVEMETASTORE* directories are valid and you can kinit using those keytab files without issues? Also, does the error happen on start up or after running for a while? Thanks Eric
... View more
02-28-2020
07:56 PM
Hi @HadoopBD, The log you provided is RM log, not application log that Ben asked earlier. Please run below command: yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log And then find the snippet that has the error, as the log will likely to be very big. I also noticed that the log you provided earlier contained your domain name, if you have concerns, we can redact them for you. Please advise. Thanks Eric
... View more
02-27-2020
06:03 PM
@MrCarlo , Have you tried to create a VIEW on top of the table in Oracle and then import the VIEW instead? It is supported in Sqoop as mentioned below: http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_selecting_the_data_to_import Cheers Eric
... View more