Member since
04-12-2016
46
Posts
73
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2713 | 03-09-2017 12:27 PM | |
428 | 02-01-2017 09:54 AM | |
3948 | 07-07-2016 08:44 PM | |
3435 | 07-05-2016 10:18 AM | |
570 | 07-01-2016 06:31 AM |
03-27-2018
01:42 PM
@Sateesh Battu please check if the hive.druid.broker.address.default propert is set . in HIVE confs
... View more
03-27-2018
01:22 PM
@pk reddy give enough memory to tez AM container. tez.am.resource.memory.mb
... View more
03-27-2018
01:18 PM
@Vinit Mahiwal You can alternatively modify below configs (/etc/hive/conf) on the host where Hiveserver2 is running (and restart HS2) 'hive.txn.manager': 'org.apache.hadoop.hive.ql.lockmgr.DbTxnManager', 'hive.support.concurrency': 'true', + . (To disable authorization add below params) 'hive.security.authorization.enabled': 'false', 'hive.security.authorization.manager': 'org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory', 'hive.server2.enable.doAs':'false',
... View more
12-22-2017
10:26 AM
1 Kudo
@Anand Babu As you pointed the doc,have you followed the last step(after linking exact connector jars)
rm -rf /var/lib/ambari-agent/tmp/mysql-* (On Hivemetastore and HIveserver2 hosts -- restart services)
... View more
12-22-2017
10:09 AM
1 Kudo
@Sai Sandeep this could also occur in case hive user password issue. modify exact password in HIve configs via Ambari. drop metastore db and rerun schema tool $HIVE_HOME/bin/schematool -initSchema -dbType postgres
... View more
12-22-2017
05:21 AM
1 Kudo
@Winnie Philip When we are trying to connect to HiveServer2Interactive via ZookeeperName space, the url should contain list of zookeeper servers uris, also use correct ZookeeperName space (check in Advanced hive-interactive-site) eg: hiveserver2-hive2. Unsecure: jdbc:hive2://<zkhost1>:2181,<zkhost2>:2181,<zkhost3>:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2 Secure: jdbc:hive2://<zkhost1>:2181,<zkhost2>:2181,<zkhost3>:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2;principal=hive/_HOST@XYZ.COM **principal=hive/_HOST@XYZ.COM --> sample hive (service user) Kerberos Principal
... View more
09-28-2017
06:20 PM
1 Kudo
@Sreejith Madhavan I see the comment "running a SELECT statement in LLAP" but using Hive1.2 env, Try using Hive2 code base. Connect to HSI (Hiveserver2Interactive) url via beeline, which will make use of llap containers
... View more
09-28-2017
06:00 PM
1 Kudo
@Phaniraj G Druid-Hive integration works when you connect to HiveServer2Interactive, Hive-Druid storage handlers are available in Hive2 code base. check in /usr/hdp/<hdp_version>/hive2/lib Try executing them by connecting to HSI url via beeline.
... View more
06-29-2017
06:11 PM
2 Kudos
@Santosh Kum Use --as-avrodatafile or --as-parquetfie in the sqoop commnad line options for respective formats. Also check if you SQOOP_HOME/lib contains parqust and avro jar files.
... View more
03-27-2017
07:27 AM
1 Kudo
@zkfs hope this article might help you for correct settings https://community.hortonworks.com/articles/591/using-hive-with-pam-authentication.html
... View more
03-22-2017
07:13 PM
3 Kudos
@Harold Allen Badilla There is no disadvantage in importing a table from Sql Server directly to Hive. In fact its a single command which internally does importing data into HDFS loaction(you can specify via --warehouse-dir) creates hive table schema and Loads the data to Hive table. This create Hive table name/schema similar to the source database table. sqoop import --connect "jdbc:sqlserver://11.11.111.11;databaseName=dswFICO" \ --username sqoop\ --password sqoop \ --driver com.microsoft.sqlserver.jdbc.SQLServerDriver \ --table KNA1 \ --warehouse-dir <HDFS path> --hive-import -> additionally you can specify --hive-overwrite if you want to overwrite any exiting data on the Hive table(if exists) -> If you want to load data into a table(hive) of your choice you can use --create-hive-table --hive-table <table name>
... View more
03-09-2017
01:17 PM
2 Kudos
@elliot gimple seems this doc might help you. https://forums.databricks.com/questions/7599/create-a-in-memory-table-in-spark-and-insert-data.html
... View more
03-09-2017
12:27 PM
8 Kudos
@Padmanabhan Vijendran Can you try with '\073' select split(fieldname,'\073')[0] from table name Similar issue seen in beeswax connection used by Hue. https://issues.cloudera.org/browse/HUE-1332
... View more
03-09-2017
12:15 PM
2 Kudos
@Baruch AMOUSSOU DJANGBAN pls go through the Jira which got fixed https://issues.apache.org/jira/browse/AMBARI-18157 This was fixed in ambari 2.4 patches.
... View more
03-09-2017
12:08 PM
2 Kudos
@Tin Huynh Since the db dump is huge MR is getting timed out, try using this option in the Sqoop job to customize timeout. -D mapred.task.timeout=1500
... View more
03-07-2017
07:03 AM
3 Kudos
@Jawad Shaik pls check the connection string It should be like --connect jdbc:oracle:thin@<host>:<port>/<db_name> eg: --connect jdbc:oracle:thin@10.10.10.10:1521/orcl If Oracle driver could not be located try also using --driver oracle.jdbc.driver.OracleDriver
... View more
02-06-2017
12:56 PM
2 Kudos
@Pavani N Check the following params core-site.xml <property>
<name>hadoop.proxyuser.<loginuser>.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.<loginuser>.groups</name>
<value>*</value>
</property>
hdfs-site.xml <property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
... View more
02-01-2017
10:11 AM
3 Kudos
@Nilesh Shrimant Try to create table in parquet format , and set this config set hive.fetch.task.conversion=more; https://issues.apache.org/jira/browse/HIVE-11785 hive> create table repo (lvalue int, charstring string) stored as parquet;
OK
Time taken: 0.34 seconds
hive> load data inpath '/tmp/repo/test.parquet' overwrite into table repo;
Loading data to table default.repo
chgrp: changing ownership of 'hdfs://nameservice1/user/hive/warehouse/repo/test.parquet': User does not belong to hive
Table default.repo stats: [numFiles=1, numRows=0, totalSize=610, rawDataSize=0]
OK
Time taken: 0.732 seconds
hive> set hive.fetch.task.conversion=more;
hive> select * from repo; Option 2: There is some info here: http://stackoverflow.com/questions/26339564/handling-newline-character-in-hive Records in Hive are hard-coded to be terminated by the newline character (even though there is a LINES TERMINATED BY clause, it is not implemented).
Write a custom InputFormat that uses a RecordReader that understands non-newline delimited records. Look at the code for LineReader / LineRecordReader and TextInputFormat . Use a format other than text/ASCII, like Parquet. I would recommend this regardless, as text is probably the worst format you can store data in anyway.
... View more
02-01-2017
10:01 AM
1 Kudo
You can try specifying driver name, if it is different for the two jars. --driver oracle.jdbc.driver.oracledriver
... View more
02-01-2017
09:54 AM
1 Kudo
@Nic Hopper You can directly import table to hive, with --hive-import sqoop import --connect "jdbc:sqlserver://ipaddress:port;database=dbname;user=username;password=userpassword" --table policy --warehouse-dir "/user/maria_dev/data/SQLImport" --hive-import --hive-overwrite
It creates the hive table and writes data into it(generally managed table finally moves data to hive.warehouse.dir)
... View more
12-21-2016
02:40 PM
1 Kudo
@Praveen PentaReddy If your table has primary key by default sqoop will identify it and split by its key column. Sqoop needs a criterion by which it can split the workload. Sqoop uses a splitting column to split the workload. By default, Sqoop will identify the primary key column (if present) in a table and use it as the splitting column. The low and high values for the splitting column are retrieved from the database, and the map tasks operate on evenly-sized components of the total range Source: https://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html
... View more
08-22-2016
06:42 AM
2 Kudos
@Andrew A Cloudera Connector for Teradata 1.1.1 do not support imports from views as is documented inlimitations section of the user guide. The connector will try to create temporary tables in order to provide all or nothing semantics, which I'm expecting is the reason for the exception. If you do not have such privileges on the main database, you can instruct the connector to create the staging tables in any other database where you have enough privileges Please use this link has enough explanation. http://stackoverflow.com/questions/16855710/sqoop-teradata-connector-issue-error-the-user-does-not-have-create-table-acce Please look into last answer in the link which has enough explanation.
... View more
07-14-2016
08:53 AM
2 Kudos
@chennuri gouri shankar Main issue could be : The system was not able to create new process(es), because of the limits set for nproc in /etc/security/limits.conf file. Increase the value of "nproc" parameter for user or all user's in /etc/security/limits.d/90-nproc.conf
example of /etc/security/limits.d/90-nproc.conf file.
<code><user> - nproc 2048 <<<----[ Only for "<user>" user ]
Please use this link to debug for redhat linux flavours: https://access.redhat.com/solutions/543503 Other thing you can check is the user limit in creating processes. Check with : ulimit -a. (TO check Limits in your shell) Esp check for 'ulimit -u' max user processes,
... View more
07-11-2016
09:41 AM
1 Kudo
@Agne Zuk Need an escape sequence before first quotes (") before xxx.yyy... like below --table "_abc_def.\"xxx.yyy.xxx.yyy/xx_SALES\"" Also the other issue as one mentioned / in table name is not viable as it is used to make directory structure with table name to store MR code in Sqoop
... View more
07-11-2016
06:55 AM
1 Kudo
@Parinita Kothari Yes you can do that too.. Create a Hive paritioned table and set hive.exec.dynamic.partition.mode=nonstrict in a case of dynamic -Import table from DB to the partitioned table
here you need not specificy partition-keys and partition-values , Just -table <tblname> --hcatalog-table <tblname> is enough..
... View more
07-07-2016
08:52 PM
1 Kudo
@Josh Persinger Since the Data you got from Sqoop runs MR job it creates partfiles and logs in the particular directory. Note: When the directory contains non data file ( in HDFS), Load data fails to execute. eg: has generated a dir called my_data_dir. Under my_data_dir there are two data files /my_data_dir/part-m-00000 & /my_data_dir/part-m-00001 . There is also a log file named /my_data_dir/_logs Delete the log file and then try to execute the LOad from the directory, command should works fine.
... View more
07-07-2016
08:44 PM
3 Kudos
@Parinita Kothari To import into Multi-key partitioned Hive table , you can make use of --hcatalog-table -table <tblname> --hcatalog-table <tblname> --hcatalog-partition-keys k1,k2 --hcatalog-partition-values 'v1,v2' eg: -table DW_FACT_PUNTER_TEST_TEMP --hcatalog-table DW_FACT_PUNTER_TEST_TEMP --hcatalog-partition-keys TIME_YEAR,TIME_MONTH --hcatalog-partition-values '2014-04-11,2014-APR'
... View more
07-07-2016
02:31 PM
1 Kudo
@Dinesh E This depends on the file system we specify in the path(if you dont specify any file system it will take local file system path) generic: hive.aux.jars.path=file:///hdp/apps/2.3.4.0-3485/hive/auxjars/* HDFS specific: hive.aux.jars.path=hdfs:///hdp/apps/2.3.4.0-3485/hive/auxjars/* hope this clears your doubt, if so accept the answer
... View more
07-05-2016
10:18 AM
@Dinesh E You can set this in hive config - so that every Hive/beeline session takes this path hive.aux.jars.path=hdfs:///hdp/apps/2.3.4.0-3485/hive/auxjars/* Yes you can use HDFS path here.(recommended to to Use HDFS path... if you go for local path it should also be present in the node where HS2 is running) https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Alternate to it you can also set this: added this to the script hive-env.sh HIVE_AUX_JARS_PATH=<path of jar location>
... View more
07-01-2016
06:31 AM
3 Kudos
@Simran Kaur In sqoop --hive import --warehouse directory is the temporary HDFS directory location which collects the imported data finally it moves the data (metadata of files) to hive.warehouse.dir (generally /apps/hive/warehouse- as we specify in our hive-site.xml)
... View more