Member since
03-07-2019
151
Posts
17
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
768 | 03-27-2019 04:52 AM | |
3464 | 11-21-2018 10:21 PM | |
8312 | 09-14-2016 07:35 PM | |
6662 | 07-01-2016 06:56 PM | |
1316 | 06-07-2016 04:22 PM |
09-13-2018
11:05 PM
@Roger Sliva, Yes that's another way of getting it done.
... View more
09-13-2018
07:06 PM
1 Kudo
Since Hive client is removed, using hive-import, hive-create-table and things like that will wait for credentials to be entered. if using sqoop CLI and not via oozie, then supply the hive credentials just as you do via beeline and this would do the needful.. If this is not acceptable then you would need to change the sqoop syntax to use hcatalog options to achieve the same.
... View more
09-12-2018
06:58 PM
1 Kudo
@n c its because that TBLPROPERTIES should the last one, use the below and it should help: +++++++++++++++++ CREATE EXTERNAL TABLE staging3 ROW FORMAT SERDE 'org.apache.hadoop.hive.serd2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION '/tmp/staging' TBLPROPERTIES ('avro.schema.url'='hdfs:///tmp/avroschemas/testtbl.json'); +++++++++++++++++
... View more
03-23-2018
04:50 PM
@Haitam Dadsi Thanks for the information. with the provided info, it seems like you would need to use sql adaptor of some sort to track the changes and push it out. I did come across these articles which can help you out (I guess) and pass in the information to the cluster using somesort of listener which can read the data from the intermediate client/server. https://github.com/christiandelbianco/monitor-table-change-with-sqltabledependency https://www.codeproject.com/Articles/63568/HTTP-Push-from-SQL-Server-Comet-SQL
... View more
03-22-2018
09:59 PM
1 Kudo
@raghavendra v A similar question was answered on another post here: https://community.hortonworks.com/questions/36868/ranger-audit-logs-copy-to-a-local-folder.html
... View more
03-22-2018
09:53 PM
@Haitam Dadsi I see there could be 2 approaches to this one. 1. One use sqoop to import the dataset from sql server to hdfs and then you can build a dashboard on the same and after that use incremental import to import the changes. (or) 2. Use NiFi with Database processor to get the records from DB and using store that at relevant location.
... View more
03-22-2018
03:46 PM
@Mohan Kumar Running a psql script using shell action would need the psql command to be available on all the NodeManagers and the user launching the oozie job, needs to be able to launch the psql command across the NM's. Can you check to see if psql is available across all the NM's? Also, it would be great if you can also post in the exception that you are seeing from the oozie launcher job for this shell action as shell action would be executed inside the oozie launcher.
... View more
03-22-2018
03:42 PM
@Carol Elliott, I think the issue is related to the jt400.jar which is launching java awt windows to lauch a pop-up for login. I don't this this would work for you in this case. As you are connecting to DB2, I would suggest to use DB2 JDBC driver for ISeries, specifically these two jars: Type 4 JDBC driver: db2jcc4.jar and a license file: db2jcc_license_*xxx* .jar. IBM article link which talks about the same: http://www-01.ibm.com/support/docview.wss?uid=swg21959125
... View more
03-21-2018
11:54 PM
@Carol Elliott May be you can try using this option to see if this works: f you're on the main display, then export DISPLAY=:0.0 or if you're using csh or tcsh setenv DISPLAY :0.0
... View more
03-20-2018
11:26 PM
@Sana Faraz You would need to check the Yarn RM UI and see if there was any application launched by oozie. The name of the job would be oozie launcher, you would need to drill in to the task logs to see if there was any exception. By default the launcher job would say succeeded, you can ignore this status and drill down to see the exception trace. Or you can issue the command: yarn logs -applicationId <applicationId_FromRM_UI>, this would need to be run from command line after login to the sandbox cli.
... View more
08-10-2017
09:45 PM
@prakyath pai, Can you check the Hive Interactive site with 'llap' in the filter section under the Hive configs and see if any of the memory config there set to 0. If Yes, this would be the cause of it.
... View more
08-10-2017
09:33 PM
@Jyoti Rathod There are 2 versions of Sqljdbc4 drivers one for Java 7 and one for Java 8. The driver available in the sqoop/lib location would be built to be used using Java 7. You might need to update the sqljdbc jar file to be used with Java 8. You can get the driver from here: https://www.microsoft.com/en-us/download/details.aspx?id=54671
... View more
08-10-2017
09:26 PM
@sai harshavardhan The value that you are receiving is as expected, this sample java code show the info. If you are wanting to get the exact value as an output, you should convert that value to string. import java.lang.*; public class DoubleDemo { public static void main(String[] args) { double d = 12345678910.231; System.out.println("Value of d = " + d); } }
... View more
08-03-2017
08:44 PM
@Aaron Dunlap You may want to check the Number of executors per LLAP Daemon and hive.llap.io.threadpool.size and make sure that they match. If the threadpool.size is set to 2 ensure that the Number of executors is also set to 2 via Ambari and restart the service after saving the config.
... View more
02-14-2017
07:00 PM
@bsaini Hive always takes timezone into consideration. May be what you want to try is to set that field as String which will retain the same value.
... View more
02-09-2017
10:24 PM
@Shashant Panwar Here is the link which talks about the HDFS quota. http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html -Mahesh
... View more
02-09-2017
09:36 PM
@Shashant Panwar I dont understand the usecase of limiting a database to use a X percent. Lets assume you have a table which needs more than 25% of the disk space then you wont be able to do anything unless you delete and free up the space on that DB to allow data into the table. Also one cannot set limitation at Hive level to state how much space a database can use. It needs to be set at HDFS user level quota. -Mahesh
... View more
09-28-2016
06:50 PM
The reason this exception comes because that the Pig needs a local Tez client, if its not available then this exception is thrown even though the tez.lib.uris is present and the hdfs has the tez.tar.gz under correct hdp release.
... View more
09-14-2016
07:35 PM
@Sonny Heer Can you do this to see if the table does exist in hive. hive -e 'use mydb; show tables;' and see if this lists the table test1. If not then this would state why you are seeing the exception.
... View more
07-01-2016
07:26 PM
@suresh krish Yes, you will not be able to set this at runtime unless you have included that in the whitelist until this is done and you have restarted hive service along with HiveServer2 this would not take effect.
... View more
07-01-2016
06:56 PM
@Johnny Fugers From what I understand is that you have set of files inside a hdfs directory which is split by dates, would the directory structure look like this: /user/test/data/2016-03-12/<FILES> /user/test/data/2016-03-13/<FILES> /user/test/data/2016-03-14/<FILES> Like these, if yes then you will not be able to create a partition based table, what you can do is to create a regular table pointing the location to /user/test/data and set the below properties, this will look at all the files inside the subdirectory and get the result set. set hive.input.dir.recursive=true; set hive.mapred.supports.subdirectories=true; set hive.supports.subdirectories=true;
set mapreduce.input.fileinputformat.input.dir.recursive=true;
... View more
06-07-2016
10:06 PM
@Tim Dobeck You can look at this http://gethue.com/hue-3-7-with-sentry-app-and-new-search-widgets-are-out/ which has the information of how to perform the install along with the tarball of 3.7
... View more
06-07-2016
04:22 PM
3 Kudos
@Sagar Shimpi By default the shell actions are not allowed to run as another user as sudo is blocked. If you want a yarn application to run as someone other than yarn (i.e. the submitter), then you need to enable the linux container executor so that the containers are started up by the submitting user. Also note the below setting information which also needs to be changed as well to achieve this. With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody') With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow. Stating that there are issues around this also where it does not work as expected because of the issues
https://issues.apache.org/jira/browse/YARN-2424 https://issues.apache.org/jira/browse/YARN-3462
The current suggestion that I can make is to add line to change the ownership of the file which was created using shell.
... View more
05-31-2016
08:38 PM
@sankar rao You may want to visit these links as they are documented about the use of different Authorizations. Hive Authorization Storage Based Authorization in the Metastore Server
... View more
05-12-2016
09:03 PM
@Sushil Saxena It seems like there are likely to be multiple versions of hive jars. When I state multiple version of hive jars meaning there different hdp release hive jars in the oozie share lib location. Was this a upgraded cluster from 2.1.x/2.2.x to 2.3.4 release?. One thing that I can suggest here is to perform the following provided you have not copied over thirdparty jars like oracle jar/ mysql jar file to the oozie share lib location. If you have then you would need to copy over all the 3rd party jars again to hdfs. From the oozie server host, using oozie user: 1. hdfs dfs -rm -r /user/oozie/share/lib -> to remove the lib folder completely 2. cd to /usr/hdp/current/oozie-client/bin 3. run "./oozie-setup.sh sharelib create -fs <get the fs.defaultFS from core-site.xml>" 4. Restart Oozie service This should help to address the issue.
... View more
05-12-2016
04:07 PM
@simran kaur, it seems like this is a duplicate of the post https://community.hortonworks.com/questions/30437/error-in-oozie-class-orgapacheoozieactionhadoopsqo.html#answer-33008 where you were able to get past the issue. If you do, please update the solution here and close this thread.
... View more
05-12-2016
04:05 PM
@simran kaur, There are couple of ways to address this. One as @rbiswas mentioned and the other one is to create a lib folder where the workflow is receding in HDFS and place the mysql connector jar file there. For example, if your workflow is located in hdfs at /user/simran/sqoopwf/workflow.xml. Take the path "/user/simran/sqoopwf/" and create a lib folder here hdfs dfs -mkdir -p /user/simran/sqoopwf/lib and then place the mysql connector jar to this location hdfs dfs -put <mysql-connector-java-version.jar> /user/simran/sqoopwf/lib/. Then kick off the oozie job which should work.
... View more
05-11-2016
11:06 PM
@Chris McGuire I'm not sure whether this would work on saveAsTable command since I have very limited to no knowledge on spark. I'm hoping that this property should work for the spark streaming job as well.
... View more
05-11-2016
09:29 PM
@Mamta Chawla, Hive does not do data validation based on the fields. Its the users responsibility to check the data and see if it matches with the table being created. May be this link can help you out with what you are looking for: https://community.hortonworks.com/articles/1283/hive-script-to-validate-tables-compare-one-with-an.html
... View more
- « Previous
-
- 1
- 2
- Next »