Member since
09-25-2015
82
Posts
93
Kudos Received
17
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4001 | 06-06-2017 09:57 AM | |
1136 | 03-01-2017 10:26 AM | |
1175 | 11-22-2016 10:32 AM | |
955 | 08-09-2016 12:05 PM | |
1739 | 08-08-2016 03:57 PM |
03-30-2016
03:06 PM
have you verified that the sqoop command works by itself? run it manually on the command line outside of oozie. are you running your workflow through hue or via the oozie command line? if through hue, try running it on the oozie command line to verify it works as well.
... View more
03-30-2016
03:02 PM
2 Kudos
Hi @nejm hadj does NiFi write the files to one directory? You should be able to specify an EXTERNAL table with the location as a directory. Then all the files in the dir will be part of the one table. For example: CREATE EXTERNAL TABLE tweets (col1 string, col2 string)
LOCATION '/path/to/dir'; Obviously use a JSON SerDe to parse the data into your table columns as well, but the LOCATION clause is sufficient to load all your files into your table.
... View more
01-20-2016
11:08 PM
1 Kudo
@Neeraj Sabharwal I don't have a doc or demo, but it's simple enough to demonstrate. Enable SBA (should be already by default) and and then give the hive warehouse dir e.g. 750 permissions for hive:hadoop (recursively). Then if people try to access tables through Hive CLI as any other user, they will get a permission denied error. This won't apply to access via HiveServer2 because you'd manage table and DB permissions via Ranger Hive repository or SQL Standard authorisation. @Alex Miller's suggestion goes on top of that as well. To be truly secure, users must come in via Knox.
... View more
01-20-2016
12:09 PM
5 Kudos
I think you have to make sure Storage Based Authorization (SBA) is enabled for the Metastore
hive.metastore.pre.event.listeners = org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener hive.security.metastore.authorization.manager = org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider And then lock down filesystem permissions for data warehouse dir. I don't think there's a way to actually prevent users from issuing the "hive" command. I may be wrong, though - happy to be corrected.
... View more
01-15-2016
09:38 AM
3 Kudos
Hi @Junichi Oda - This is expected behaviour and it is the reason why it is recommended to have all hive processes run as hive user when you secure Hive with ranger. There are two options in order to secure access to hive with Ranger : Solution 1 Use both a repository HDFS and Hive to handle rights
Keep "run as end user instead of hive" (hive.server2.enable.doAs=true) This means the dual maintenance that you describe Solution 2 Give rights to the hive user on the /apps/hive/warehouse arborescence in Ranger HDFS repository Lock down filesystem permissions on HDFS (for example, chmod 750) Use the Ranger Hive repository to handle rights on Hive tables Run as hive instead of end user (hive.server2.enable.doAs=false) --- Solution 2 is the way to go. You may be concerned about auditability, but the Hive audits in Ranger will show the correct user. The HDFS audits and the YARN audits will still show "hive" yes, but you will be able to tell who ran the query.
... View more
12-24-2015
12:48 PM
1 Kudo
Did you attempt and abort an upgrade already? Can you please check the clusterconfigmapping table in the Ambari DB and make sure there are no duplicates in the type_name column? If there are, and then make sure that only 1 of them has a value of "1" in the "selected" column - the one with a later created_timestamp value. Take a backup of the Ambari DB (!!) and then update the "selected" value to 0 for the other one. Then re-attempt the upgrade.
... View more
12-22-2015
06:06 PM
2 Kudos
EDIT: oops - just read the end bit of your post... still try to do it in the shell action itself and see? --- I believe this is a known limitation in non-secure clusters, whereby the containers are running as YARN user and not running as user xyz. Try to pass in the HADOOP_USER_NAME variable as illustrated below: $ `export HADOOP_USER_NAME=xyz ; hive -hiveconf hive.execution.engine=mr -e 'show databases' `
... View more
12-22-2015
05:56 PM
5 Kudos
According to this Hive Language documentation, you can achieve the same thing using regular expressions: A SELECT statement can take regex-based column specification in Hive releases prior to 0.13.0, or in 0.13.0 and later releases if the configuration property hive.support.quoted.identifiers is set to none. * We use Java regex syntax. Try http://www.fileformat.info/tool/regex.htm for testing purposes. * The following query selects all columns except ds and hr. SELECT `(ds|hr)?+.+` FROM sales
... View more
12-10-2015
05:47 PM
Hi @Bhupendra Mishra You don't need the --table argument - it's only relevant for imports and exports (not eval). Try this: sqoop eval --connect jdbc:mysql://localhost.localdomain/sqoop_test --username root --query "SELECT * FROM emp"
... View more
12-10-2015
12:10 PM
Do you have Kerberos enabled on this cluster? Also - are you using HDP 2.3.0 or HDP 2.3.2?
... View more