For the sake of creating POC, i have create local groups in the policy file sentry-provider.ini [databases] # Defines the location of the per DB policy file for the customers DB/schema dw_arcticblue_staging = hdfs://quickstart.cloudera:8020/home/cloudera/Desktop/dw_arcticblue_staging_policy.ini [groups] # Assigns each Hadoop group to its set of roles group1 = analyst_role group2 = admin_role group3 = analyst_role[users] user1 = group1, group2, group3 user2 = group2, group3 [roles] # The uris below define a define a landing skid which # the user can use to import or export data from the system. # Since the server runs as the user "hive" files in that directory # must either have the group hive and read/write set or # be world read/write. analyst_role = server=server1->db=dw_arcticblue_staging ->table=*->action=select admin_role = server=server1->db=dw_arcticblue_staging ->table=*->action=select, \ server=server1->db=dw_arcticblue_staging ->table=*->action=Insert # Implies everything on server1 -> customers. Privileges for # customers can be defined in the global policy file even though # customers has its only policy file. Note that the Privileges from # both the global policy file and the per-DB policy file # are merged. There is no overriding. arcticeblue_admin_role = server=server1->db=dw_arcticblue_staging # Implies everything on server1. admin_role = server=server1 DW_ArcticBlue_staging_policy.ini [groups] group1 = dw_all_access [roles] dw_all_acess = server = NirvanaServer -> db = dw_arcticblue_staging -> table = * -> action = Insert,\ server = NirvanaServer -> db = dw_arcticblue_staging -> table = * -> action = Select I have updated all configurations in hive and yarn as recommended. Few things which i dont understand How to test the user1,user2 that whether they are authorized or not ? I want to test from hive / impala cli .User1, User2 etc created in the policy file are not real users to login with. We need to create a HDFS sentry-provider.ini file. How we could create ini file in linux? i did not find any relevant document Any help how to test the authorization using sentry on hive??? ... View more

Hi, On the staging database we are getting updated data from relational database. As HDFS/Hive supports append only. So we need to delete the partitions where data is modified and replace with updated data partitions. Lets say we have a T_UpdatedData table which gets populated by Sqoop per day. If on any data we have updated data from multiple partitions, we need to backup, delete those partitions from the T_ProductionData. Partitions are created on Year, Month, Account columns. Backup should be saved on the T_HistoryData How to can select the different partitions from the T_UpdatedData & copy those partitions from T_ProductionData to T_HistoryData ? Inserting dynamic partitions can be done with following syntax. INSERT OVERWRITE TABLE tablename1 PARTITION (partcol1=val1, partcol2=val2...) Two ways to select the distinct partition data could be 1. SELECT * FROM T_ProductionData WHERE (year,month,account) IN (SELECT distinct year, month, account FROM T_UpdatedData) -- Supported by Oracle. 2. Using joins. Using joins could be more costly as it will try to perform join in all T_ProductionData. What is the best practice to backup the partitions and insert the modified partitions ? Thanks ... View more

Hi, I have been copying data from non-partitioned hive table to partitioned hive table but its giving following error and log message. Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask Log : Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : number of splits:2 INFO : Submitting tokens for job: job_1455271075351_0034 INFO : The url to track the job:http://quickstart.cloudera:8088/proxy/application_1455271075351_0034/INFO : Starting Job = job_1455271075351_0034, Tracking URL =http://quickstart.cloudera:8088/proxy/application_1455271075351_0034/INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1455271075351_0034 INFO : Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0 INFO : 2016-02-18 03:56:36,684 Stage-1 map = 0%, reduce = 0% INFO : 2016-02-18 03:56:45,448 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 2.35 sec INFO : 2016-02-18 03:57:10,663 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.35 sec INFO : MapReduce Total cumulative CPU time: 2 seconds 350 msec ERROR : Ended Job = job_1455271075351_0034 with errors ... View more

Hi, i have import sql server data through Sqoop into hdfs into -target-dir /user/hive/warehouse/db_abc.db/t_tableName . I have import into hdfs as i need to import using the --query switch. Now i want to import data from hdfs into hive. I have tried LOAD DATA INPATH '/user/hive/warehouse/db_abc.db/t_tableName' INTO TABLE db_abc.t_staging_table Above command throwing invalid path : no files matching path error. How i could load data from hdfs to hive or sql server to hive through sqoop? Though i did not find any way to directly import into hive where --query switch is used in Sqoop. Thanks ... View more

Hi, We are performing some sort of stress testing. How fast data can ingested in running 5 parallel imports? We have many clients so during POC, we like to check how much load we can put during data ingestion in parallel. While i was importing the data from single sql server instance it was working fine. Now while i am trying to import data from 5 sql server databases on the same sql server instance using 5 different sqoop import command, no job is finishing its job. All are showing the import failure message. We have oracle VM virtual box setup with following resources 1. RAM : 24 GB 2. Disk : 256 GB 3. Processors assigned : 6 (Hypertheading enabled) What could be potential reasons for the data ingestion failure? Do i need to have more resources assigned to the VM ? Going to real physical cluster is not an option yet for us. Cloudera forum is mostly unresponsive that why i have been posting questions here ... View more

Hi , We are connecting Hadoop cloudera CDH distribution through ODBC driver. Queries are generated from SSRS. Few queries are working fine with parameters augmented through ? placeholder. Few other queries with parameters augmented through ? are not executing. Error [HY000][Cloudera][ImpalaODBC] (100) error while executing a query in Impala[HY000] : AnalysisException : syntax error in line 1 where Date >= ? and Date <= ? ^Encountered : Unexpected characterExpected : Case... Exception : syntax error. If i remove where Date >= ? and Date <= ? or supply the hard coded value then query is working perfect. Few other queries with same filter are working perfect. 1. What should be recommended investigation points? 2. Where could i get the exact impala transformed query to investigate whether query is generated correct or not ? ... View more

Hi, We have some fact tables which contains large number of rows. We have partition applied on the month right now. It is most more likely that in coming future we might need to apply partition by week number. As update command is missing in Hive so whenever there is situation to update the historical data we just drop the partition & create a new partition. So applying partitions is necessary. I am wondering is applying partitioning on the existing columns in hive table POSSIBLE ? How to handle the situation where we have to apply the partitioning dynamically based on the load ? i think dropping the table & recreating table for most of the requirement is not good thing ... View more

Hi, I am new to Hadoop ecosystem. For data ingestion into hive, we are using the sqoop import commands. Data ingestion populates data into staging tables. Now we need to clean up data & insert into the production hive tables. I have written the hive udf to simulate the auto increment feature which works fine in hive shell. Hive query is taking very long to clean up data, generate auto_incremented number. Impala queries are working good. I m wondering if i can use the same hive defined udf in the impala . Is there any way to use the hive udf in the impala shell to generate the auto_incremented number? Thanks ... View more

Hi, We are ingesting data from SQL Server (OLTP) to Hive. Incremental updates for updated data not working against SQL Server. It's still open issue https://issues.apache.org/jira/browse/SQOOP-2421 Is there any work around to get the updated data into Hive tables? Thanks ... View more