Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Permission Error: To load into Hive using NiFi

Permission Error: To load into Hive using NiFi

We are trying to load data from Hive External table to Hive Managed table and facing Hive permission issues. NiFi uses user: nifi to load into hive that's where its crashing.

Note: Avro files are been created after pulling from source Hive External table.

Please refer to the screenshot attached.

86604-puthivestreaming-error.jpg

7 REPLIES 7
Highlighted

Re: Permission Error: To load into Hive using NiFi

Super Guru
@Mustafa Ali Qizilbash

Could you please update the question with SelectHiveQL, PutHiveStreaming configs and DDL for PutHiveStreaming table.

Highlighted

Re: Permission Error: To load into Hive using NiFi

Dear @Shu

Please find screenshots.

86615-selecthiveql.jpeg

86616-puthivestreaming.jpeg

Highlighted

Re: Permission Error: To load into Hive using NiFi

@Mustafa Ali Qizilbash

take a look at the following post, it may be relevant to your issue:

https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.htm...

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Permission Error: To load into Hive using NiFi

Super Guru
@Mustafa Ali Qizilbash

As your SelectHiveQL processor Max rows flowfile set to 0 if you are having huge data then it's better to use SplitAvro processor to Split into 1000 records(because HiveStreaming configured records per transactions is 1000) for each flowfile then feed to PutHiveStreaming processor.

--> Try with small table name instead of long table name that you are having now in PutHiveStreaming.

-->If you are having HA enabled for Hive Meta store then mention all the meta store URI with comma separated so that Hive Streaming processor will try to connect another meta store if the configured meta store is busy.

-->If the error still exists then try to use String datatype (although which is not optimal but to debug this issue you can try this)for all the columns and try to insert data into table, if this works then find out the column that having issues with the data types.

Refer to this link for more details regards to Hive Streaming Datatypes.

Highlighted

Re: Permission Error: To load into Hive using NiFi

@Shu

Still same error. Done following as per your direction.

  • Reduced Table Name
    • From: UFM_Intra_Cell_Handover_Measurement_Per_Cell
    • To: ufmichmpc
  • There is only 1 row in the source table for testing, so SelectHiveQL processor Max rows flowfile set to 0 is not an issue, anymore.
  • There is only 2 columns in target table for testing
    • Column#1: HW_2G_Result_Time
    • Column#2: HW_2G_BSC_Name (this columns is partition column)
  • Converted all columns data types to string, so now source and target columns datatypes both are string.
  • SelectHiveQL processor has this query: select HW_2G_Result_Time, HW_2G_BSC_Name from ext_UFM_Intra_Cell_Handover_Measurement_Per_Cell.

Please advice.

Highlighted

Re: Permission Error: To load into Hive using NiFi

@Shu

Can it be a permission issue to create Partition directories by Nifi as loading in non-partition table is not an issue. The only issue we are facing after loading in non-partition table is, we are unable to see the data from command line where can see data in HDFS hive directory, well we will raise this issue in different thread.

As far as this thread is concern, NiFi --> PutHiveStreaming is unable to load in Hive Partitioned table.

Please advice what to check next, really appreciate your help here @Shu.

Highlighted

Re: Permission Error: To load into Hive using NiFi

Issue Resolved.

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.

Cheers.

Don't have an account?
Coming from Hortonworks? Activate your account here