Hi @Sebastien F, Couple of things needs to be considered to achieve this, as the optimizer will not be knowing till the execution of what are the values from "currentpartitiontables" (t2) is going to produced to generate the optimal explain plan. Thus, we need to make sure that Cost base optimizer (CBO) is aware of the data profile to ensure optimal explain plan, On the same context we know that t2 is very small, hence the optimizer should opt for map side join using the t2. MAPJOINs are processed by loading the smaller table into an in-memory hash map and matching keys with the larger table as they are streamed through, we can provide the hints to hive so that it will be fully aware of the data demographics, two ways we can do this With Hints select /*+ MAPJOIN(t2) */ count(*) from mypartitionedtable t1 INNER JOIN currentpartitiontable t2 on t2 ON t1.YEAR=t2.YEAR etc. etc. With the Auto Join Conversion set hive.auto.convert.join=true; //When auto join is enabled, there is no longer a need to provide the map-join hints in the query. The auto join option can be enabled with two configuration parameters: set hive.auto.convert.join.noconditionaltask = true; set hive.auto.convert.join.noconditionaltask.size = 10000000; Ensure that statistics are updated to so that optimizer will across the data demographics. the other alternative is to pass the values pragmatically(much easier in some cases to achieve), If the values are fewer ( year,month and day ) the simplistic approach is to pragmatically retrive the values and pass as variables in user program ( might not be possible in all the cases ex: BI Utilities) more on the join optimization can be found in hive Language manual Hope this helps !! ... View more

Hi @Karan Alang, For an external Partitioned table, we need to update the partition metadata as the hive will not be aware of these partitions unless the explicitly updated that can be done by either ALTER TABLE power_k1 RECOVER PARTITIONS; //or MSCK REPAIR TABLE power_k1; more on this can be found from hive DDL Language Manual. Hope this helps !! ... View more

Hi @Mayan Nath, Can you please ensure to set "Strict Host Key Checking to False " and also remove the known_hosts entries for the target host (under the directory ~/.ssh/knon_hosts or nifi user) the best thing is to do the sftp manually from the command line and test once thats working incorporate to the NiFi cluster.( please ensure that you have performed the same across all he hosts) Hope this helps !! ... View more

Hi @Erkan ŞİRİN, From the error, I could see that, it was the access issue that server2 is not allowing user from Server1 ( where you initiated the sqoop connection - user@server1), for mysql. the solution would be provide appropriate mysql access for the user which is initiating the connection from Server1. GRANT <PREV LIST> PRIVILEGES ON *.* TO '<user>'@'%' IDENTIFIED BY 'PASSWORD' WITH GRANT OPTION; FLUSH PRIVILEGES; % indicates from all the hosts, it is good to give the access to range of hosts as The connect string you supply will be used on TaskTracker nodes throughout your MapReduce cluster( pick one of node from your cluster) before you initiate the connection you can verify the communication ( if you have mysql "Client" installed on the server1 ) by mysql -h <Server2> -u <User1> -p <PASSWORD> Hope this helps !! ... View more

Hi @Timothy Spann, Looks thats the error form File system Unavailability, Could you please check the file system size ( not 100% full ) or if thats in NAS/SAN - ensure it was not disconnected at that time ( can be verified from /var/log/messages). ... View more