@ChineduLB Apache Impala does not enable multi-statement transactions, so you cannot perform an atomic transaction that spans many INSERT statements directly. You can achieve a similar effect by combining the INSERT INTO commands into a single INSERT INTO... SELECT statement that includes a UNION ALL. This method assures that all partitions are loaded within the same query run.   you can consolidate your insert statements into one query INSERT INTO client_view_tbl PARTITION (cobdate, region) SELECT col, col2, col3, '20240915' AS cobdate, 'region1' AS region FROM region1_table WHERE cobdate = '20240915' UNION ALL SELECT col, col2, col3, '20240915' AS cobdate, 'region2' AS region FROM region2_table WHERE cobdate = '20240915' UNION ALL SELECT col, col2, col3, '20240915' AS cobdate, 'region3' AS region FROM region3_table WHERE cobdate = '20240915'; Single Query Execution: This approach consolidates multiple INSERT statements into one, which can improve performance and ensure consistency within the query execution context. Simplified Management: Managing a single query is easier than handling multiple INSERT statements. Ensure that your source tables (region1_table, region2_table, region3_table) and the client_view_tbl table have compatible schemas, especially regarding the columns being selected and inserted. Be mindful of the performance implications when dealing with large datasets. Test the combined query to ensure it performs well under your data volume. By using this combined INSERT INTO ... SELECT ... UNION ALL approach, you can effectively populate multiple partitions of the client_view_tbl table in one query. "please accept it as a solution if it it helps"   ... View more

@SAMSAL  Navigate to the location of your NiFi installation. Rename the folder to remove any spaces. mv "NIFI 2.0.0M2" NIFI_2.0.0M2 Open your system environment variables settings. Update the NIFI_HOME environment variable to the new path if it's set. Ensure that the JAVA_HOME variable is correctly set and points to your Java installation directory. Open the run-nifi.bat script to ensure it correctly references the new path. Look for any hardcoded paths that may still contain spaces and update them Execute the run-nifi.bat file again to start NiFi. It should now correctly locate the org.apache.nifi.bootstrap.RunNiFi class and proceed without the previous errors. Check JAVA_HOME: Make sure that the JAVA_HOME environment variable is correct.Ensure it points to a valid Java installation path echo %JAVA_HOME% ... View more

@jpconver2  Challenges: NiFi Version: While recent versions (1.10.0+) offer improved cluster management, rolling updates can still be challenging if your custom processors introduce flow configuration changes. Nodes with the old processors won't recognize components from the updated NAR, preventing them from joining the cluster until all nodes are in sync. Flow Compatibility: NiFi requires consistent flow definitions (flow.xml.gz) across all nodes. Updates that alter the flow can disrupt cluster operations during rolling updates. Solutions: Scenario a: Single NAR Version Backward Compatibility: Prioritize backward compatibility in your custom processors. This ensures minimal changes to the flow definition and smoother rolling updates. Full Cluster Upgrade: If backward compatibility isn't feasible, consider a full cluster upgrade to the new NiFi version and custom processor NAR. Scenario b: Multiple NAR Versions Manual Version Management: Update processors manually through the NiFi UI or API after deploying the new NARs. This offers control but requires intervention. Custom Automation Scripts: Develop scripts leveraging NiFi's REST API to automate processor version updates. These scripts can: Identify custom processor instances. Update each processor to the latest available version. Update controller services and restart affected processors. Custom NiFi Extensions: Implement custom logic to handle version upgrades. This could involve creating a Reporting Task or Controller Service that checks for new versions and updates processors automatically.   Recommendations: Upgrade NiFi Version: If possible, upgrade to NiFi 1.10.0 or later for improved rolling update support. Scripting for Automation: Explore scripting with the NiFi REST API to automate processor version updates, especially if you manage multiple NAR versions. Remember: Stay updated with the latest NiFi releases to benefit from improvements and features. Carefully evaluate your specific needs and choose the approach that balances downtime and manageability.     Please accept it as a solution if it it helps ... View more

@galt    Altering the ID of a connection in Apache NiFi isn't directly endorsed or recommended because the ID serves as a unique identifier used internally by NiFi to manage its components. However, if you absolutely must change the ID for a specific reason, you could employ a workaround, though it's not advisable due to potential risks and complications. Here's a basic approach you could consider: Backup: Before making any alterations, make sure to create a backup of your NiFi flow. This step is crucial in case something goes awry and you need to revert to the previous state. Export and Modify Flow Configuration: Export the NiFi flow configuration, typically in XML format. This can be done via the NiFi UI or by utilizing NiFi's REST API. Then, manually adjust the XML to change the ID of the connection to the desired value. Stop NiFi: Halt the NiFi instance to prevent conflicts or corruption while modifying the configuration files. Replace Configuration: Substitute the existing flow configuration file with the modified one. Restart NiFi: Restart NiFi and confirm that the changes have been implemented. Keep in mind the following considerations: Risks: Altering the ID directly in the configuration files could result in unexpected behavior or even corruption of your flow. Proceed with caution and ensure you have a backup. Dependency: If any processors or components rely on this connection ID within NiFi, they may break or exhibit unexpected behavior after the change. Unsupported: This method isn't officially supported by Apache NiFi, and there's no guarantee that it will function seamlessly or without issues. ... View more

@ChineduLB  WITH data_counts AS ( SELECT COUNT(*) AS count_table1, COUNT(*) AS count_table2, COUNT(*) AS count_table3, COUNT(*) AS count_table4, COUNT(*) AS count_table5, COUNT(*) AS count_table6 FROM table1 WHERE date_partition = 'your_date' -- Replace 'your_date' with the specific date you're interested in UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table2 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table3 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table4 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table5 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table6 WHERE date_partition = 'your_date' ) SELECT CASE WHEN SUM(count_table1) > 0 AND SUM(count_table2) > 0 AND SUM(count_table3) > 0 AND SUM(count_table4) > 0 AND SUM(count_table5) > 0 AND SUM(count_table6) > 0 THEN (SELECT * FROM table1 WHERE date_partition = 'your_date') ELSE NULL -- or whatever you want to return if data doesn't exist in all tables END AS result FROM data_counts; ... View more

@ChineduLB  Impala doesn't directly support nested select statements within the WHEN clause of a CASE statement. However, you can achieve similar logic Subqueries for conditions: You can use subqueries within the WHEN clause to evaluate conditions based on data retrieved from other tables. SELECT case when (select count(*) from table1) > 0 then (select * from table1) when (select count(*) from table2) > 0 and (select count(*) from table3) > 0 then (select * from table3) else null end as result_table; This query checks if table1 has any rows. If yes, it selects all columns from table1. Otherwise, it checks if both table2 and table3 have rows. If both have data, it selects all columns from table3. If none of the conditions are met, it returns null.   ... View more

@snm1523 check if this doc can help   https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cdh_upgrade_cdp2cdp_post.html ... View more