@flashone  If a disk error is detected, HDFS can mark the affected disk as failed and stop using it. HDFS DataNodes are designed to handle disk failures gracefully. If you have replication set up correctly, the data should remain accessible, though replication might temporarily increase on other nodes to compensate for the loss. The service itself (HDFS) will usually stay operational as long as there are other healthy disks and nodes available ++++++++++++++ YARN NodeManagers can handle disk failures by marking disks as unhealthy if configured to monitor disk health, When a disk fails, NodeManager excludes that disk from the list of usable directories. The NodeManager service itself will continue running as long as other disks are healthy. ++++++++++++++++ If Impala detects a disk I/O error, it will stop using that disk. The Impala Daemon will continue running, but queries that rely on data stored on the failed disk might fail until data can be accessed from another replica or node. +++++++++++++ Kudu Tablet Servers monitor disk health, and if a disk fails, Kudu can mark it as failed and continue operating if there are other healthy disks. However, if the failure impacts multiple disks or replicas, it can lead to data availability issues. ++++++++++++ You can usually keep the services running if only a single disk fails and if replication is properly configured. However, it’s best to replace the failed disk promptly to avoid further risk. In HDFS and Kudu especially, losing additional disks could risk data loss or availability issues. ... View more

@drewski7  Ensure that the Kerberos tickets are being refreshed properly for the HBase REST server. Stale or expired tickets might cause intermittent authorization issues. Check the Kerberos cache to ensure that it is being updated correctly when group memberships change in LDAP. Restart the HBase REST server after making changes to the LDAP group and running the user sync to see if it resolves the inconsistency. Analyze the HBase REST server logs more thoroughly, especially the messages related to unauthorized access and Kerberos thread issues. Look for patterns or specific errors that could provide more clues. Verify the settings for ranger.plugin.hbase.policy.pollIntervalMs and ranger.plugin.hbase.authorization.cache.max.size again, and experiment with lowering the poll interval to see if it improves the responsiveness of policy changes. In the Ranger Admin UI, after running the user sync, manually refresh the policies for HBase and observe if this action has any immediate effect on the authorization behavior. Confirm that there are no discrepancies in the policies displayed in the Ranger Admin UI and the actual enforcement in HBase. Double-check the synchronization between FreeIPA LDAP and Ranger. Ensure that the user sync is not just updating the Ranger Admin UI but is also effectively communicating changes to all Ranger plugins. Review the user sync logs to verify that all changes are processed correctly without errors. ... View more

@ChineduLB Apache Impala does not enable multi-statement transactions, so you cannot perform an atomic transaction that spans many INSERT statements directly. You can achieve a similar effect by combining the INSERT INTO commands into a single INSERT INTO... SELECT statement that includes a UNION ALL. This method assures that all partitions are loaded within the same query run.   you can consolidate your insert statements into one query INSERT INTO client_view_tbl PARTITION (cobdate, region) SELECT col, col2, col3, '20240915' AS cobdate, 'region1' AS region FROM region1_table WHERE cobdate = '20240915' UNION ALL SELECT col, col2, col3, '20240915' AS cobdate, 'region2' AS region FROM region2_table WHERE cobdate = '20240915' UNION ALL SELECT col, col2, col3, '20240915' AS cobdate, 'region3' AS region FROM region3_table WHERE cobdate = '20240915'; Single Query Execution: This approach consolidates multiple INSERT statements into one, which can improve performance and ensure consistency within the query execution context. Simplified Management: Managing a single query is easier than handling multiple INSERT statements. Ensure that your source tables (region1_table, region2_table, region3_table) and the client_view_tbl table have compatible schemas, especially regarding the columns being selected and inserted. Be mindful of the performance implications when dealing with large datasets. Test the combined query to ensure it performs well under your data volume. By using this combined INSERT INTO ... SELECT ... UNION ALL approach, you can effectively populate multiple partitions of the client_view_tbl table in one query. "please accept it as a solution if it it helps"   ... View more

@SAMSAL  Navigate to the location of your NiFi installation. Rename the folder to remove any spaces. mv "NIFI 2.0.0M2" NIFI_2.0.0M2 Open your system environment variables settings. Update the NIFI_HOME environment variable to the new path if it's set. Ensure that the JAVA_HOME variable is correctly set and points to your Java installation directory. Open the run-nifi.bat script to ensure it correctly references the new path. Look for any hardcoded paths that may still contain spaces and update them Execute the run-nifi.bat file again to start NiFi. It should now correctly locate the org.apache.nifi.bootstrap.RunNiFi class and proceed without the previous errors. Check JAVA_HOME: Make sure that the JAVA_HOME environment variable is correct.Ensure it points to a valid Java installation path echo %JAVA_HOME% ... View more

@jpconver2  Challenges: NiFi Version: While recent versions (1.10.0+) offer improved cluster management, rolling updates can still be challenging if your custom processors introduce flow configuration changes. Nodes with the old processors won't recognize components from the updated NAR, preventing them from joining the cluster until all nodes are in sync. Flow Compatibility: NiFi requires consistent flow definitions (flow.xml.gz) across all nodes. Updates that alter the flow can disrupt cluster operations during rolling updates. Solutions: Scenario a: Single NAR Version Backward Compatibility: Prioritize backward compatibility in your custom processors. This ensures minimal changes to the flow definition and smoother rolling updates. Full Cluster Upgrade: If backward compatibility isn't feasible, consider a full cluster upgrade to the new NiFi version and custom processor NAR. Scenario b: Multiple NAR Versions Manual Version Management: Update processors manually through the NiFi UI or API after deploying the new NARs. This offers control but requires intervention. Custom Automation Scripts: Develop scripts leveraging NiFi's REST API to automate processor version updates. These scripts can: Identify custom processor instances. Update each processor to the latest available version. Update controller services and restart affected processors. Custom NiFi Extensions: Implement custom logic to handle version upgrades. This could involve creating a Reporting Task or Controller Service that checks for new versions and updates processors automatically.   Recommendations: Upgrade NiFi Version: If possible, upgrade to NiFi 1.10.0 or later for improved rolling update support. Scripting for Automation: Explore scripting with the NiFi REST API to automate processor version updates, especially if you manage multiple NAR versions. Remember: Stay updated with the latest NiFi releases to benefit from improvements and features. Carefully evaluate your specific needs and choose the approach that balances downtime and manageability.     Please accept it as a solution if it it helps ... View more

@galt    Altering the ID of a connection in Apache NiFi isn't directly endorsed or recommended because the ID serves as a unique identifier used internally by NiFi to manage its components. However, if you absolutely must change the ID for a specific reason, you could employ a workaround, though it's not advisable due to potential risks and complications. Here's a basic approach you could consider: Backup: Before making any alterations, make sure to create a backup of your NiFi flow. This step is crucial in case something goes awry and you need to revert to the previous state. Export and Modify Flow Configuration: Export the NiFi flow configuration, typically in XML format. This can be done via the NiFi UI or by utilizing NiFi's REST API. Then, manually adjust the XML to change the ID of the connection to the desired value. Stop NiFi: Halt the NiFi instance to prevent conflicts or corruption while modifying the configuration files. Replace Configuration: Substitute the existing flow configuration file with the modified one. Restart NiFi: Restart NiFi and confirm that the changes have been implemented. Keep in mind the following considerations: Risks: Altering the ID directly in the configuration files could result in unexpected behavior or even corruption of your flow. Proceed with caution and ensure you have a backup. Dependency: If any processors or components rely on this connection ID within NiFi, they may break or exhibit unexpected behavior after the change. Unsupported: This method isn't officially supported by Apache NiFi, and there's no guarantee that it will function seamlessly or without issues. ... View more

@ChineduLB  WITH data_counts AS ( SELECT COUNT(*) AS count_table1, COUNT(*) AS count_table2, COUNT(*) AS count_table3, COUNT(*) AS count_table4, COUNT(*) AS count_table5, COUNT(*) AS count_table6 FROM table1 WHERE date_partition = 'your_date' -- Replace 'your_date' with the specific date you're interested in UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table2 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table3 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table4 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table5 WHERE date_partition = 'your_date' UNION ALL SELECT COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*), COUNT(*) FROM table6 WHERE date_partition = 'your_date' ) SELECT CASE WHEN SUM(count_table1) > 0 AND SUM(count_table2) > 0 AND SUM(count_table3) > 0 AND SUM(count_table4) > 0 AND SUM(count_table5) > 0 AND SUM(count_table6) > 0 THEN (SELECT * FROM table1 WHERE date_partition = 'your_date') ELSE NULL -- or whatever you want to return if data doesn't exist in all tables END AS result FROM data_counts; ... View more