@Yuriy_but  Absolutely that's not normal. There are a couple of thinks to check the Bad disk or corruption. Force checkpoint and restart the DN to see the behavior. Ensure the NN has enough memory allocated. Remove it from the cluster and rejoin afresh after reformating. Check the NIC and connectivity. Run the balancer to equally distribute the date.     ... View more

@jlguti  Could you share your /etc/host file entry, you can scramble the hostname+domain but leave the format intact.   Can you also check the value of yarn.resourcemanager.hostname in the Advanced yarn-site if it has a correct entry for the RM     ... View more

@jlguti  There can be a barrage of reasons can you share the sit^ze of your cluster [Master & Slaves] and most important your spark-submit config. I see you have a total memory of 2GB and the logs of the container     ... View more

@AlexP  Changing the replication factor doesn't change the replication factor of existing files but only the new files that will be created after issuing the "hdfs dfs -setrep" command You will have to manually change the replication factor of the old files To bulk change the replication factor   $ hdfs dfs -setrep -R -w 2 /apps/   Changing the replication factor of a single file   $ hdfs dfs –setrep –w 3 /apps/testfile.txt   Reducing the replication factor also speeds the write performance as you are writing to fewer DataNodes and also reduces Namenode metadata but causes overhead in the reads as it's more difficult to find a node that has a replica. How to estimate how much time would this command take for a single directory (without -w)? This will depend on the size of the data and your cluster processing power   Will it trigger a replication job even if I don't use the '-w' flag? Once you change the replication factor the internal data block reporting mechanism will kick in to update the Namenode of the replicas and the excess replica sitting on the same data node will be marked as over -replicated and good fro deletion     If yes, does it mean that the NameNode will actually start deleting 'over-replicated' blocks of all existing files under a particular directory? After reducing the replication factor the data blocks will become over-replicated, the namenode will detect that using the metadata and chooses a replica to remove. The name node will remove a replica from the data node with the least amount of available disk space hence helping to rebalance the load over the cluster. HDFS fsck is used to check the health of the file system, to find missing files, over replicated, under replicated and corrupted blocks  run the below To get corrupt or under replicated files   $ hdfs dfs fsck / $ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files   Delete Under-replicated blocks in HDFS   # To turbo charge use xargs -n 500 (or --max-args 500)   $ cat /tmp/under_replicated_files |xargs -n 500 hdfs dfs -setrep 1 /tmp/under_replicated_files   You can also put the above commands in a crontab ... View more

@banshidhar_saho    Surely that was I typo error I am happy that it worked for you. If you feel that helped you solve the issue you have, it would be great to accept it as an answer so the thread can be marked a solution for other members to reference.   Happy hadooping ... View more

@Masood  Can you give some context? Is it an MR, Spark job ?? Do you have some logs to share. Can you share the application Logs? ... View more

@banshidhar_saho  If you want to restrict the [others] you will need to apply POSIX permission on the file/directory look at the below examples   Below gives read and execute to [others]   $ hdfs dfs -setfacl -m others::r-x /path/to/dir   Below gives read to [others]   $ hdfs dfs -setfacl -m others::r-- /path/to/dir   Below gives no permissions to [others]   $ hdfs dfs -setfacl -m others::--- /path/to/dir   After the changes to validate run this snippet that should show it's set and to prove as [others]   $ hdfs dfs -getfacl /path/to/dir     try to list files in the directory  Happy hadooping ... View more

@NCBank  Can you start a new thread and tag me. Please include your logs or error message. The Thread you are updating is OLD. ... View more

@Masood  You could use the CDP REST API http://<cmhost>:<port>/api/v1/clusters/<cluster-name>/services/HDFS/roles/HDFS-NAMENODE-<namenode id> And look out for  and search for "haStatus"   ... View more

@varun_rathinam    Is your nifi cluster kerberized if so you will need to provide the keytabs ... View more

@sgovi    Is there an entry in your local host's file? ... View more

@donno  I know how frustrating it is. I have just downloaded a fresh image of HDP 2.6.5  unpacked the image and performed the classic steps all all is looks fine I hope you a step that you might have missed.  Here we go. Sandbox settings, Memory, CPU, Network I enabled on Bridged Adapter as my laptop is connected to my LAN a class C network   Uncompressed the HDP  2.6.5 image and on successful extraction, I get the IP and URL with my local IP     My Local hosts file   Reset the root  default is root/hadoop and Ambari admin passwords this automatically starts the Ambari server After the successful start-up of Ambari I can now access the Ambari UI using my local LAN IP All service is up  and you can see the Ambari and HDP versions I have  demo'ed this 100's of time in here I hope you get successful and happy ending  Happy hadooping ... View more

@sriram72  Can you share screenshots I just posted a response to a similar query? Please have a look at this Sandbox  issue    Hope that helps ... View more

@ParthiCyberPunk  Unfortunately, you didn't share the connect string. below is an example you could use jdbc:hive2://host:10000/DB_name;ssl=true;sslTrustStore=$JAVA_HOME/jre/lib/security/certs_name;trustStorePassword=$password Substitute  host,port, truststore location and certificate name and password accordingly. Keep me posted     ... View more

@anhthu  You will need to fire-up your cluster  start by scrolling at the bottom  see the attached screen and start the Cloudera Manager [CM]  the blue triangular shape will give you a drop-down menu chose start you will see some startup logs and if all goes well it will be green   You can see exactly the same error I have on my Quickstart sandbox because my service are not started  again see the attached screenshot on the blue inverted triangle you will see a drop-down list chose start the services. This will start all the service in the right order ,again once all is GREEN you are good to go   Happy Hadooping ... View more

@akoudoum  Great it worked for you  Happy hadooping ... View more

@sgovi  Can you confirm you have only one network card enabled? Please share the output of the below command from the web-shell CLI $ ifconfig       Please revert    ... View more

@cbfr  Before you decide whether the cluster has corrupt files can you check the replication factor? If it's set to 2 then that normal  The help option will give you a full list of sub commands $ hdfs fsck / ? fsck: can only operate on one path at a time '?' The list of sub command options $ fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks | -replicaDetails | -upgradedomains]]]] [-includeSnapshots] [-showprogress] [-storagepolicies] [-maintenance] [-blockId <blk_Id>] start checking from this path -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data of a snapshottable directory -list-corruptfileblocks print out list of missing blocks and files they belong to -files -blocks print out block report -files -blocks -locations print out locations for every block -files -blocks -racks print out network topology for data-node locations -files -blocks -replicaDetails print out each replica details -files -blocks -upgradedomains print out upgrade domains for every block -storagepolicies print out storage policy summary for the blocks -maintenance print out maintenance state node details -showprogress show progress in output. Default is OFF (no progress) -blockId print out which file this blockId belongs to, locations (nodes, racks)   It would be good to first check for corrupt files and then run the delete    $ hdfs fsck / -list-corruptfileblocks Connecting to namenode via http://mackenzie.test.com:50070/fsck?ugi=hdfs&listcorruptfileblocks=1&path=%2F ---output-- The filesystem under path '/' has 0 CORRUPT files A simple demo here my replication factor is 1  see above screenshot when I create a new file in hdfs the default repélication factor is set to 1 $ hdfs dfs -touch /user/tester.txt Now to check the replication fact  see the number 1 before the group:user hdfs:hdfs $ hdfs dfs -ls /user/tester.txt -rw-r--r-- 1 hdfs hdfs 0 2020-10-18 10:21 /user/tester.txt   Hope that helps            ... View more

@sgovi  I have just downloaded a Virtual box sandbox image and imported it into the VirtualBox successfully. In my configuration, I  enabled only one network card Bridge Adapter so it picks the IP from my LAN of 192.168.0.x   After uncompressing the Docker image, the initial screen shows it picked my local LAN IP which I used to access the browser CLI as shown below.  Make sure you update your  windows hosts file in C:\Windows\System32\drivers\etc  Using the above  URL  change the initial  root and Ambari passwords see steps I completed the below steps changing the initial root password Hadoop and then reset the Ambai user password. Once that is successful is starts the Ambari server    sandbox-hdp login: root root@sandbox-hdp.hortonworks.com's password: You are required to change your password immediately (root enforced) Last login: Sat Oct 17 20:21:47 2020 Changing password for root. (current) UNIX password: New password:   Ambari user password reset steps   ambari-admin-password-reset Please set the password for admin: Please retype the password for admin: The admin password has been set. Restarting ambari-server to make the password change effective... Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................................ Server started listening on 8080 DB configs consistency check: no errors and warnings were found.   Using the Local IP given to the VirtualBox from my LAN,I could access Ambari with the new password I reset above and restarted all serive though some were running. Can you confirm you followed those steps and still failed?? Happy hadooping ... View more

@nilesh_shrimant  If you want to get help you should share more detail like datatype and  the create external table snippet.   But I think this  post should your answer ... View more

@bhoken  In a kerberized cluster the Kafka ACL is leveraged by Ranger,if the Kafka plugin is enabled don't look further than the Ranger  Please share you Ranger-Kafka policy  ... View more

@kumarkeshav  The parameter you are looking for hbase.regionserver.global.memstore.size is found in the  hbase-site.xml   I don't know where this value can be separately edit or centrally by Ambari ... View more