@cbfr  Before you decide whether the cluster has corrupt files can you check the replication factor? If it's set to 2 then that normal  The help option will give you a full list of sub commands $ hdfs fsck / ? fsck: can only operate on one path at a time '?' The list of sub command options $ fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks | -replicaDetails | -upgradedomains]]]] [-includeSnapshots] [-showprogress] [-storagepolicies] [-maintenance] [-blockId <blk_Id>] start checking from this path -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data of a snapshottable directory -list-corruptfileblocks print out list of missing blocks and files they belong to -files -blocks print out block report -files -blocks -locations print out locations for every block -files -blocks -racks print out network topology for data-node locations -files -blocks -replicaDetails print out each replica details -files -blocks -upgradedomains print out upgrade domains for every block -storagepolicies print out storage policy summary for the blocks -maintenance print out maintenance state node details -showprogress show progress in output. Default is OFF (no progress) -blockId print out which file this blockId belongs to, locations (nodes, racks)   It would be good to first check for corrupt files and then run the delete    $ hdfs fsck / -list-corruptfileblocks Connecting to namenode via http://mackenzie.test.com:50070/fsck?ugi=hdfs&listcorruptfileblocks=1&path=%2F ---output-- The filesystem under path '/' has 0 CORRUPT files A simple demo here my replication factor is 1  see above screenshot when I create a new file in hdfs the default repélication factor is set to 1 $ hdfs dfs -touch /user/tester.txt Now to check the replication fact  see the number 1 before the group:user hdfs:hdfs $ hdfs dfs -ls /user/tester.txt -rw-r--r-- 1 hdfs hdfs 0 2020-10-18 10:21 /user/tester.txt   Hope that helps            ... View more

@sgovi  I have just downloaded a Virtual box sandbox image and imported it into the VirtualBox successfully. In my configuration, I  enabled only one network card Bridge Adapter so it picks the IP from my LAN of 192.168.0.x   After uncompressing the Docker image, the initial screen shows it picked my local LAN IP which I used to access the browser CLI as shown below.  Make sure you update your  windows hosts file in C:\Windows\System32\drivers\etc  Using the above  URL  change the initial  root and Ambari passwords see steps I completed the below steps changing the initial root password Hadoop and then reset the Ambai user password. Once that is successful is starts the Ambari server    sandbox-hdp login: root root@sandbox-hdp.hortonworks.com's password: You are required to change your password immediately (root enforced) Last login: Sat Oct 17 20:21:47 2020 Changing password for root. (current) UNIX password: New password:   Ambari user password reset steps   ambari-admin-password-reset Please set the password for admin: Please retype the password for admin: The admin password has been set. Restarting ambari-server to make the password change effective... Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................................ Server started listening on 8080 DB configs consistency check: no errors and warnings were found.   Using the Local IP given to the VirtualBox from my LAN,I could access Ambari with the new password I reset above and restarted all serive though some were running. Can you confirm you followed those steps and still failed?? Happy hadooping ... View more

@nilesh_shrimant  If you want to get help you should share more detail like datatype and  the create external table snippet.   But I think this  post should your answer ... View more

@bhoken  In a kerberized cluster the Kafka ACL is leveraged by Ranger,if the Kafka plugin is enabled don't look further than the Ranger  Please share you Ranger-Kafka policy  ... View more

@kumarkeshav  The parameter you are looking for hbase.regionserver.global.memstore.size is found in the  hbase-site.xml   I don't know where this value can be separately edit or centrally by Ambari ... View more

@mike_bronson7  Once you connect the 10 new data nodes to the cluster Ambari automatically distributes the common hadoop config file i.e hdfs-site.xml,Mapred-site.xml,yarn-site.xml etc to those new nodes so they can start receiving data blocks.   My suggestion as a workaround would be to add these 10 new datanodes hostnames FQDN or IP (separated by a newline character) in the dfs.exclude file on the NameNode machine, edit the <HADOOP_CONF_DIR>/dfs.exclude file and where <HADOOP_CONF_DIR> is the directory for storing the Hadoop configuration files. For example, /etc/hadoop/conf.   First, ensure the DNS resolution is working or your /etc/hosts are updated and the passwordless connection is working with those hosts. Once the 10 new nodes are in the dfs.exclude file the namenode will consider them as bad nodes so no data will be replicated to them as long as these hosts remain in the dfs.exclude file once you have updated the NameNode with the new set of excluded DataNodes.   On the NameNode host machine, execute the following command:   su <HDFS_USER> hdfs dfsadmin -refreshNodes   where <HDFS_USER> is the user owning the HDFS services   That should do the trick, once these hosts are visible in Ambari turn maintenance mode on so you don't receive any alerts The day you will decide to add/enable these 10 new datanodes you will simply cp or mv the dfs.exclude to dfs.include file located <HADOOP_CONF_DIR>/dfs.include these nodes will start heartbeating and notifying the NameNode that  thes DataNodes are ready to start receiving files and participating in the data distribution in the cluster.   On the NameNode host machine remember to execute the following command:   su <HDFS_USER> hdfs dfsadmin -refreshNodes     Don't forget to disable Maintenance mode on the new datanodes and remove them from  dfs.exclude file if you didn't rename or delete it.   Run the HDFS Balancer a tool for balancing the data across the storage devices of an HDFS cluster.   sudo -u hdfs hdfs balancer   The above balancer command has a couple of options either threshold or again the dfs.include and dfs.exclude see explanation below                                                 Include and Exclude Lists   When the include list is non-empty, only the datanodes specified in the list are balanced by the HDFS Balancer. An empty include list means including all the datanodes in the cluster. The default value is an empty list.   [-include [-f <hosts-file> | <comma-separated list of hosts>]]   The datanodes specified in the exclude list are excluded so that the HDFS Balancer does not balance those datanodes. An empty exclude list means that no datanodes are excluded. When a datanode is specified in both in the include list and the exclude list, the datanode is excluded. The default value is an empty list.   [-exclude [-f <hosts-file> | <comma-separated list of hosts>]]     If no df s.include file is specified, all DataNodes are considered to be included in the cluster (unless excluded explicitly in the dfs.exclude file). The dfs.hosts and dfs.hosts.exclude properties in hdfs-site.xml are used to specify the dfs.include and dfs.exclude files. Hope that helps  ... View more

@lxs    I have helped resolve this kind of issue a couple of times. Can you help with screenshots of your configuration of the sandbox? Memory Network  Splash screen After restarting the sandbox and any other screenshot you deem important ... View more

@vinod_artga    The first step is to check the Cloudera upgrade path using the My environment matrix calculator See screenshot below     After filling in all the information request this generates for you a report and warnings like Warning Upgrades from CDH 5.12 and lower to  CDP Private Cloud Base  are not supported. You must upgrade the cluster to CDH versions 5.13 - 5.16 before upgrading to  CDP Private Cloud Base . Warning For upgrades from CDH 5 clusters with Sentry to Cloudera Runtime 7.1.1 (or higher) clusters where Sentry privileges are to be transitioned to Apache Ranger, the cluster must have Kerberos enabled before upgrading It also gives you comprehensive details about the best approach and component incompatibilities, this is your source of true I would suggest you try it and revert HTH ... View more

@pazufst                                                                  How Ranger policies work for HDFS Apache Ranger offers a federated authorization model for HDFS. Ranger plugin for HDFS checks for Ranger policies and if a policy exists, access is granted to user. If a policy doesn’t exist in Ranger, then Ranger would default to the native permissions model in HDFS (POSIX or HDFS ACL). This federated model is applicable for HDFS and Yarn service in Ranger. For other services such as Hive or HBase, Ranger operates as the sole authorizer which means only Ranger policies are in effect. The option for the fallback model is configured using a property in Ambari → Ranger → HDFS config → Advanced ranger-hdfs-security   xasecure.add-hadoop-authorization=true   The federated authorization model enables to safely implement Ranger in an existing cluster without affecting jobs that rely on POSIX permissions to enable this option as the default model for all deployments. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=XXXXX, access=READ, inode="/user/.snapshot/user_201806150000":w93651:hdfs:drwx------ Is self-explanatory does the user w93651 exist on both clusters with valid Kerberos tickets if the cluster is kerberized? Ensure the CROSS-REALM is configured and working.   Is your ranger managing the 2 clusters? HTH ... View more

@pazufst    You should toggle to RECURSIVE for the RWX permission on the hdfs should look like this  /databank/* The selection should look like      Hope that helps   ... View more

@pazufst  Doesn't it look strange that permissions are    /databank/.snapshot/databank_201904250000 :hdfs:hdfs:d---------   The normal permissions show be    rwx   Try that and revert  ... View more

@kvinod    If you have installed Yarn and MRV2 can you check the value of the below parameter in the yarn-site.xml  yarn.nodemanager.aux-services stop the services and change it to look like below <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> Restart and let me know ... View more

@GangWar  Can you regenerate the keytabs through  Cloudare manager? That could resolve  the problem if it doean't please revert with the error cêncountered? ... View more

@wert_1311  Domain name changes will affect the KDC database. Kerberos is super sensitive to domain changes according to experience you will have to recreate the KDC database and regenerate the keytabs/principals to enable you applications to reconnect. Cluster hostname If the hosts in the cluster were re-named ie host1.old.com to host1.new.com then ensure those changes are also reflected or resolved by the DNS.   This is going a tricky one but fortunately, CM or Ambari will make your work easy now that your domain has changed the earlier generated keytabs have the old domain name . A keytab contains a pair of principals and an encrypted copy of that principal's key it's unique to each host since the principal names include the hostname and may be concatenated with the domain name Delete the old KDC database Usually, as the root user call the Kerberos database utility kdb5_util destroy assuming the old domain was OLD.COM this should delete the keytabs and principals linked to the old REALM,   # kdb5_util -r OLD.COM destroy You will need to manually delete the keytabs liked to the old REALM on the file system /etc/security/keytabs/ [HDP] or /etc/hadoop/conf/[CDH]. You will be prompted to confirm before destroying the database, usually, this is a better option if you have second thought rather than using the kdb5_util destroy -f will naturally not prompt you for a confirmation   Recreate the New KDC database Use the Kerberos database utility kdb5_util create [-s] assuming the new domain was NEW.COM # kdb5_util NEW.COM create # kdb5_util -r NEW.COM create -s With the -s option, kdb5_util will stash a copy of the master key in a stash file this allows a KDC to authenticate itself to the database utilities, such as kadmin, kadmind, krb5kdc, and kdb5_util best option.   Update Kerberos files. Make sure you update the below files to reflect the new REALM assuming your MIT KDC server's domain isn't changed. krb5.conf kdc.conf kadm5.acl Auth-to-local Rules jaas.conf files [if being used by applications] Enable Kerberos Using CM or Ambari the process is straight forward.   Please let me know if you need more help   ... View more

@anass  Hive and Impala have 2 distinct use-cases Hive, a data warehouse system is used for analyzing structured data. Uses HQL or the Hive Query Language which gets internally converted to MapReduce jobs which is fault-tolerant and a very good candidate for ETLs and batch-processing. On the other hand, Impala executes faster using an engine designed especially for the mission of interactive SQL over HDFS although unlike Hive, Impala is not fault-tolerance. But a fantastic MPP (Massive Parallel Processing) engine.   - Hive generates query expressions at compile time whereas Impala does runtime code generation for “big loops” with no need for data movement and data transformation for storing data on Hadoop, . - Impala no java knowledge is required programmatically accessing the data in HDFS or HBase a basic knowledge of SQL querying can do the work. - Impala performs best when it queries files stored as Parquet format. It's good for sampling data - Apache Hive is not ideal for interactive computing query whereas Impala is meant for interactive computing. - Hive is batch-based Hadoop MapReduce whereas Impala is more like MPP database. - Hive supports complex types but Impala does not. - Apache Hive is fault-tolerant whereas Impala does not support fault tolerance. So its the best candidate for batch processing which is prone to failures When a hive query is run and if the DataNode goes down while the query is being executed, the output of the query will be produced as Hive is fault-tolerant. However, that is not the case with Impala. If a query execution fails in Impala it has to be started all over again. - Hive can transform SQL queries into Spark or MR jobs making it a good choice for long-running ETL jobs for which it is desirable to have fault tolerance because developers do not want to re-run a long-running job after executing it for several hours.   For better comparison here is the benchmark HAWQ,Hive and Impala Hope that helps ... View more

@BBFayz  Below is the link with all the passwords you could be interested in. The default root password by default is root/hadoop but you will be requested to change that on the first logon   Sandbox passwords  learning rope Setup Static IP on RHEL Hope that helps ... View more

@kvinod  Can you share the steps you have accomplished and attach the specific errors that you are encountering? ... View more

@amateur  There is a Jira out there  ... View more

@Seaport  I would think there is a typo error  the dash  [-] after - put  and before the hdfs path  hdfs dfs -cat /user/testuser/stage1.tar.gz | gzip -d | hdfs dfs -put - /user/testuser/test3   try this after removing the dash -   hdfs dfs -cat /user/testuser/stage1.tar.gz | gzip -d | hdfs dfs -put /user/testuser/test3  Hope that helps  ... View more

@Seaport  It shouldn't be strange to you that Hadoop doesn't perform well with small files, now with that in mind the best solution would be to zip all your small files locally and then copy the zipped file to hdfs using copyFromLocal there is one restriction that is the source of the files can only be on  a local file system. I assume the local Linux box had is the edge node and had the hdfs client installed. If not you will have to copy the myzipped.gz to a node usually the edge node and perform the below steps $ hdfs dfs -copyFromLocal myzipped.gz /hadoop_path". Then unzip the myzipped.gz gzipped file in HDFS using $ hdfs dfs -cat /hadoop_path/myzipped.gz | gzip -d | hdfs dfs -put - /hadoop_path2 Hope that helps ... View more

@Stephbat  Bizarre all the symlinks should point to the newer version 3.1.4.0-315. You should recreate the symlink point to the new version. The re-run the steps   ... View more

@Stephbat  Did you follow the Workaround:1 through 3? I see you have a similar error File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py 1) go to each ambari-agent node and edit the file remove_previous_stacks.py # vi /var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py 2) go to line: 77 edit the line from : all_installed_packages = self.pkg_provider.all_installed_packages() to all_installed_packages = self.pkg_provider.installed_packages() 3)  R etry the operation via curl again, please substitute only the Ambari_host below  ex: curl 'http://Ambari_host:8080/api/v1/clusters/<cluster_name>/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"3.1.0.0-78"}}, "Requests/resource_filters": [{"hosts":"Ambari_host"}]}'  And please revert    ... View more

@Stephbat    This thread should help you achieve that the Ambari version could be an issue there is a workaround documented note the DISCLAIMER !! Please let me know ... View more

@Krpyto84  Your permission issue is linked to ZK ACL's my good guess is your Kafka is kerberized. Zookeeper requires you to set up a superuser using the zookeeper.DigestAuthenticationProvider.superDigest property. I don't know how you will integrate that procedure in your Ansible playbook You will then need to append this in y your KAFKA_OPTS env variable to set the JVM parameters export KAFKA_OPTS=-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf Please let me know whether that is your situation if that's the case then I will try to help you out   ... View more