@rbloughThank you for the continued support.   2) The command is being run as the hdfs user.   1) The detailed output showed that there are 603,723 blocks in total. Looking at the HDFS UI, the Datanodes report having 586,426 blocks each.   3) hdfs fsck / -openforwrite says that there are 506,549 blocks in total.   The discrepancy in block count seems to be there still. Below are the summaries of the different fsck outputs.   hdfs fsck / -files -blocks -locations -includeSnapshots Status: HEALTHY Number of data-nodes: 3 Number of racks: 1 Total dirs: 64389 Total symlinks: 0 Replicated Blocks: Total size: 330079817503 B (Total open files size: 235302 B) Total files: 625308 (Files currently being written: 129) Total blocks (validated): 603723 (avg. block size 546740 B) (Total open file blocks (not validated): 122) Minimally replicated blocks: 603723 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Blocks queued for replication: 0 Erasure Coded Block Groups: Total size: 0 B Total files: 0 Total block groups (validated): 0 Minimally erasure-coded block groups: 0 Over-erasure-coded block groups: 0 Under-erasure-coded block groups: 0 Unsatisfactory placement block groups: 0 Average block group size: 0.0 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 0 Blocks queued for replication: 0 FSCK ended at Wed Sep 30 12:23:06 CEST 2020 in 23305 milliseconds hdfs fsck / -openforwrite Status: HEALTHY Number of data-nodes: 3 Number of racks: 1 Total dirs: 63922 Total symlinks: 0 Replicated Blocks: Total size: 329765860325 B Total files: 528144 Total blocks (validated): 506549 (avg. block size 651004 B) Minimally replicated blocks: 506427 (99.975914 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9992774 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Blocks queued for replication: 0 Erasure Coded Block Groups: Total size: 0 B Total files: 0 Total block groups (validated): 0 Minimally erasure-coded block groups: 0 Over-erasure-coded block groups: 0 Under-erasure-coded block groups: 0 Unsatisfactory placement block groups: 0 Average block group size: 0.0 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 0 Blocks queued for replication: 0 FSCK ended at Wed Sep 30 12:28:06 CEST 2020 in 11227 milliseconds    ... View more

Hi @Madhur ,   Thank you for your prompt response. Unfortunately I do not have a subscription with Cloudera, I am unable to access the Knowledge Base. So far we have managed to get by with the free version of CDH 6.3 and the help of the community on and off these forums 🙂   Kind regards, Gyuszi Kovacs ... View more

@rblough- thank you very much for your prompt reply.   1) I got the block numbers from the HDFS status page in Cloudera Manager. Based on your question I checked the numbers on the NameNode UI. For the DEV environment all three DataNodes are online, showing 1,613,019 blocks on each (CM shows 1,613,104). For the PROD environment the NameNodeUI shows 477,464 blocks on each of the three DataNodes.   2) Yes, all of the DataNodes are showing as online. dfsadmin -report confirms this, so does the NameNodeUI and CM. Coincidentally, the report did not include the total block count, just the number of missing or under replicated blocks - everything sits at zero.   3) The replication factor is set to 3 in both environments.   Kind regards, Gyuszi Kovacs ... View more

Hello there, What happens when you try a manual kinit on host2 with explicitly specifying the principal and the realm, ie: kinit -kt keytab.file -p  user/HDY@HDY Also, could you share your krb5.conf files from both hosts, as well as the list of keys stored in the keytab you exported (klist -kt keytab.file)? Without this information it's a bit hard to analyze this issue. Kind regards, Julius  ... View more

Hi @palaaniappan , According to your screenshot I suspect there are invisible characters (new-line, tab, space, etc.) present in the name of your folder - see how there is a new line present after your IoT folders, but no new-lines for other folders. Please check your NiFi processors and make sure the file paths for creating the folders are correct. Sometimes when copy-pasting between different windows you might inadvertently paste some invisible characters as well. Kind regards, Julius ... View more

Bumping this thread as this issue seems to be still unresolved. Today I noticed that the documentation has been updated recently and for CDH 6.3 there are some new steps as far as Hue and HBase are concerned -https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hue_enable_apps.html The article also references this video - https://www.youtube.com/watch?v=FGDTGmaxvsM&feature=emb_title - which was uploaded to YouTube only recently, but shows a rather old version of both CDH and Hue 😕   ... View more

Hi @GangWar, The SOLR principal is not present in CM. ... View more

*bump* 🙂 any ideas? ... View more

I'm terribly sorry for the necro-bump. The issue I'm experiencing seems to be the same one the OP was facing. We're running CDH 6.3, and I encountered a problem when trying to install the Kudu Python client. I created the symlinks as recommended by mpercy, but I'm still unable to install the kudu-python client. Our cluster does not have direct access to the internet, so when possible we use an offline install. The official documentation says that the  Kudu C++ client libraries and headers are needed for the Kudu Python client. On Oracle Linux 7 trying to install devtoolset-3-toolchain ends in a failure, as a number of dependencies are missing:   Error: Package: devtoolset-3-gdb-7.8.2-38.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libpython2.6.so.1.0()(64bit) Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Disregarding that, running pip install --no-index --find-links file:///data0/home/jkovacs/kudu-python-1.10.0.tar.gz  results in the following errors: ERROR: Command errored out with exit status 1: command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rK2Qbk/kudu-python/pip-egg-info cwd: /tmp/pip-install-rK2Qbk/kudu-python/ Complete output (43 lines): Building from system prefix /usr/local /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/client.pxd tree = Parsing.p_module(s, pxd, full_module_name) /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/schema.pxd tree = Parsing.p_module(s, pxd, full_module_name) /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/errors.pxd tree = Parsing.p_module(s, pxd, full_module_name) Compiling kudu/client.pyx because it depends on kudu/config.pxi. Compiling kudu/errors.pyx because it depends on /usr/lib64/python2.7/site-packages/Cython/Includes/libcpp/string.pxd. Compiling kudu/schema.pyx because it depends on kudu/config.pxi. [1/3] Cythonizing kudu/client.pyx [2/3] Cythonizing kudu/schema.pyx [3/3] Cythonizing kudu/errors.pyx WARNING: The wheel package is not available. DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at <a href="<a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a>" target="_blank"><a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a</a>> WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3910>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3110>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3090>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e50>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e90>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ ERROR: Could not find a version that satisfies the requirement pytest-runner (from versions: none) ERROR: No matching distribution found for pytest-runner Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-rK2Qbk/kudu-python/setup.py", line 216, in <module> test_suite="kudu.tests" File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup _install_setup_requires(attrs) File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs replace_conflicting=True, File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve replace_conflicting=replace_conflicting File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match return self.obtain(req, installer) File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain return installer(requirement) File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg return fetch_build_egg(self, req) File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 121, in fetch_build_egg raise DistutilsError(str(e)) distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp8eqkO2', '--quiet', 'pytest-runner']' returned non-zero exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. I'd like to ask for some guidance, what to try next?     Thank you!   ... View more

Corrected: export HADOOP_CONF_DIR=/etc/spark/conf/yarn-conf/*:/etc/hive/conf:$HADOOP_CONF_DIR ... View more

We encountered the same problem after upgrading to CDH 6.3 from 5.15. The steps outlined by Bender helped us resolve the issue, with the following small differences: We modified the advanced configuration of the Spark 2 service: Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh The following line was added: export HADOOP_CONF_DIR=/etc/spark/conf/yarn-conf/*:/etc/hive/conf No cluster or service restart was necessary, simply re-deploying the client configs did the trick. ... View more

Hi Ben,   Thank you for your prompt response! The roles are as follows: HBase REST server, HBase Thrift Server, Master, 3x RegionServer.    Here are the latest logs I obtained this morning after: stopping (CM - HBase Actions - Stop, I get asked "Are you sure you want to run the HBase Graceful Shutdown command on the service HBase?", I click Stop) and trying to start the HBase service (CM - Hbase Actions - Start, here the prompt asks "Are you sure you want to run the Start command on the service HBase?", I click Start, the process then continues with "Starting 6 roles on service) ... and then it all started without issues. I'm baffled, but also happy that it's working correctly now. I did 2 stop&starts and 1 restart, the issue seems to be gone.   Yesterday I was "weeding out" a Java 1.7 installation that popped up on our DEV cluster and it was causing issues with Kerberos. I think it also interfered with our deployment of HBase.   Sorry for the false alarm, please feel free to remove this thread.   Thank you for your time!   Kind regards, Julius   ... View more

Thank you for this well written, comprehensive tutorial, it's very useful! Cheers! ... View more