Solved

I am trying to deploy a HDP cluster with ambari. I met an error when deployingIt reported the error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in <module> BeforeInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 33, in hook install_packages() File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/shared_initialization.py", line 37, in install_packages retry_count=params.agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 125, in __new__ cls(names_list.pop(0), env, provider, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/env

Hello,while attempting to start ListenHTTP (port=10000) on a fresh cluster, I obtain the following error. Do you know what to change to get it work? ListenHTTP[id=6222ab5d-0177-1000-ffff-ffffaeaba266]Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.Failure is due to java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:10000: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:10000Cloudera Flow Management (CFM) 2.0.4.01.11.4.2.0.4.0-80 built 09/27/2020 09:52:49 CESTTagged nifi-1.11.4-RC1 Best regardsJaro

Hello Everyone,   I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).   I'm running the distcp command on the secure cluster like follows: hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder   It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).   When I looked into the task attempt logs, I find this error:     2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.Met

Dear All, I am looking for mini exercises/projects in Hadoop Cloudera to get familiar with the framework. Do you know any website or youtube channel that provides that? I d like to start basic from 'copy data from local to HDFS' to intermediate level projects. Thank you

Hi there.I'm using NiFi to get data from remote windows servers using the GetSmbFile processor.It works but I have a number of servers exposing data on SMB share and I'm trying to understand if I can catalog all my remote servers in a SQL table with needed parameters (eg. hostname, user, pass, share) to be passed to that processor.I see that GetSmbFiles doesn't support EL so I haven't found a working solution nor anything helpful on line.Is this achievable without writing my own processor or running external code?Any help is appreciated

I tried to move my HSI to another node on my production cluster without stopping hiveserver or metastore components, so i disabled it first from ambari UI (no problem at this stage).Now when i try to enable Interactive Query it shows a pop with an error message : You cannot enable Interactive Query now because Interactive Query is in the process of being disabled. This may take a few minutes. Try again later. Nothing in ambari logs neither in hive-interactive logs because it was uninstalled. Any advise please ? 

Hi,I want to  remove some old disk and added new disk with new mount points. Now the new disk are part of  CDH. Now I want to  transfer the blocks from old disk /data01 to new disk /data05 and then remove /data01 after move of all blocks.Is there a way to do this ?can I take datanode services down and then manually copy? will Cloudera able to find the new blocks on /data05 (new disk) ?

I'm currently working on CDH5.13 (Cloudera Distribution Hadoop), and i have a couple of questions: 1- I want to get the latest version of CDH(6.3.3). When i try to download it, i have this message (Your MyCloudera account is not currently connected with a CDH subscription which is required for CDH 6.3.3 and later.). Where do I go to get a subscription? And what is the cost of the subscription?  2- I tried to download older versions of CDH, but it says ( As of February 1, 2021, all downloads of CDH and Cloudera Manager require a username and password and use a modified URL. You must use the modified URL, including the username and password when downloading the repository contents described below.). I'm assuming i have to get a license to have this username and password. Where should I go to buy this? 3- I also tried to download Hortonworks Data Platform, but i have this message (You must be a HDP customer to access these downloads. If you believe

Hello,  I have just upgraded the JDK version of my Cloudera 5.10 cluster from 1.7 to 1.8 and everything starts ok but when I try to run a job from Talend the following error appears in HDFS Name Node log: 2021-03-22 13:11:32,711 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 69.117.42.187 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: DES CBC mode with MD5 encryption type not in permitted_enctypes list)]] If I go back to JDK 1.7 it runs perfectly.I have tried with 1.8.201 and also with 1.8.91 (since this was one of the tested ones).I think I have JCE and java.security correctly configured. We do not want to allow DES CBC in the KDC since it a weak algorith. Has it happened to anyone? Thanks in advance, Ángel

I am trying to transport a python environment with a PySpark according to: https://community.cloudera.com/t5/Community-Articles/Running-PySpark-with-Conda-Env/ta-p/247551https://henning.kropponline.de/2014/07/18/virtualenv-hadoop-streaming/https://community.cloudera.com/t5/Community-Articles/Using-VirtualEnv-with-PySpark/ta-p/245932 Last try I issued the following commands:   #Creating my relocatable environment:conda create -y -n py35 python=3.5 numpy pandas scikit-learnconda activate py35cd /opt/cloudera/parcels/Anaconda/envs/zip -r py35.zip py35#Runing the job:PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \pyspark2 \--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \--master yarn \--deploy-mode client \--archives /opt/cloudera/parcels/Anaconda/envs/py35.zip#py35 test.py   test.py:  

Hello, in our local filesystem we see thousands of subdirectories like "????????-????-????-????-????????????_resources" older than 5 days (and this is for guarantee nothing from any still running process).Additionally there are thousands of files and hundreds of subdirectories in "/tmp/hive" older than 5 days.Where do those relicts come from?How to get rid of them in an automated way? $ find -O1 /tmp -type d -name "????????-????-????-????-????????????_resources" -mtime +5 | wc -l26263$ find /tmp/hive -type d -mtime +5 | wc -l538$ find /tmp/hive -type f -mtime +5 | wc -l5784 Best RegardsCarsten

Hi all, I am getting error while executing the phoenix query. Error: Can't get the location for replica 0 (state=08000,code=101)org.apache.phoenix.exception.PhoenixIOException: Can't get the location for replica 0at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:117)at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1126)at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1418)at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2190)at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:872)at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:194)at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)at org

I am seeing frequent Cloudera Manager Service Monitor outages: SERVICE_MONITOR_PAUSE_DURATION has become bad: Average time spent paused was 39.5 second(s) (65.76%) per minute over the previous 5 minute(s). despite increasing the heap size to 7g and the 'off-heap' size to 24g. The machine often sees a high load (a NodeManager is also on the same machine), like 90 on a 24 core machine, so I suspect it might be starved of cpu when doing aggregation. The process regularly has +700 files open.  I have motivation to fix this issue since it causes data loss in the time series because SM pulls data  and misses data points for +15 minutes at times.  The WARN: AggregatingTimeSeriesStore: run duration exceeded desired periodis correlated with the above.  Is there a documented procedure to move Service Monitor to another machine while keeping existing data?  Perhaps like:0. stop SM to quiesce changes to /var/lib/cloudera-service-moni

I am trying to build a new maven spark project but I cannot seem to pull the spark maven dependencies. I keep getting the following error:  Could not resolve dependencies for project za.co.sanlam.custhub.spark:party-matching-rules:jar:1.0.0-SNAPSHOT: Failed to collect dependencies at org.apache.spark:spark-core_2.11:jar:2.4.0.7.1.4.9-1 -> org.apache.avro:avro:jar:1.8.2.7.1.4.9-1: Failed to read artifact descriptor for org.apache.avro:avro:jar:1.8.2.7.1.4.9-1: Could not transfer artifact org.apache.avro:avro:pom:1.8.2.7.1.4.9-1 from/to cloudera (https://repository.cloudera.com/artifactory/cloudera-repos/): Transfer failed for https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/avro/avro/1.8.2.7.1.4.9-1/avro-1.8.2.7.1.4.9-1.pom ProxyInfo{host='10.0.0.132', userName='null', port=8080, type='http', nonProxyHosts='null'}: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested

I have a NiFi flow [on a 2 processor machine using 8/5 thread counts] where the InvokeHTTP block is no longer processing files from the queue [shown below]:As you can see there are 8 processes running but these seem to be stuck for hours and the only way out of it is manually stopping and terminating the block. This results in the following log message:2020-05-18 08:31:06,228 ERROR [Timer-Driven Process Thread-12 <Terminated Task>] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=f2f13d29-016b-1000-293d-c0cb04835881] Routing to Failure due to exception: java.net.SocketTimeoutException: timeout: java.net.SocketTimeoutException: timeoutjava.net.SocketTimeoutException: timeout at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:593) at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:601) at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:146) at okhttp3.internal.http2.Http2Code

Hi,We need help with respect to order of records in Nifi. We have couple of custom processors which work in batch.There is processor which reads the data from source (Processor A) & another processor which writes the data to SFTP (processor B).Processor A outputs the data in form of multiple flow files (100 records from source are written in one flow file). Assume if there are 600 records in source, we get 6 FFs out of processor A.Process A & Processor B are directly connected to each other & entire execution happens in Single node only. However, we are finding that one some occasions the Process B is processing flow files out of order. For example, third flow gets processed first & then the second flow file. Understood the flow fill prioritizer needs to be chosen explicitly in the connection between Process A & Process B. However, confused up on FirstInFirstOut prioritizer & OlderFlowFilePrioritizer. Kindly provide pointers to guarantee the

We run multiple NiFi clusters for dev, test, integration, and failover.  After recently upgrading from NiFi 1.0.8 to 1.11.4, we've found that NiFi now registers *all* processor state changes as Registry changes.  Since these processors must be in different states between the various clusters, it's become very problematic. Is there any way to configure NiFi and/or the Registry to ignore processor states like it used to?  I've looked through the properties files and admin docs for both, and haven't found anything configuration that supports that.

Hello, I have a processor scheduling setup as below - Run Schedule - 30 secRun Duration  - 0 msConcurrent Tasks - 1 In this case, assuming that each task takes more than 30 secs to complete once it starts, can 2 (or more than 2) processor tasks run at the same time?  0s - Task1 starts 30s - Task2 starts (Task 1 is still running) Is there any way to prevent overlapping task runs i,e Task 2 should start at 30s only if Task1 that started at 0s is complete? Thanks!

@ApacheNifi I have this flow that take files from Azure Blob Storage (with ListAzureBlobStorage processor).Now.. is there some trigger or notification mechanism that indicates that new files to be processed have been uploaded to the Azure blob??Thank you so much 

Hi everyone, I noticed that after CDP 7.1.2.1, you aren't exposing the yum repositories anymore, at least in the URLs that I know. For example, in https://archive.cloudera.com/p/cdh7/7.1.2.1/redhat7/yum/ we have access to packages like hadoop, but in https://archive.cloudera.com/p/cdh7/7.1.6.0/ we don't have access to yum path anymore and it becomes impossible to install the packages that you distribute with CDP 7.1.6 using yum. Do you provide access to the yum repository in any other location or are we forced to install cloudera manager if we want to have access to some packages that could be installed previously with yum but now only using parcels? Thanks in advance.

I am doing the coursera course in big data. I followed the instruction and did the installation. I realised then i had to enable the Virtual Technology in BIOS (which was enabled anyway). However, i still get the same error: 'Failed to Open a session for the virtual machine cloudera-quickstart-vm-5.4.2-0-virtualbox' The virtual machine 'cloudera-quickstart-vm-5.4.2-0-virtualbox' has terminated unexpectedly during startup with exit code 1 (0x1). More details may be available in 'C:\Users\badao\VirtualBox VMs\cloudera-quickstart-vm-5.4.2-0-virtualbox\Logs\VBoxHardening.log'.Result Code: E_FAIL (0x80004005)Component: MachineWrapInterface: IMachine {b2547866-a0a1-4391-8b86-6952d82efaa0} Please help me with a comment

I am integrating Apache NiFi 1.9.2 (secure cluster) with HDP 3.1.4. HDP contains Zookeeper 3.4.6 with SASL auth (Kerberos). NiFi nodes successfully connect to this Zookeeper, sync flow and log heartbeats.Meanwhile, NiFi processors using Zookeeper are not able to connect. GenerateTableFetch throws: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)  ListHDFS fails attempting to write its state to ZooKeeper: o.apache.nifi.processors.hadoop.ListHDFS ListHDFS Failed to retrieve timestamp of last listing from the State Manager. Will not perform listing until this is accomplished. But GetHDFS (which doesn't communicate with zk) works OK.Enable HiveConnector works without errors.Thank you in before for any ideas and support.

Hi all.I'm facing a CM configuration error I can't understand.I've kerberized a Cloudera 5.16 cluster which authenticates against an AD/DC controller.I've set aes256-cts and aes256-cts-hmac-sha1-96 encryption types.CM reports a configuration issue:I've a separate cluster, configured with the same values and authenticating to the same AD/DC controller, that have no errors. I'm struggling on that error but can't solve. Any idea?  

Hello all, I tried to set up Ambari  for a research project, however I get  "ERROR 403:  FORBIDDEN" (Access denied ) when trying to download the repositories from  Hortonworks & I don't know what I can do know or how to install Ambari and all its HDP services manually. Is anyone else here encountering this problem or does anyone have a solution for it?I tried it with multiple versions already. Thanks in advance!

Hi all. First of all, let me wish you a happy and prosperous new year.I am trying to install HDP via Ambari. Based on the latest docs (https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/access_ambari_paywall.html). According to it, starting from HDP 3.1.5 / Ambari 2.7.5, accessing Ambari repositories requires a special set of credentials. Does anybody has an idea of how to get such credentials? Unfortunately, instructions on the specific documentation site are not intuitive.Thanks in advance!