Solved

Our ambari webui is encountering some errors when ambari-server restart. When some users are connected and we restart the ambari-server, it doesn't logout connected users and 401 errors appears This behaviour is very constraining because it generate lot of lines in ambari logs (Thread)How to force ambari to log out all connected users when ambari-server restart ?

Hi experts,As the root user, I am trying to delete a directory in HDFS which was created by root.However, when I try to delete it, it says "Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x"Why does it say permission denied on "/user" when I am trying to delete the directory "/tmp/root/testdirectory" The error message is below. [root@test02 ~]# hdfs dfs -ls /tmp/root/Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0Found 2 itemsdrwxrwxrwx - root hdfs 0 2021-08-09 20:35 /tmp/root/testdirectory-rw-r--r-- 3 root hdfs 0 2021-08-10 13:54 /tmp/root/test[root@test02 ~]# hdfs dfs -rmr /tmp/root/testdirectoryPicked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0rmr: DEPRECATED: Please use '-rm -r'

I am wondering what difference in IO can be expected for hbase with storage in the cloud VS storage on hdfs.I would expect that when data is retrieved from hdfs, it will be a lot faster than from the cloud (like adls-in my specific case adls gen2=abfs).Is there somewhere where I can test this?Or find a previous study for this?If this is the case, then one would expect that current hbase performance for reading data is a bit less than some years ago when everything was on premise and typically hdfs was used?  Maybe I am missing something obvious here, so any insight is appreciated ! 

Hello Experts, I have a requirement where nifi flows imp log information should be written to MongDB, so that another application can read mongodb and display those interesting logs info on web UI. I am looking for best way to do this.One way I am thinking of is tail nifi log (using tail file processor) and filter for imp log and put into MongoDb using native processor. Please suggest if any other better approaches,thanks in advance. ThanksMahendra

getting error for access to mysql  1. [cloudera@quickstart ~]$ mysql -uroot -pclouderaERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)2. [cloudera@quickstart ~]$ mysql -urootERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)3. [cloudera@quickstart ~]$ mysql -u root -p clouderaEnter password:ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)  in 3 step putting password as cloudera but it not works 

Hello, I am facing a challenge with authentication and authorizing Active Directory users on Cloudera CDP 7.1.6. I followed the steps here to make the necessary configurations (https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/security-kerberos-authentication/topics/cm-security-external-authentication-ad.html).The challenge is that I am able to login with AD users but there is no group to role mapping, which results in a blank page for the user.I checked this other page (https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/managing-clusters/topics/cm-security-authorization-user-roles.html) which handles mapping external authentication to roles. However, it skips Active Directory instructions and mentions only that of external programs and SAML scripts. Does anyone have an idea of how to map groups from an Active Directory source to Cloudera Roles? or is there some other documentation I should refer to? Thanks in advance for the support. 

I have to extract the first character from each string provided in the Flowfile attribute.Suppose I have one attribute as:KeyValue : Nifi Test ProjectExpected output : Key1: NTP Please let me know how to get this and which processor to be used. Thanks in advance.

Hi,i am running HDP 3.1 (3.1.0.0-78) , i have 10 data nodes , Hive execution engine is TEZ, when i run a query i get this errorERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex failed, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00, diagnostics=[Vertex vertex_1557754551780_1091_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1557754551780_1091_2_00 [Map 1] failed as task task_1557754551780_1091_2_00_000001 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0INFO : Completed executing command(queryId=hive_20190516161715_09090e6d-e513-4fcc-9c96-0b48e9b43822); Time taken: 17.935 secondsError: Error while processing statement: FAI

I would like to pass dynamic values to Max Bin Age" & "Max Number Of Entries" in MergeContent Processor. But it's not supporting expression language.Please suggest if there is any way to achieve this.

How do I install a module to run a Python script? I'm getting the following error: Traceback (most recent call last): File "/home/test.py", line 1, in <module> import nipyapi ModuleNotFoundError: No module named 'nipyapi' It works from the command line.

Is use of Flume in 2021 still the way to go? I tried searching for some insights on this topic and did not find much. Looking at the latest release it is in 2019, little or no activity in github, lots of unmerged pull requests, too. I'm planning on coding a custom source for Flume. My data is binary (not text), unstructured, arriving from a legacy control system. I do not plan (ATM) on having any filtering or processing applied to the data stream. The data stream will be then routed through memory channel to HDFS. FWIW, the environment I will be working in is closed and I can not use cloud based solution. Also the data rates will be to big to pipe them to through the internet. Looking for alternatives, there is lots of suggestions that sometimes do not feel like a Flume alternative, but then again, I'm new to this "big data" ecosystem and I might be mistaken. For example, Apache NiFi, Spark and alike. It might be that I could go with Kafka and skip the Flume al

Hello,I am trying to execute hdfs fsck command and am getting below error: [root@server1 root]# hdfs fsck / > /home/test/fsck_output_2-11-21Connecting to namenode via http://server1.com:50070/fsck?ugi=hdfs&path=%2FException in thread "main" java.net.SocketTimeoutException: Read timed outat java.net.SocketInputStream.socketRead0(Native Method)at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)at java.net.SocketInputStream.read(SocketInputStream.java:171)at java.net.SocketInputStream.read(SocketInputStream.java:141)at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)at sun.net.www.protocol.http.HttpURLC

Hi Everyone, I am quite new to CDP platform and I would just like to know if spark jobs running in CDE can possibly connect in Oracle database deployed on-prem in order to extract data?I appreciate any input.  Thanks,Glenn 

I've read query plans in SQL Server forever using a GUI, but text plans are like Greek to me (I don't speak Greek).  I'm trying to figure out if statements like this calculate the date "constant" for each row and cause performance lag, or if Impala knows to calculate it once and apply to all rows. select *from myTable-- the "months_add" is basically a constant, but I can see -- why a function might be calculated for each rowwhere tran_timestamp > months_add(now(), -1)limit 1;select *from myTable-- but what about this? this is definitely a constant-- so is this different than the query above and is-- the "constant" calculated for each row?where tran_timestamp > months_add('2021-01-01', -1)limit 1

Hi Cloudera Team, I have cleared the CCA175 exam but have not received my digital certificate yet. I have sent multiple emails to certification@cloudera.com but haven’t got any response from them or resolution. Could you look into this and check why this is taking so long!!

I have successfully deployed one node quickstart on AWS but are there any docs for a decent sized 3-5 node cluster on AWS to learn cloudera ? I am new to Cloudera and wanted to know what gets installed first etc ..If you have a doc or steps to install Cloudera ...say 3  or 5 node cluster in AWS it would be ideal ...thanksSam

Hi All I am working for a customer to set up a DR environment.  Customer has created a DR as a Separate instance of Cloudera and PROD is a different instance.  The replication is schedule-based as of now.  Can we use Kafka production offset location in DR instance? the customer is working on replication of Hive data and Kafka topics. ? will this work out ?  What is the ideal way to restore a streaming job in DR when PROD is down? Any best practices?  Let me know if you need more details.  Thanks in advanceMuthu   

I'd like to update stats (compute stats) for every table I have.  I have a ton of queries that are running without them.  I personally have access only to Hue for Impala, no console kind of access to server or anything.  I'm able to see status of stats for a table with describe formatted <table>, but it's a major pain to look through, and I have a ton of tables.   Is there any way to get a list of all tables and the status of their stats from Hue?   Thanks!

Hi,  is there some way that nifi could remove the attributes inside the XML tag name? Please advise with some example to help me. Appreciate. Thanks  Example: <particular>   <name>Nicholas</name>   <height>183</height>   <weight>75</weight></particular><particular>   <name>Debbie</name>   <height163</height>   <weight>45</weight></particular> to something like: <particular>   <name>Nicholas</name>   <height unit = "meter">183</height>   <weight unit = "kilogram" unit = "lb">75</weight></particular><particular>   <name>Debbie</name>   <height unit = "meter">163</height>   <weight unit = "kilogram" unit = "lb">45</weight></particular>

Hi all, I have a NiFi flow using DistributedMapCacheServer 1.11.3. I've noticed that this controller service is actually excluded when I create a template from the process group. I see other controller services that are being referenced from the flow in the template, but not this one, presumably because it isn't referenced. How exactly do I get this exported to a template? Any ideas? Regards

Is it possible to mask or greyed out the Token passed in Custom Property (Authentication )? I have one Token received from one API and have to pass it to another request.Issue: If I pass the token in custom property, it will be visible to all users who can access the flowfile. I need to mask these token as these value are displayed in Attribute and Content. Thanks for early response.   

I have the above flow and I am successfully listing the files in a given blob storage container. When I feed that to the FetchAzureBlobStorage processor however, the queue starts to fill up but not a single task in the queue ever gets processed. Any ideas regarding why this happens?

Every now and then, one of my processors fills the bulletin board with connection error messages from a FetchSentinelData 1.5 processor. Is there a way to suppress only those messges or route them to an output file to prevent them from filling the bulletin board each time?

Hello - Cloudera Manager is returning the following warning when connecting to MySql hosted in AWS RDS. How can this warning be resolved / suppressed? Log Warning: May 17 12:15:48 ip-10-179-61-80 cm-server[23717]: Mon May 17 12:15:48 BST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL SSby setting useSSL=false, or set useSSL=true and providetruststore for server certificate verification OS Type: [root@ip-10-179-61-233 ~]# cat /etc/redhat-releaseRed Hat Enterprise Linux Server release 7.9 (Maipo) Cloudera Binaries: [root@ip-10-179-61-233 ~]# rpm -qa | grep clouderacloudera-manager-daemons-6.3.1-1466458.el7.x86_64cloudera-ma

Hello Experts, I have configured Git as FlowPersisteceProvider for nifi registry.Tested with test repo and worked fine, but now pointed to my company repo and getting error in push/commit. The error is due to the policy is set in my org git repo, it accepts push/commit from the accout with email as *@myorg.com only.But Even though I configured my credentials (user & PAT) which belongs to my org for Git access from registry, NiFi registry pushes the code to Git with email as "nifiregistry@<hostname>". As its org level git repo ops team cant change this policy, so is there any way I can override this default "nifiregistry@<hostname>" to "my-user@myorg.com" so that git push will not be rejected ? This is how nifi reg is pushing and comitting, Authored and Committed Identity should be changed from default   Tried this : https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.2.0/administering-nifi-registry/content/git-user-configuratio