Q&A

Hello, I am facing a challenge with authentication and authorizing Active Directory users on Cloudera CDP 7.1.6. I followed the steps here to make the necessary configurations (https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/security-kerberos-authentication/topics/cm-security-external-authentication-ad.html).The challenge is that I am able to login with AD users but there is no group to role mapping, which results in a blank page for the user.I checked this other page (https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/managing-clusters/topics/cm-security-authorization-user-roles.html) which handles mapping external authentication to roles. However, it skips Active Directory instructions and mentions only that of external programs and SAML scripts. Does anyone have an idea of how to map groups from an Active Directory source to Cloudera Roles? or is there some other documentation I should refer to? Thanks in advance for the support. 

Hi, I have a table in Impala and it had some files in its location in hdfs. All of a sudden it can not read the files. I tried to invalidate and recover the partitions but nothing happened. I also moved the files from this folder into a new's folder path but it couldn't read them too. What can I do ? UPDATETried to repair the table in hiveset hive.msck.repair.batch.size=1;set hive.msck.path.validation=ignore;msck REPAIR TABLE db.tablebut received this error :Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask  

Hi experts,We are trying to copy hive tables from one cluster to another to do some testing.What is the proper way of doing this? Is it possible to distcp the hive tables at the hdfs level first and then somehow run a hive query to somehow have those hive tables recognized by hive? Any help is much appreciated. Thanks,

I have to extract the first character from each string provided in the Flowfile attribute.Suppose I have one attribute as:KeyValue : Nifi Test ProjectExpected output : Key1: NTP Please let me know how to get this and which processor to be used. Thanks in advance.

Hi,i am running HDP 3.1 (3.1.0.0-78) , i have 10 data nodes , Hive execution engine is TEZ, when i run a query i get this errorERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00Vertex failed, vertexName=Map 1, vertexId=vertex_1557754551780_1091_2_00, diagnostics=[Vertex vertex_1557754551780_1091_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1557754551780_1091_2_00 [Map 1] failed as task task_1557754551780_1091_2_00_000001 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0INFO : Completed executing command(queryId=hive_20190516161715_09090e6d-e513-4fcc-9c96-0b48e9b43822); Time taken: 17.935 secondsError: Error while processing statement: FAI

I would like to pass dynamic values to Max Bin Age" & "Max Number Of Entries" in MergeContent Processor. But it's not supporting expression language.Please suggest if there is any way to achieve this.

Hi, Can somebody please advise me on how to connect Cloudera impala to Dbeaver and query the impala tables via Dbeaver.  I have tried some online resources, but that is not helping me much.  Thank You!  

I live in a GMT +2 timezone but we collect all our data as GMT. My PutDatabaseRecord processor receives JSON with one field being the timestamp, e.g.: 2021-07-19 14:52:39 If I look at the flowfile just before the PutDatabaseRecord processor this timestamp is correct. However it gets inserted into the database as 2021-07-19 16:52:39 I can't find any documentation on how to prevent this in 1.14.0. Any idea what to do? Currently I'm subtracting an additional 2 hours to make up for this but that's not a good permanent fix. Edit:Is this perhaps related to this issue?  @mburgess  : if not and this is a bug then I can give more info.https://issues.apache.org/jira/browse/NIFI-8768?jql=text%20~%20%22time%20zone%20JsonTreeReader%22 Edit 2: this worked previously in 1.12.2

How do I install a module to run a Python script? I'm getting the following error: Traceback (most recent call last): File "/home/test.py", line 1, in <module> import nipyapi ModuleNotFoundError: No module named 'nipyapi' It works from the command line.

Is use of Flume in 2021 still the way to go? I tried searching for some insights on this topic and did not find much. Looking at the latest release it is in 2019, little or no activity in github, lots of unmerged pull requests, too. I'm planning on coding a custom source for Flume. My data is binary (not text), unstructured, arriving from a legacy control system. I do not plan (ATM) on having any filtering or processing applied to the data stream. The data stream will be then routed through memory channel to HDFS. FWIW, the environment I will be working in is closed and I can not use cloud based solution. Also the data rates will be to big to pipe them to through the internet. Looking for alternatives, there is lots of suggestions that sometimes do not feel like a Flume alternative, but then again, I'm new to this "big data" ecosystem and I might be mistaken. For example, Apache NiFi, Spark and alike. It might be that I could go with Kafka and skip the Flume al

I have been some issues to configuring Apache Nifi using an existing certificate.My use case is: I'm generating the truststore and the keystore from an existing tls.pem and tls.key that my ingress is using, from this I set the referent configurations of TLS and OpenId (I have created a custom image based on the official Nifi's image).  Everything its working, although when I try to access the UI and the redirects to the openId occurs the Nifi throw an exception, these are the last logs shown in the nifi-user.log: 2021-07-22 09:58:05,814 INFO [NiFi Web Server-25] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=MY-HOST) GET https://MY-HOST/nifi-api/flow/current-user (source ip: xx.xxx.xxx.xxx)2021-07-22 09:58:05,815 INFO [NiFi Web Server-25] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=MY-HOST2021-07-22 09:58:05,818 INFO [NiFi Web Server-25] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=MY-HOST], groups[] does not have permission to

I set up a process group to pull ~10^8 S3 files, consolidate them and save to HDFS. My FetchS3Object processor fails after 2-17 hours with the following error. When this happens, a ConsumeKafka_2_6 processor in a separate group fails with the same error message. Other processors have this message flicker in and out, but can typically self-resolve.  16:29:54 UTC ERRORFetchS3Object[id=50cb6c89-c4a3-3df6-807b-7f92555fd572] FetchS3Object[id=50cb6c89-c4a3-3df6-807b-7f92555fd572] failed to process session due to Failed to import data from com.amazonaws.services.s3.model.S3ObjectInputStream@1a9a94ff for StandardFlowFileRecord[uuid=bd5ccda4-d432-4086-8f4f-bb763afd108b,claim=,offset=0,name=Apr-2020/20200420-17H53M38S.json,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.FileNotFoundException: /opt/nifi/content_repository/993/1627576194712-2847713 (No such file or directory); Processor Administratively Yielded

Is it possible to get Impala hostname or datanodes information using Cloudera Manager API?I'm trying to automate some tests where I require to get the Impala hostname to run some connection checks. I dont want to hardcode the Impala hostnames in the tests. The tests would run on different environments so would like to find the hostname information or IP address of the configured Impala data nodes from Cloudera Manager API?I'm able to get all the hosts IP addresses from the API but cannot filter or query a specific IP belonging to Impala from the response. Our organisation is using v33 APICloudera Manager 6.3.0Thanks,DevDB

Hello , In our cluster Namenode switchover is happening everyday . Can anyone please help me in knowing which logs to check for the root cause of switchover which is happening again and again. @namenode   

Hi, recently a problem with HBase Master and Region Servers arose.  The error in CM is following: The Cloudera Manager Agent is not able to communicate with this role's web server.  for master and all three workers as well.   The problem is connected with Kerberos (from my guess after investigating logs): Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)   So far I tried: Restart the cluster completely Regenerate Kerberos credentials: Stop Cluster Go to Administration->Security->Kerberos Credentials Select all and Regenerate selected Start Cluster Regenerate Keytab Stop Cluster Go to Hosts->All hosts Select all hosts and select Regenerate Keytab from Actions Start Cluster Regenerate Keytab and then regenerate Kerberos credentials point 3 and then point 2 Manually try of kinit I have successfully kinit on HTTP on HBase master node and workers HBase on HBase master node and workers No

How to list all files owned by a specific user in hdfs?

I can not start ambari server due to an errors:   2021-07-27 09:55:47,338 WARN [C3P0PooledConnectionPoolManager[identityToken->1br2om1ai1rsa6bt17v8a7t|338fc1d8]-HelperThread-#2] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@56b6d697 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(Co

which processor can be used to extract(read) only the header( 1st row of the file) of a csv file.

There is no SHC available for Spark 3.x and Scala 2.12.x. Why did Community stops support for the updated spark&scala version? Is there any alternative opetion for Spark Hbase Connector for Spark 3.x?   Thanks in advance.!

Hello,I am trying to execute hdfs fsck command and am getting below error: [root@server1 root]# hdfs fsck / > /home/test/fsck_output_2-11-21Connecting to namenode via http://server1.com:50070/fsck?ugi=hdfs&path=%2FException in thread "main" java.net.SocketTimeoutException: Read timed outat java.net.SocketInputStream.socketRead0(Native Method)at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)at java.net.SocketInputStream.read(SocketInputStream.java:171)at java.net.SocketInputStream.read(SocketInputStream.java:141)at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)at sun.net.www.protocol.http.HttpURLC

Hi Team, For the last two months, I am facing issues with the hue. Every 3 hr once hue is down and I can see the webpage as, Proxy ErrorThe proxy server received an invalid response from an upstream server.The proxy server could not handle the request GET /accounts/login.I have changed multiple configurations but still facing issues. Please help me with this.I am using cdh5.11.1I am attaching the screenshot too. Regards,GD

Hi Everyone, I am quite new to CDP platform and I would just like to know if spark jobs running in CDE can possibly connect in Oracle database deployed on-prem in order to extract data?I appreciate any input.  Thanks,Glenn 

Hi guys, I'm looking for a way to kill impala queries that runs more than X hours in our cluster. Because there is no built in way, we thought of several ways: we want to get all queries in flight (or from every coordinator or from all of them together - doesn't matter).is there a way to do it and get the duration of each query? via the impala-shell/api? we can easily write a python/ bash script that uses the way but we don't know a way to get all running queries duration. Thank you!  

I've read query plans in SQL Server forever using a GUI, but text plans are like Greek to me (I don't speak Greek).  I'm trying to figure out if statements like this calculate the date "constant" for each row and cause performance lag, or if Impala knows to calculate it once and apply to all rows. select *from myTable-- the "months_add" is basically a constant, but I can see -- why a function might be calculated for each rowwhere tran_timestamp > months_add(now(), -1)limit 1;select *from myTable-- but what about this? this is definitely a constant-- so is this different than the query above and is-- the "constant" calculated for each row?where tran_timestamp > months_add('2021-01-01', -1)limit 1

Hi Cloudera Team, I have cleared the CCA175 exam but have not received my digital certificate yet. I have sent multiple emails to certification@cloudera.com but haven’t got any response from them or resolution. Could you look into this and check why this is taking so long!!