Q&A

I am having trouble with the Impala connection from R with the following error:  'Error: nanodbc/nanodbc.cpp:983: 00000: [unixODBC][Cloudera][DriverSupport] (1100) SSL certificate verification failed because the certificate is missing or incorrect.' Here is the connection details in our code: impala = src_impala(      drv = drv,      driver = "/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.so",      host = "cdp-tdh-de3-master0.cdp-tdh.u5te-1stu.cloudera.site",      database = db,      port = 21050,      uid = username,      pwd = password,      AuthMech = 3,      transportMode="http",      httpPath="cdp-tdh-de3/cdp-proxy-api/impala",      ssl = 1,     sslTrustStore="/home/csso_innovation.cdh/gateway-client-trust.jks") return(impala) Any help is appreciated.Thanks,Gozde

Hi,I would like to convert this json to csv. But it failed because of the special characters in the json content. Below is the json i would like to convert to csv. So the responseMessage value is where the problem is & its auto-generated in that format. Thank you.INPUT{"responseMessage": "hdhdhdhdh:dgdgdgdg "fdfdf" bbbbbbb","result": "failed","code": "34"} DESIRED OUTPUT{"responseMessage": "hdhdhdhdh dgdgdgdg fdfdf bbbbbbb","result": "failed","code": "34"}

We restarted kafka server. It complained about offline partitions. We used kafka-topics.sh command to find more info. It didn't show anything. Could someone instruct me how to delete offline partitions manually in detail? Thanks.

hi, I download latest python swagger manager api, but not found api keys for:get current resource pools config: GET cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues  add resource pool, and reconfigure old: POST /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues/root get scheduler configs: GET /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/configs reconfigure pool scheduler: PUT /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/configs reconfigure pool parameters:PUT /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues/root.pool_name please add it, or update

Hi All,After the integration of AD with kerberos we are not able to start the RM due the "User impersonate" issue. [Error: User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM] Loggers for RM:2021-04-21 16:10:08,878 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in state STARTEDorg.apache.hadoop.service.ServiceStateException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COMat org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.hadoop.ser

Hi, I am new to Nifi, so please excuse if this is a noob question. I have set up a 5 node Nifi cluster on GCP and using LDAP for authentication. I generated self-signed certs using the toolkit and I am able to login and access the UI at https://<hostname>:8443/nifi/. Everything looks good with this setup. Now, I added a HTTPS load balancer that routes the requests to backend Nifi nodes, when I try to access the Nifi UI at https://<load-balancer>/nifi/ it throws below error  INFO [NiFi Web Server-251] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started xx.xx.xxx.x [<anonymous>] GET https://nifi.xxxxx.com/nifi-api/flow/current-userWARN [NiFi Web Server-251] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed xx.xx.xxx.x GET https://nifi.xxxxx.com/nifi-api/flow/current-user [Anonymous authentication has not been configured.] I generated the certificate using below command ./tls-toolkit.sh standalone -n 'nifixxxxxx[1-5].xxxx.c

In Apache Nifi, there are connections between each processors, which acts like queue of FlowFiles, and Nifi by default persists data content of FlowFile on disk. Does it mean each of such connection persists FlowFiles on disk? If that were true, each time of delivery of FlowFiles from one processor to another would mean one disk read and write, thus more processors would lead to more disk reads and writes, which in turn would lower the entire throughput. Is my understanding correct? and what is the best practice to avoid it, writing all things in one processor? Thanks.

Hello Everyone,We are a leading bank and have implemented Cloudera in our on-premises environment. Our management has taken a decision to implement Kubernetes in future so I would like to know which Cloudera services supports Kubernetes and which services does not.  Appreciate if someone can help me in this regard. 

When we restart or start kudu tablet server component through cloudera manager, it takes a long time to communicate with kudu master, it takes more than 10 minutes. why does this occur? when we start the kudu tablet server component and follow the log, it keeps loading the lines as shown in the screenshot below: 

Hi community, I am a newbie at Nifi and unfortunately i have to build 3 node securedcluster and nifi registry on my own. The company had a standalone nifi node which is connected to nifi registry. I wanted from network team to clone this standalone nifi node and wanted to build 3 node cluster with them only adjusting configurations. Lets say the standalone node was server1 and I also have server2,server3(clones of server1) at the moment. My plan was build 2 node cluster with clone servers first because i didn't want to malfunction the original server while i was trying to configure cluster. If i succeed 2 node cluster without a problem then i will add the original server to cluster. The certificates of server2 and server3 didn't created by nifi-toolkit it was provided by another team in the company and i used them in keystore and truststores for this servers. I thought the configurations mostly correct because the automated process on the original server also works on clone n

How do you connect to a CDP Data Hub NiFi Registry from a local machine? 

I'm new to hive trino. I would like to concatenate three text columns with "_" as separator. So I wrote following code select A.*,B.Name_group,mail_group,Occupation_group,CONCAT_WS("_",mail_group,Occupation_group,Name_group) combined_error from hive.employee.Job A left join hive.employee.Name B on A.Name_id=B.Name_id left join hive.employee.mail C on A.mail_id= C.mail_id left join hive.employee.Occupation D on A.Occupation_id=D.Occupation_id   But I got following error message Colum '_' cannot be resolved. Can you help me to fix the issue

Hi All, Good Morning! Please help me in removing Duplicate Records without using cache services (Apache NIFI)Error Log

Can't start datanode -----------------------------------------------------------------------------------------------------------------------------------STARTUP_MSG: java = 1.8.0_312************************************************************/2022-05-03 08:42:09,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]2022-05-03 08:42:09,809 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMainjava.io.IOException: Running in secure mode, but config doesn't have a keytabat org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:210)at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2259)at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2308)at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2485)at org.apache.hadoop.hdfs.server.datanode.Data

there are like 1k finished jobs and i want to check allocated CPU and memory of those jobs. how to do that? i tried in command line but it shows when i enter job id and in my yarn UI CPU and memory metrics are not there.

We specified the environment variables as below: export AWS_ACCESS_KEY_ID=xMK6bdX8iY**************************************export AWS_SECRET_KEY=34*************************************** After connecting to a hive session , we specified the s3a credentials: set fs.s3a.endpoint=cluster.domain.*;set fs.s3a.access.key=$$$$$$$$$$$$$$###;set fs.s3a.secret.key=####$$$$; Tried to create a table using the below query (with directory location in s3 bucket : (s3a://test/dir2/) and received the preceding error ; even though the s3 credentials were already specified as stated above: 0: jdbc:hive2://> CREATE EXTERNAL TABLE s3dir (. . . . . . . . > col1 int,. . . . . . . . > col2 string,. . . . . . . . > col3 string,. . . . . . . . > col4 string. . . . . . . . > ). . . . . . . . > ROW FORMAT DELIMITED FIELDS TERMINATED BY ','. . . . . . . . > LOCATION 's3a://test/dir2/'. . . . . . . . > TBLPROPERTIES (. . . . . . . . > "s3select.format" = "csv".

Hi,Please, is there a way to reference a key part of JSON file that can come in different structures.For example i want to get the value of "requestUid" from these different JSON structures just with a single processor. So, irrespective of JSON formats or structures sent, i would want to get the value of requestUid.Thanks.For example:1){"responseMetaData": {"type": "ACK","userId": "SYSTEM","requestUid": "1742963e-f02f-4db6-88e0-6af8c4d4825b",,"systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","responseMessage": "","execTime": "1216"}} 2){"responseMetaData": {"type": "ACK","sendTime": "23/03/2019 09:16:25.305277","timeZone": "4","userId": "SYSTEM","branchCompanyId": "784100","branchLocationId": "784101","systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","requestUid": "yywhhnan56289200200","execTime": "1216"}}

Hi,I am unable to start cloudera management services. The "Continue" button is enabled and all the nodes are added in cloudera manager as shown in ss below.   I haven't created the cluster yet and haven't distributed the parcels aswell. I face the same issue when I try to create cluster it does not move forward even the continue button is enabled as seen in screenshot below: Kindly help me resolve the issue.  Regards

 Hi,There are two HikariCP jars in the class path:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar/data/perftools/collector-spark-sandbox/lib/HikariCP-3.2.0.jarThe 1st is pre-packaged with spark, the 2nd is included by me. Since /data/perftools/spark-prod-reports-3.0.0/bin/spark-submit's list of jars is checked first before checking my jar list, the classloader is loading HikariCP-2.5.1.jar instead of the desired one, HikariCP-3.2.0.jar that is provided via the flag  --jars .  The problem is shown below. Notice the 4th and 5th lines are loading Hikari from the default spark package jar folder, not from my jar list located at /data/perftools/collector-spark-sandbox/lib/.[Loaded slick.util.ConfigExtensionMethods$ from file:/data/perftools/collector-spark-sandbox/lib/slick_2.12-3.3.3.jar][Loaded slick.jdbc.JdbcDataSource$$$Lambda$1478/767152605 from slick.jdbc.JdbcDataSource$][Loaded slick.jdbc.hikaricp.HikariCPJd

Hi, I'm trying to setup the data engineering experiences (ECS) on top of a CDP base installation.These are licensed installations, so we first installed and configured a base cluster and we have been working on it.We want to setup the data experiences so we can move some work between the cdp public cloud and the cdp private cloud experiences since the control planes are similar. The goal is to work from the CDP private ECS enabled cluster and not use the base cluster unless any components are needed for the ECS/experiences cluster from the base. Trying to comb through the documentation to find the relationship between CDP base and CDP ECS enabled experiences cluster. What roles/manager/services are being used from base cluster in the experiences cluster? Understand a base "core" (manager/runtime etc.) is needed but not sure if I understand the dependencies between the two. The documentation says start with a base cluster and install ECS - well there a many flavors for base te

Hi, I am unfamiliar with Cloudera and tasked with enforcing a log retention policy on all services, currently working in NIFI. Our NIFIis part of CFM-2.2.3.2.I have changes to make to logback.xml, but they are overwritten when a new process directory is created. What is the source of the config.zip that is downloaded to create the process directory and how can I rebuild config.zip to include a copy of logback.xml with my changes? I couldn't find this in the docs I had access to, nor clearly in the cloudera logs I examined.  Many thanks in advance!

We have a NIFI cluster which have 3 nodes. All the nodes were working fine earlier. But now if all 3 nodes are connected to cluster and we right click on NIFI UI canvas and select "Configure" then we are getting error - "An unexpected error has occurred. Please check the logs for additional details."   Below are more details fetched from network tabs on inspection on chrome. The same error can be found if we access NIFI UI using any node's hostname - Request URL:https://<node1>:<port>/nifi-api/flow/process-groups/{process_group_id}/controller-servicesRequest Method:GETStatus Code:500 Internal Server ErrorIf we disconnect node 3 from cluster then we are able to click on "Configure" and list all controller services on NIFI UI. Issue seems to be with node 3. We have synced flow.xml.gz file from node2 to other node ie.node1, node3 and also restarted the NIFI. But still the issue is there.We have got this error in nifi-user.log -"o.a.nifi.web.api.config.

Hi community, I am a newbie to Nifi and unfortunately, We don't have a system admin with the knowledge of Nifi installation for production and I  have to build a 3 node secured cluster without Zookeeper. I understood that with the NiFi 0.x baseline, the cluster has a cluster manager which serves a similar role and did not use a zookeeper at all. But I am not sure how to use NiFi 0.x baseline, If someone can help me with steps/guidance to configure using NiFi 0.x baseline to set up a three-node cluster that will be great. Best,Shanoj

Hi! So I am very confused about how regular expressions and groups work in nifi.I read documentation and I saw that ExtractText processors always exctracts more attributes than needed somehow.So I have this file with the line like this9999, text And I wrote regular expression to extract  value 9999 for attribute call number.  (\d{4})But instead of one attribute number I am getting number0, number and number1 attributes. Can someone please explain me why is this happening, because documentation explanation is quite complex really.  Thank you beforehand!

Hello! Sorry for my english. I am completely new to nifi and I am learning SplitText processor. So here's the case. I have the comma separated txt file, something like this:KeyWord, SomeInformation   <---1st line is schema.KeyWord1, "information"KeyWord2, "information"KeyWord1, "another information"KeyWord2, "another information" and so on. So the question is how can I split this file on a few files based on KeyWord? So the every line with KeyWord1 go to one file, every line with KeyWord2 go to another file and so on? Thank you beforehand!