Find answers, ask questions, and share your expertise
Recently active
Hi, I've been successfully secure EFM to Keycloak server with oidc auth. But Minifi agent wont show up in EFM Dashboard. I've check both EFM and minifi log but not found any clues. Need your help. Here my conf/efm.properties config:# Web Server TLS Propertiesefm.server.ssl.enabled=trueefm.server.ssl.keyStore=/home/efm/certs/keystore.jksefm.server.ssl.keyStoreType=jksefm.server.ssl.keyStorePassword=ksPasswdefm.server.ssl.keyPassword=ksPasswdefm.server.ssl.trustStore=/home/efm/certs/truststore.jksefm.server.ssl.trustStoreType=jksefm.server.ssl.trustStorePassword=changeitefm.server.ssl.clientAuth=WANT# User Authentication Propertiesefm.security.user.auth.enabled=trueefm.security.user.auth.adminIdentities=adminefm.security.user.auth.autoRegisterNewUsers=trueefm.security.user.auth.authTokenExpiration=12hefm.security.user.certificate.enabled=trueefm.security.user.oidc.enabled=trueefm.security.user.oidc.issuerUri=https://keycloak.domain.com:8443/realms/efmefm.security.user.oidc.clientId=
queries in hue using impala are returning the following error: "results have expired, rerun the query if needed" as shown in the screenshot below:what could be causing this error and how to fix this issue?
When I am tring to deploy new cluster, I am getting the below error. Error for parcel CDH-7.1.7-1.cdh7.1.7.p74.21057765-el7.parcel : Hash verification failed.
I am using NiFi v.1.16.1 When I attempt to retrieve the weather data from the below URL, I don't get any response unless I specify "Always Output Response", and then I get an Access Denied response. However, these are free data feeds that work just fine using any other client,e.g. curl https://w1.weather.gov/xml/current_obs/KSFO.xml Anyone have any ideas what may be going on here?
Hi all, I am hoping I can get some help with this. I have a process that's sending a set of JSON records to a NiFi flow. Each JSON record in the array is a set of criteria/conditions for which I run QueryElasticSearchHTTP to get matching records from ElasticSearch. I am using a Merge Content processor at the end to merge all of this data and send back the JSON array as response. It works fine when for example I send 10 JSON criteria records in the array and QueryElasticSearchHTTP is able to find a document in the index for each criteria it receives. It bundles up the 10 resultant records from Elastic and sends the response through. However, if QueryElasticSearchHTTP finds only 8 records matching the criteria, it then waits forever at the Merge Content processor. I realise this is because it has received 10 records initially and split them into 10 flow files. When it looks to merge, it finds only 8 flow files and is waiting for the remaining 2 which never come through. This bl
Hi,is there any documentation to install R and SparkR in a gateway node of a DataHub?I have CM 7.5.2 and a CDP Public subscription.Spark version currently configured:spark 2.4.8spark 3.1.2 Thank youAndrea
Testing this on our non-prod cluster and finding Cloudera manager won't start after switching from Oracle to Open JDK. Followed the Cloudera documentation and it appears to be working, but after removing Oracle RPM things went kablooey. Anyone have any suggestions? From CM log: cResourcePool$ScatteredAcquireTask@1f258f49 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.n
Hi, I'm new in big DATA,I'm working with HDP 2.6.5, Ambari 2.6.2, HBase 1.1.2I would like to make a weekly full backups of all tables of HBase (all regionservers and my HBase master)and daily incremental backups of same thingsNo one has made a backup, so that's reason I am trying to make one.HBase is kerberised. so I did a kinit as hbase userI tried those commands on my hbase master (in local on it) :hbase backup create full hdfs://[FQDN server]:[port]/ -set /etc/hbase/hbase_backup_full/test -w 3goal : backup all tables in the file called test (I've created it so I don't understand why it tells me that's empty, and I've made a chown hbase:hbase on it as well as on the directory hbase_backup_full)issue : 634 ERROR [main] util.AbstractHBaseTool: Error running command-line toolIOException: Backup set '/etc/hbase/hbase_backup_full/test' is either empty or does not existcould anyone help me please ?Do I need to modify some config files to do this ? if yes which ?Thanks 🙂
I am having trouble with the Impala connection from R with the following error: 'Error: nanodbc/nanodbc.cpp:983: 00000: [unixODBC][Cloudera][DriverSupport] (1100) SSL certificate verification failed because the certificate is missing or incorrect.' Here is the connection details in our code: impala = src_impala( drv = drv, driver = "/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.so", host = "cdp-tdh-de3-master0.cdp-tdh.u5te-1stu.cloudera.site", database = db, port = 21050, uid = username, pwd = password, AuthMech = 3, transportMode="http", httpPath="cdp-tdh-de3/cdp-proxy-api/impala", ssl = 1, sslTrustStore="/home/csso_innovation.cdh/gateway-client-trust.jks") return(impala) Any help is appreciated.Thanks,Gozde
Hi,I would like to convert this json to csv. But it failed because of the special characters in the json content. Below is the json i would like to convert to csv. So the responseMessage value is where the problem is & its auto-generated in that format. Thank you.INPUT{"responseMessage": "hdhdhdhdh:dgdgdgdg "fdfdf" bbbbbbb","result": "failed","code": "34"} DESIRED OUTPUT{"responseMessage": "hdhdhdhdh dgdgdgdg fdfdf bbbbbbb","result": "failed","code": "34"}
We restarted kafka server. It complained about offline partitions. We used kafka-topics.sh command to find more info. It didn't show anything. Could someone instruct me how to delete offline partitions manually in detail? Thanks.
hi, I download latest python swagger manager api, but not found api keys for:get current resource pools config: GET cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues add resource pool, and reconfigure old: POST /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues/root get scheduler configs: GET /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/configs reconfigure pool scheduler: PUT /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/configs reconfigure pool parameters:PUT /cmf/clusters/cluster-name/queue-manager-api/api/v1/environments/dev/clusters/cluster-name/resources/scheduler/partitions/default/queues/root.pool_name please add it, or update
Hi All,After the integration of AD with kerberos we are not able to start the RM due the "User impersonate" issue. [Error: User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM] Loggers for RM:2021-04-21 16:10:08,878 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in state STARTEDorg.apache.hadoop.service.ServiceStateException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COMat org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.hadoop.ser
Hi, I am new to Nifi, so please excuse if this is a noob question. I have set up a 5 node Nifi cluster on GCP and using LDAP for authentication. I generated self-signed certs using the toolkit and I am able to login and access the UI at https://<hostname>:8443/nifi/. Everything looks good with this setup. Now, I added a HTTPS load balancer that routes the requests to backend Nifi nodes, when I try to access the Nifi UI at https://<load-balancer>/nifi/ it throws below error INFO [NiFi Web Server-251] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started xx.xx.xxx.x [<anonymous>] GET https://nifi.xxxxx.com/nifi-api/flow/current-userWARN [NiFi Web Server-251] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed xx.xx.xxx.x GET https://nifi.xxxxx.com/nifi-api/flow/current-user [Anonymous authentication has not been configured.] I generated the certificate using below command ./tls-toolkit.sh standalone -n 'nifixxxxxx[1-5].xxxx.c
In Apache Nifi, there are connections between each processors, which acts like queue of FlowFiles, and Nifi by default persists data content of FlowFile on disk. Does it mean each of such connection persists FlowFiles on disk? If that were true, each time of delivery of FlowFiles from one processor to another would mean one disk read and write, thus more processors would lead to more disk reads and writes, which in turn would lower the entire throughput. Is my understanding correct? and what is the best practice to avoid it, writing all things in one processor? Thanks.
Hello Everyone,We are a leading bank and have implemented Cloudera in our on-premises environment. Our management has taken a decision to implement Kubernetes in future so I would like to know which Cloudera services supports Kubernetes and which services does not. Appreciate if someone can help me in this regard.
When we restart or start kudu tablet server component through cloudera manager, it takes a long time to communicate with kudu master, it takes more than 10 minutes. why does this occur? when we start the kudu tablet server component and follow the log, it keeps loading the lines as shown in the screenshot below:
Hi community, I am a newbie at Nifi and unfortunately i have to build 3 node securedcluster and nifi registry on my own. The company had a standalone nifi node which is connected to nifi registry. I wanted from network team to clone this standalone nifi node and wanted to build 3 node cluster with them only adjusting configurations. Lets say the standalone node was server1 and I also have server2,server3(clones of server1) at the moment. My plan was build 2 node cluster with clone servers first because i didn't want to malfunction the original server while i was trying to configure cluster. If i succeed 2 node cluster without a problem then i will add the original server to cluster. The certificates of server2 and server3 didn't created by nifi-toolkit it was provided by another team in the company and i used them in keystore and truststores for this servers. I thought the configurations mostly correct because the automated process on the original server also works on clone n
How do you connect to a CDP Data Hub NiFi Registry from a local machine?
I'm new to hive trino. I would like to concatenate three text columns with "_" as separator. So I wrote following code select A.*,B.Name_group,mail_group,Occupation_group,CONCAT_WS("_",mail_group,Occupation_group,Name_group) combined_error from hive.employee.Job A left join hive.employee.Name B on A.Name_id=B.Name_id left join hive.employee.mail C on A.mail_id= C.mail_id left join hive.employee.Occupation D on A.Occupation_id=D.Occupation_id But I got following error message Colum '_' cannot be resolved. Can you help me to fix the issue
Hi All, Good Morning! Please help me in removing Duplicate Records without using cache services (Apache NIFI)Error Log
Can't start datanode -----------------------------------------------------------------------------------------------------------------------------------STARTUP_MSG: java = 1.8.0_312************************************************************/2022-05-03 08:42:09,494 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]2022-05-03 08:42:09,809 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMainjava.io.IOException: Running in secure mode, but config doesn't have a keytabat org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:210)at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2259)at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2308)at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2485)at org.apache.hadoop.hdfs.server.datanode.Data
there are like 1k finished jobs and i want to check allocated CPU and memory of those jobs. how to do that? i tried in command line but it shows when i enter job id and in my yarn UI CPU and memory metrics are not there.
We specified the environment variables as below: export AWS_ACCESS_KEY_ID=xMK6bdX8iY**************************************export AWS_SECRET_KEY=34*************************************** After connecting to a hive session , we specified the s3a credentials: set fs.s3a.endpoint=cluster.domain.*;set fs.s3a.access.key=$$$$$$$$$$$$$$###;set fs.s3a.secret.key=####$$$$; Tried to create a table using the below query (with directory location in s3 bucket : (s3a://test/dir2/) and received the preceding error ; even though the s3 credentials were already specified as stated above: 0: jdbc:hive2://> CREATE EXTERNAL TABLE s3dir (. . . . . . . . > col1 int,. . . . . . . . > col2 string,. . . . . . . . > col3 string,. . . . . . . . > col4 string. . . . . . . . > ). . . . . . . . > ROW FORMAT DELIMITED FIELDS TERMINATED BY ','. . . . . . . . > LOCATION 's3a://test/dir2/'. . . . . . . . > TBLPROPERTIES (. . . . . . . . > "s3select.format" = "csv".
Hi,Please, is there a way to reference a key part of JSON file that can come in different structures.For example i want to get the value of "requestUid" from these different JSON structures just with a single processor. So, irrespective of JSON formats or structures sent, i would want to get the value of requestUid.Thanks.For example:1){"responseMetaData": {"type": "ACK","userId": "SYSTEM","requestUid": "1742963e-f02f-4db6-88e0-6af8c4d4825b",,"systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","responseMessage": "","execTime": "1216"}} 2){"responseMetaData": {"type": "ACK","sendTime": "23/03/2019 09:16:25.305277","timeZone": "4","userId": "SYSTEM","branchCompanyId": "784100","branchLocationId": "784101","systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","requestUid": "yywhhnan56289200200","execTime": "1216"}}
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.