Q&A

Hello all, I have installed Apache Knox on a CDP 7.1.6 cluster and switched Shiro from PAM to LDAP (as described here https://is.gd/FmexUD). The changes are also done in the providers. PAM is disabled via the authentication.param.remove switch. Nevertheless PAM (KnoxPamRealm) is used for authentication instead of LDAP (KnoxLdapRealm). Does anyone have useful hints where to look for the cause? Thanks a lot! Regards,Daniel 

Hi all, My company is now moving work to CDSW under Citrix Gateway.As my local computer is not connected to the cloud, there are many troubles like file transfer, copy and paste texts which make me a suffering life. I would like to ask for help for sending email with attachment in CDSW under Citrix Gateway.There is no outlook in my Citrix Gateway. I am not sure can it get done. Does anyone has the python script that can help?Many thanks in advance.

Hi All, Thanks in Adv... Wanted top some POC, for that i want install CDH or HDP hadoop cluster on AWS. Now after merging new product is : CDP Can i still install HDP 3.1  with ambari  as before without any license issue. if yes kindly help the link to downloads source. If not can i install CDP without license or trail versions. if kindly provide links to download and installation document as i never installed CDP. Thanks,Suryakant  

Hi All, Before I have worked in CDH 5.X,Could you please let me know, is there any CDP sanbox/quickstart VM like as Hortenworks sand-box, CDH quick start VMPlease provide the link do download and configure , if Cloudera is providing CDP sanbox/quickstart VM

Hello everyone, i have a problem i set from Ambari those values:yarn.nodemanager.resource.memory-mb = 7820yarn.scheduler.minimum-allocation-mb = 1024yarn.scheduler.maximum-allocation-mb = 7820I restart Yarn but the same error keeps coming, why i can't have a bigger AM memory than 3072? Where this 3072 MB comes from?? Should i edit some other settings, cause i cant found any?Thanks

We tried with root user and with test user also, but still getting the above error. We also did below config change for root usersudo vi /etc/ssh/sshd_configSearch and change the below lines as below,PermitRootLogin yessudo systemctl restart sshd For test user, we did below config changesudo vi /etc/sudoersadded below lineec2-user ALL=(ALL)  ALLHowever, with doing the the above changes we are not able to login with both root and test user. Can you please help us on this

Hi Guys,   I'm currently having an issue where a schema that I'm using is erroring out in a RecordLookup processor ( with a Rest Service ). The message I get is the one in the title of this post, "Can't redefine org.apache.nifi.dataType". Below you can find the schema that I'm using.             { "type": "record", "name": "FullTicketData", "fields": [ { "name" : "terminal", "type": { "type": "record", "name" : "terminal_array", "fields": [ { "name": "success", "type": "string" }, { "name": "data", "type": { "type": "array", "items": { "name": "data_array", "type": "record", "fields": [ { "name": "ID", "type": "int" } ] } } }, { "name": "

Hi,   I have the following Json. I would like to create a Jolt Spec to remove all nulls from all levels. How can I do that? Any help would be appreciated. Thanks {"ID": "c22b657e-227b-4c65-e6f8-08d88a3e1118","Name": "SomeName","TaggedItemName": "SomeValue","ITRName": "SomeValue","TestReference": null,"JobCardName": null,"Comments": "test","PrimaryHandoverName": null,"SecondaryHandoverNumber": null,"CertificationGroupingName": null,"DocumentCode": "SomeValue","AssignedToName": "SomeName","DocumentReference": null,"TagITRCompletionStatusName": "Rejected","ScheduleRevision": null,"Completed": {"SignOff_AuthP": "John","SignOff_Date": null},"Accepted": {"SignOff_AuthP": "Smith","SignOff_Date": null},"Approved": {"SignOff_AuthP": "Ali","SignOff_Date": null},"ScheduleGroup": null,"DownloadUri": "SomeUrl","UpdatedDate": "2022-01-18T11:36:33.9057626Z","CreatedDate": "2020-11-30T10:57:20.8534287Z"}

Trying to determine if HDF/HDP is a valid choice going forward.Using this document (https://www.cloudera.com/downloads/paywall-expansion.html), it seems like these legacy tools require to an active subscription to download the binaries (as well as get support like bug-fixes and such). My question is if I set up a HDF/HDP cluster while I have an active subscription, then subscription lapses, what happens? Do I need to shutdown the cluster? Can I spin up local hosts and add them to a pre-existing cluster? What are the limitations? What about older clusters before this paywall was in place?EDIT: Adding this link here: https://www.cloudera.com/products/pricing/pricing-update.html"Subscription for binary accessA key part of adopting the Red Hat model was that access to enterprise software binaries requires an active subscription. Cloudera requires a subscription to any compute and storage capacity assigned to run Cloudera software. All Cloudera software version releases, maintenan

Hi, I have installed Ranger in my CDP cluster with RangerAdmin, Tagsync and Usersync instances. I have a problem when trying to access the admin web UI (6080) with the credentials admin:admin or ranger:ranger.I get "Unable to connect to DB" on the interface and  WARN com.mchange.v2.resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@2bf8ec30 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.WARN com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@781133d5 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:org.postgresql.util.PSQLException: FATAL: password authentication failed for user "ranger" in the ran

Hi,   i am having the error message in the title, but I am stuck as I have already checked the followings:   * ranger permissions: cm_s3 / all - bucket, path -- the user in the the list for read / write / all permissions * IDBroker role: user has a cdp-datalake-admin-role, which has a cdp-datalake-admin-policy-s3access   any other idea and what to check? thanks  

For security reasons I need to encrypt a column before putting the data in the repository. How can I do this using python or some processor. I tried to use the cryptography library but without success. If you can guide me which is the best way I would be very grateful.

I'm using ExecuteSQL,SplitAvro,ConvertAvroToJSON,EvaluateJsonPath,ReplaceText,ExecuteSQL.  I am trying to replace the content in the flowfile using replaceText processor.Now, i can replace like this. -> INSERT INTO values (${id},'${name}') . ReplaceText process like this : INSERT into x values(1,'xx')INSERT into x values(2,'yy')INSERT into x values(3,'zz')I'm sending INSERT query for each line. But i want to send executesql process like thisINSERT INTO x  values (1,'xx'),(2,'yy'),(3,'zz')   

Hi,I am provisionning a CDP 7.1.5 cluster with CM 7.2.4  and encountered a connexion problem between the agents and cloudera manager server after tls setup through API.There is nothing in the server log and there is the following error in the agent log.Could you help me understand where the problem stands ? Activating further logging or doing some tests upon the provided certificates ?Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1430, in _send_heartbeatself.cfg.max_cert_depth)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 185, in __init__self.conn.connect()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connectsock.connect((self.host, self.port))File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connectret = self.connect_ssl()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/C

I have many zip files each containing a csv file. The header of the csv file contain fields and some field names have space between words e.g "field one" instead of "field_one" which is the one in the database.Please, what is the optima method to change the csv field names to match the ones in the database.NOTE: I am using MSSQL as the DB. And my NIFI implementation is1) GetFile : from a repository that contain zip files of very large size2)UnpackContent: to extract the csv from the zipped file3) PutDabaseRecord: to persist the csv data into db because of its large size.  Thank you for your kind help.

Hello Experts, I want to run a simple Hive Insert SQL statement in Nifi periodically for example:insert overwrite Table1 select * from Table2; All SQL values are fixed hardcoded and don't need to be changed dynamically in the flow. As a newbie I was thinking I could write the flow as: ReplaceText->success->PutHiveQL Search Value:  (?s)(^.*$)Replacement Value:  insert overwrite Table1 select * from Table2;  But I have an error in ReplaceText which probably needs an incoming flowfile which I don't have since the SQL is fixed hardcoded:"Upstream Connections is invalid because Processor requires an upstream connection but currently has none."  The other option I could try but not sure it will work is:GenerateFlowFile -> success -> PutHiveQLGenerateFlowFile Custom Text: insert overwrite Table1 select * from Table2;     

From Cloudera Manager it appears like Atlas Service is in good health, but it doesn't start at all. When I try to access the Web UI, it shows an Http Error 503 Service Unavailable but the application.log doesn't show any error; it just halts. Here is the application.log whe I restart Atlas service from Cloudera Manager:2022-01-31 12:04:28,657 INFO - [shutdown-hook-0:] ~ ==> Shutdown of Atlas (Atlas$1:63)2022-01-31 12:04:28,668 INFO - [shutdown-hook-0:] ~ <== Shutdown of Atlas (Atlas$1:69)2022-01-31 12:09:49,319 INFO - [main:] ~ Loading atlas-application.properties from file:/var/run/cloudera-scm-agent/process/6754-atlas-ATLAS_SERVER/conf/atlas-application.properties (ApplicationProperties:137)2022-01-31 12:09:49,327 INFO - [main:] ~ Using graphdb backend 'janus' (ApplicationProperties:317)2022-01-31 12:09:49,327 INFO - [main:] ~ Using storage backend 'hbase2' (ApplicationProperties:328)2022-01-31 12:09:49,327 INFO - [main:] ~ Using index backend 'solr' (ApplicationProperties

Error: Package: cloudera-manager-agent-7.4.4-15850731.el8.x86_64 (cloudera-manager)          Requires: libc.so.6(GLIBC_2.28)(64bit)this is centos 7 which comes with gcc 217 [root@cdp-master yum.repos.d]# cat /etc/centos-releaseCentOS Linux release 7.9.2009 (Core)I'm trying the install direct from cloudera. Thoughts?  It shouldnt be this type of issue

Hello,I have one big table in one database that I have to sink on destination.Firs I have flow with incremental insert and it works fine. Than in other flow I have incremental delete by ID column.I am trying to delete 3179958 records from table.  I have one table on source and I have to do increment delete on destination.  This is my flow. I am sending every processor separate:generatetablefetch:execute sql: putdatabaserecord - delete When I have much less records it all works fine. But now flow can't finished... Is there any other solution. Thanks in advance

I am facing an error during hiverserver2 start as "caught exception: ZooKeeper node /hiveserver2 is not ready yetand when I debug more I see that there is no hiveserver2 instance in zookeeper.    Welcome to ZooKeeper!2022-02-03 01:58:35,986 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)JLine support is enabled2022-02-03 01:58:36,072 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /127.0.0.1:59736, server: localhost/127.0.0.1:21812022-02-03 01:58:36,093 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x27ebd79aecc0141, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null[zk: localhost:2181(CONNECTED) 0] ls /hi

我刚加入CDH社区。如何申请archive.cloudera.com/cdh6 谢谢您的帮助！

I'm trying to use kafka data using pyspark but I having difficult because it's in Hashmap type The Question is, how can I convert this to a useful df to be treated in pyspark? This is the output and my actual code: This is the outputThis is my actual code Any suggestion and steps? 

We have file names that start with a dot (e.g. ".~tempfile.bin") that we are attempting to list with the ListFile processor.  We've tried setting the File Filter to an empty string, as well as various RegEx's (e.g. ".*") and it will not list file names starting with a dot.  As soon as we remove the starting dot from the file name, it lists the files.  Any ideas on how to list these files starting with a dot?

Hi everybody, i'm submitting jobs to a Yarn cluster via SparkLauncher.Im under HDP 3.1.4.0Now, i'd like to have only 1 executor for each job i run (since ofter i found 2 executor for each job) with the resources that i decide (of course if those resources are available in a machine).So i tried to add .setConf("spark.executor.instances", "1").setConf("spark.executor.cores", "3")But even if i set spark.executor.instaces to 1, i have 2 executors, do you know why? (I read somewhere that the N° of executors =  spark.executor.instances * spark.executor.cores .I don't know if that's true, but it seems true.Is there a way to achieve my goal of have MIN and MAX 1 executor for each job???Could be achieve with dynamicAllocation (i'd prefer not to set that since it's not designed for that and can do a lot of stuff that i don't need) ? Thanks in advance!!!

I have a Spark v2.4.5 Spark-Streaming application running on CDP 7.1.7 that listens to relatively small JSON messages from a Kafka topic, de-serializes them, and then writes to an appropriate unpartitioned Hive table with parquet records. This issue also occurred with CDH 5.16 and CDH 6.3.4.The code looks similar to this: def processStream(stream: DStream[ConsumerRecord[String, KafkaMessageWrapper]]): Unit = { stream.foreachRDD(rdd => { // Peek at the message to extract the event type and raw payload val rowRdd = rdd.flatMap(MessageProcessor.processRecord) val rootDF: DataFrame = rowRdd .toDF("topic", "partition", "offset", "eventName", "version", "payload") rootDF.createOrReplaceTempView("data") // Process only the USER records spark.sql(s"select eventName, version, payload from data where eventName = 'USER_CREATE'") .na.drop() // Map the records to the expected schema // This flatten the incoming JSON and does some validation /