Solved

Ranger Group level authorization for NIFI is not working whereas user-level it's working do we need to enable or integrate underlying servers to AD/LDAP in order to work?

@stevenmatisonis it possible to apply filter condition on invoke HTTP processor that give all the alarm from web app in every 5 min but will not fetch the duplicate alarm that already fetched before ? or else is it possible when the alarm will come in to my web app then workflow in NIFI will trigger and gives recent alarm which came just in 10 min ...i have tried with DetectDuplicate processor ...but not getting any result ..i think i made mistake in configuration in detecduplicate processor.. please direct me ....and if you have any other route for this please provide me the whole way(configuration an all) ..i'm new in nifi ..don't know much more about that  detectduplicate processorwhen i clicked in arrow in side of RedisDistributedMapCacheClintService ...this below config has open(RedisConnectionPoolService)..and in connection string option i have put web address with port number is it wrong ..or else we have to put nifi localhost address with port no.??  

Hi all,    I have reinstalled ambari and trying to setup a three-node cluster. However, I keep getting the same error on the last step about multiple ambari-hdp-repos-1-2 and now 51. I am running centos7 and the install is being done offline through local repositories.    I know this has been covered in another thread but the answer is not very clear to me. My HDP, HDP-UTILS and SOLR repo urls are correct. A   Any help please??? Thank you!     2020-04-15 15:50:40,562 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=None -> 2.62020-04-15 15:50:40,566 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf2020-04-15 15:50:40,567 - Group['livy'] {}2020-04-15 15:50:40,609 - Adding group Group['livy']2020-04-15 15:50:42,177 - Group['spark'] {}2020-04-15 15:50:42,177 - Adding group Group['spark']2020-04-15 15:50:42,200 - Group['solr'] {}2020-04-15 15:50:42,201 - Adding group Group['solr']2020-04-15 15:50:42,229 -

Hello, I want to create a Case in the Cloudera support Portal. Even after finishing registeration, it redirects me this page belowCan someone help me please?

Hello All,     My Ranger DB is corrupted and I have no backup to restore it. Can I drop the corrupt db and recreate a new db for Ranger. This is for my Kerberized cluster in my test environment.   Any advice.

 I am working on a simple code to get a row from HBase which is not working through HBase Java API but the same row is retrievable from the shell.Example row Ids: F\x0A@z\x11\x0D\xB4\xD3\x05"S273320039#999279720=\x0A@z\x1AI\xA4\xD8h\x12\x09754427158#999598625F\x0A@z\x11\x0D\xA6@\x0C\x04\xDA490229991#999279958M\x0A@z\x13\x092\x89a\x12\x111059947933#999223281\x00\x0A@E\xFFn[\x18\x9F\xD4-1447846881#5241968320Result from one of the RowId through shell:   hbase(main):003:0> get 'OSMNodes_z3_geometry_ingestionTimestamp_v6',"\x00\x0A@E\xFFn[\x18\x9F\xD4-1447846881#5241968320"COLUMN CELL d: timestamp=1588532276719, value=\x03\x00\x06\x02\x00\x12\x00-\x005\x00=\x00R\x00Y\x01x\x00\x00\x0 0\x00\x01\x08\x03\xC0V\xE9\x9AU\x81;\xD5\xBF\xF5\xA7FE^\xAE\xE2\x7F\xF8\x00\x00\x00\x00\x00\x00\

Hello All, I had a DE575 exam scheduled for Feb 2020 but, due to the platform unavailability, I could not do it.I was expecting a mail from cloudera when the platform is live again. But I havent received any communication from Cloudera ever since. Recently I heard that the Spark(CCA 175) exam is available in the upgraded platform, but no news on DE575.Anyway, out of curiosity I checked in examslocal to see  if it allows me to reschedule the DE575 and it did. I just wanted to make sure that the exam (DE575) is really available .Does anyone have any clue. Thanks a lot!

Hi Team, Can any one please help me,how to download CDP Data Center Trial Download in Windows?My Doubt is --> Is it possible to download in windows and can we work on windows? Regards,Siva. 

how to disable the download button in hue interface?  I've tried to edit the hue.ini but it is not taking effect.

I have a zip folder stored in HDFS containing multiple video files. I want to unzip  the folder in HDFS and want to access individual video file. Can someone help me to find the solution for this?

Hi guys,  I want the Hive table created by sqoop to have two partitions: One for 2019 and another for 2020. But for some reason it is putting "2019,2020" in the year column. Command: sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/user/lrm0613/mydb2.jceks \--connection-manager org.apache.sqoop.manager.SQLServerManager \--driver net.sourceforge.jtds.jdbc.Driver \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Reporting' \--username 'lrm0613' \--password-alias sqlserver2.password \--query 'select jobhistoryid, requeueid, jobid,name,event,eventtime,submittime,starttime,service,year(EventTime) as year from JobHistory where year(EventTime)=2019 or year(EventTime)=2020 and $CONDITIONS' \--hcatalog-database dataengsandbox \--hcatalog-table JobHistoryPartitioned \--hive-partition-key year \--hive-partition-value 2019,2020 \--compress \--compression-codec snappy \--map-column-hive jobid=int \-m 1 \-

Previously, the recommendation for vm.swappiness was always to set it to 0. I have seen some posts stating that it changed. Since when and what was the reasoning?

I was looking at setting up Impala in a container from one of the docker images on docker hub and noticed it pulls in the bitcoin repo in the docker file: https://hub.docker.com/r/cloudera/impala-dev/dockerfile I just wanted to check if this was a legitimate dependency and an official cloudera user publishing the images.

Hi Team What is the stable release in CDP, which version we can prefer in CDP to set up for our new data lake? Please share a useful link(s) to check the stable release in CDH and CDP ThanksBalu D

Hi,   Nifi 1.9.2, single node, single cluster.   I am using 5 ExecuteSql processor to bulk collect records from Oracle DB. (each select query suppose to pick separate set of one fifth of total volume, logic is based on MOD function). Predecessor for all the executesql processors is same (generateflowfile processor).   ExecuteSql processors are configured as ("TIMER Driven", concurrent task=40, run schedule=30 sec, "maximum rows per flowfile"=500 and "output batch size"=1). We are starting the nifi flow, then generateflowfile processor is generating one flow file which is triggering executeSql processors.   But randomly in some of the runs, data is getting duplicated.    Not sure if this is the issue with concurrent tasks or timer driven property. Please help.

Hi,I have a flowfile json request which looks like this. Actually flowfile is a csv file which I am converting to json first so that another json request can be created for invokehttp processor. But if this csv to json conversion is not required for it then I would prefer to keep it csv only.{"type": "record","name": "Location","fields": [{"name": "Company_name","type": ["null", "string"]},{"name": "City","type": ["null", "string"]},{"name": "Country","type": ["null", "string"]},{"name": "Latitude","type": ["null", "string"]},{"name": "Longitude","type": ["null", "string"]}]}I need to send a request to invokehttp processor which look like below - {  "Code":"A-5678",  "Latitude":"78.495603",  "Longitude":"5.969756",  "Continent":""}Latitude and longitude is to be taken from flowfile fields while Code and Continent (To be sent as blank string) are not available in flowfile and has to be created. How can I create this custom request. I tried ev

Hi Team ,  I need some advice on how to tune our Nifi Cluster. We have a 3 Node Nifi Cluster that processes events in a real time streaming fashion which means our flows are running 24 by 7 . Each of the Nifi Linux servers have a RAM of 31GB. We have set the JVM memory setting in the bootstrap conf file to 28 G.  #JVM memory Settingsjava.arg.2=-Xms28Gjava.arg.3=-Xms28G However when I run a free -h command on the servers this is what I get which clearly shows that the used Memory is 30 G.   As a result of this , we keep getting incidents raised from the Linux team stating that the Nifi servers are always hovering over the  95 % mark and on top of that if any processor in the flow starts misbehaving , it shoots up to 100 percent and the server goes unresponsive.  So my question is , even after setting the threshold to 28 G in the bootstrap conf file , why do we still see Nifi taking up almost all the memory allocated to the serv

I tried to find out on myself, but after days of trying I decided that I need help. My usecase is to concat splitted zip-files (FilaA.zip.001, 002, 003 - 00x) and  I don't really know how many files will be delivered.  The files  are big and delivery takes time... The last file  to be delivered is a notification-file "ready.txt" My current workflow is like this:- receive alle files-extract   number of zip-file   (001-x) and build a priority (999-number of zip-file) of it-move to a queue with a PriorityAttributePrioritizer-extract max number of files (i.e. 7 from 007 - the file with priority 992)-move files via  notify processer to the wait processor- in the wait processor I want to wait until the notfication-file  arrived and  deliver all the files in wait-queue-after the the files will be concatenated  with the MergeContent processor and unzipped. My configuration works only one time.  ... The second delivery w

Hi, Following the instruction described on this page:https://docs.cloudera.com/documentation/enterprise/6/latest/topics/installation.html I made some good progress until up to step 6:  Install CDH and Other Software:https://docs.cloudera.com/documentation/enterprise/6/latest/topics/install_software_cm_wizard.html Now I encountered error. Please see the picture below: I completely wiped out the databases, retried twice, and still seeing the error. Could you advise where to check to fix the trouble, and move forward? Thank you! 

I'm able to create a System DSN using the "ODBC Data Source Administrator" with a successfull test.My Authentication mecanism is "User Name and Passowrd"SASL checkedSSL Options: "Enable SSL" checkedSSL Options: I'm providing the path to a certificate BUT, trying to connect using pyodbc with the same parameters (server, port, username and password) gives always the same error below whatever the auth_mechanism provided:pyodbc.connect("DRIVER={Cloudera ODBC Driver for Impala};\host={server_address};\port=21051;\SSL=1;\auth_mechanism=GSSAPI;\user=myusername;\password=mypassword;\trustedcerts=cert.pem;\") Error: ('HY000', u'[HY000] [Cloudera][DriverSupport] (1170) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct. (1170) (SQLDriverConnect); [HY000] [Cloudera][DriverSupport] (1170) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct. (1170)')

Hi, do I need to install or enable kerberos first in my CDH 6.3.x cluster before I can use Sentry Column Level security feature?

Is it okay to do some clean up on mysql database of Cloudera navigator Audit?

Hi There,   I am a big data developer but we are not using any streaming concepts in the project. So, I don't have much experience in streaming technologies like Flume or Kafka. But, I am planning for CCA 175 Spark and Hadoop developer certification. Should I have knowledge in Flume or Kafka to attend CCA 175? Please suggest.   Thank you!   Regards, Ganesh B

How do I enable further debugging on cloudera-scm-agents?   I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:   [07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeatself.cfg.max_cert_depth)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__self.conn.connect()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connectsock.connect((self.host, self.port))File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connectret = self.connect_ssl()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_sslreturn m2.ssl_connect(self.ssl, self._timeout

Hello, I have a CSV input similar to this, which I need in JSON. Name der Abteilung (DepartmentName),Name des Kontos (AccountName),ID des Kontobesitzers (AccountOwnerId),Zusätzliche Informationen (AdditionalInfo)ExampleDepartment,,dnajas93477ajdn,"{ ""UsageType"": ""ComputeHR"", ""ImageType"": ""Windows Server BYOL""}"  In the CSV there is a nice JSON embedded. I use an Avro Schema and the CSV Reader controller service to get rid of the human readable localized header and to read the the data. The Avro Schema currently looks like this:  { "type" : "record", "name" : "DailyCSV", "fields" : [ { "name" : "DepartmentName" , "type" : ["string", "null"] }, { "name" : "AccountName" , "type" : ["string", "null"] }, { "name" : "AccountOwnerId", "type" : ["string", "null"] }, { "name" : "AdditionalInfo", "type" : [ "null", "string" ] }]}  This does reads the CSV nicely, however, the embedded JSON will be left as string, resulting