Q&A

how to disable the download button in hue interface?  I've tried to edit the hue.ini but it is not taking effect.

I have a zip folder stored in HDFS containing multiple video files. I want to unzip  the folder in HDFS and want to access individual video file. Can someone help me to find the solution for this?

Hi guys,  I want the Hive table created by sqoop to have two partitions: One for 2019 and another for 2020. But for some reason it is putting "2019,2020" in the year column. Command: sqoop import \-Dhadoop.security.credential.provider.path=jceks://hdfs/user/lrm0613/mydb2.jceks \--connection-manager org.apache.sqoop.manager.SQLServerManager \--driver net.sourceforge.jtds.jdbc.Driver \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Reporting' \--username 'lrm0613' \--password-alias sqlserver2.password \--query 'select jobhistoryid, requeueid, jobid,name,event,eventtime,submittime,starttime,service,year(EventTime) as year from JobHistory where year(EventTime)=2019 or year(EventTime)=2020 and $CONDITIONS' \--hcatalog-database dataengsandbox \--hcatalog-table JobHistoryPartitioned \--hive-partition-key year \--hive-partition-value 2019,2020 \--compress \--compression-codec snappy \--map-column-hive jobid=int \-m 1 \-

Previously, the recommendation for vm.swappiness was always to set it to 0. I have seen some posts stating that it changed. Since when and what was the reasoning?

I was looking at setting up Impala in a container from one of the docker images on docker hub and noticed it pulls in the bitcoin repo in the docker file: https://hub.docker.com/r/cloudera/impala-dev/dockerfile I just wanted to check if this was a legitimate dependency and an official cloudera user publishing the images.

Hi Team What is the stable release in CDP, which version we can prefer in CDP to set up for our new data lake? Please share a useful link(s) to check the stable release in CDH and CDP ThanksBalu D

Hi,   Nifi 1.9.2, single node, single cluster.   I am using 5 ExecuteSql processor to bulk collect records from Oracle DB. (each select query suppose to pick separate set of one fifth of total volume, logic is based on MOD function). Predecessor for all the executesql processors is same (generateflowfile processor).   ExecuteSql processors are configured as ("TIMER Driven", concurrent task=40, run schedule=30 sec, "maximum rows per flowfile"=500 and "output batch size"=1). We are starting the nifi flow, then generateflowfile processor is generating one flow file which is triggering executeSql processors.   But randomly in some of the runs, data is getting duplicated.    Not sure if this is the issue with concurrent tasks or timer driven property. Please help.

Hi,I have a flowfile json request which looks like this. Actually flowfile is a csv file which I am converting to json first so that another json request can be created for invokehttp processor. But if this csv to json conversion is not required for it then I would prefer to keep it csv only.{"type": "record","name": "Location","fields": [{"name": "Company_name","type": ["null", "string"]},{"name": "City","type": ["null", "string"]},{"name": "Country","type": ["null", "string"]},{"name": "Latitude","type": ["null", "string"]},{"name": "Longitude","type": ["null", "string"]}]}I need to send a request to invokehttp processor which look like below - {  "Code":"A-5678",  "Latitude":"78.495603",  "Longitude":"5.969756",  "Continent":""}Latitude and longitude is to be taken from flowfile fields while Code and Continent (To be sent as blank string) are not available in flowfile and has to be created. How can I create this custom request. I tried ev

Hi Team ,  I need some advice on how to tune our Nifi Cluster. We have a 3 Node Nifi Cluster that processes events in a real time streaming fashion which means our flows are running 24 by 7 . Each of the Nifi Linux servers have a RAM of 31GB. We have set the JVM memory setting in the bootstrap conf file to 28 G.  #JVM memory Settingsjava.arg.2=-Xms28Gjava.arg.3=-Xms28G However when I run a free -h command on the servers this is what I get which clearly shows that the used Memory is 30 G.   As a result of this , we keep getting incidents raised from the Linux team stating that the Nifi servers are always hovering over the  95 % mark and on top of that if any processor in the flow starts misbehaving , it shoots up to 100 percent and the server goes unresponsive.  So my question is , even after setting the threshold to 28 G in the bootstrap conf file , why do we still see Nifi taking up almost all the memory allocated to the serv

I tried to find out on myself, but after days of trying I decided that I need help. My usecase is to concat splitted zip-files (FilaA.zip.001, 002, 003 - 00x) and  I don't really know how many files will be delivered.  The files  are big and delivery takes time... The last file  to be delivered is a notification-file "ready.txt" My current workflow is like this:- receive alle files-extract   number of zip-file   (001-x) and build a priority (999-number of zip-file) of it-move to a queue with a PriorityAttributePrioritizer-extract max number of files (i.e. 7 from 007 - the file with priority 992)-move files via  notify processer to the wait processor- in the wait processor I want to wait until the notfication-file  arrived and  deliver all the files in wait-queue-after the the files will be concatenated  with the MergeContent processor and unzipped. My configuration works only one time.  ... The second delivery w

Hi, Following the instruction described on this page:https://docs.cloudera.com/documentation/enterprise/6/latest/topics/installation.html I made some good progress until up to step 6:  Install CDH and Other Software:https://docs.cloudera.com/documentation/enterprise/6/latest/topics/install_software_cm_wizard.html Now I encountered error. Please see the picture below: I completely wiped out the databases, retried twice, and still seeing the error. Could you advise where to check to fix the trouble, and move forward? Thank you! 

I'm able to create a System DSN using the "ODBC Data Source Administrator" with a successfull test.My Authentication mecanism is "User Name and Passowrd"SASL checkedSSL Options: "Enable SSL" checkedSSL Options: I'm providing the path to a certificate BUT, trying to connect using pyodbc with the same parameters (server, port, username and password) gives always the same error below whatever the auth_mechanism provided:pyodbc.connect("DRIVER={Cloudera ODBC Driver for Impala};\host={server_address};\port=21051;\SSL=1;\auth_mechanism=GSSAPI;\user=myusername;\password=mypassword;\trustedcerts=cert.pem;\") Error: ('HY000', u'[HY000] [Cloudera][DriverSupport] (1170) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct. (1170) (SQLDriverConnect); [HY000] [Cloudera][DriverSupport] (1170) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct. (1170)')

Hi, do I need to install or enable kerberos first in my CDH 6.3.x cluster before I can use Sentry Column Level security feature?

Is it okay to do some clean up on mysql database of Cloudera navigator Audit?

Hi There,   I am a big data developer but we are not using any streaming concepts in the project. So, I don't have much experience in streaming technologies like Flume or Kafka. But, I am planning for CCA 175 Spark and Hadoop developer certification. Should I have knowledge in Flume or Kafka to attend CCA 175? Please suggest.   Thank you!   Regards, Ganesh B

How do I enable further debugging on cloudera-scm-agents?   I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:   [07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeatself.cfg.max_cert_depth)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__self.conn.connect()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connectsock.connect((self.host, self.port))File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connectret = self.connect_ssl()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_sslreturn m2.ssl_connect(self.ssl, self._timeout

Hello, I have a CSV input similar to this, which I need in JSON. Name der Abteilung (DepartmentName),Name des Kontos (AccountName),ID des Kontobesitzers (AccountOwnerId),Zusätzliche Informationen (AdditionalInfo)ExampleDepartment,,dnajas93477ajdn,"{ ""UsageType"": ""ComputeHR"", ""ImageType"": ""Windows Server BYOL""}"  In the CSV there is a nice JSON embedded. I use an Avro Schema and the CSV Reader controller service to get rid of the human readable localized header and to read the the data. The Avro Schema currently looks like this:  { "type" : "record", "name" : "DailyCSV", "fields" : [ { "name" : "DepartmentName" , "type" : ["string", "null"] }, { "name" : "AccountName" , "type" : ["string", "null"] }, { "name" : "AccountOwnerId", "type" : ["string", "null"] }, { "name" : "AdditionalInfo", "type" : [ "null", "string" ] }]}  This does reads the CSV nicely, however, the embedded JSON will be left as string, resulting

Hello,tdWhenever i add hive service into my cluster it always fails to start.following is the stderr and stdout.Can someone help me resolve this issue?Thank you in advance.stderr: Traceback (most recent call last):File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restartself.status(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in statuscheck_process_status(status_params.hive_metastore_pid)File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_statusraise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module>HiveMetastore().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/scr

1、Config values: 2、Actual Values: 3、Appreciate any advice  🙂

Hi Friends,I have mongo query which running perfectly fine from mongo shell.b.test650.aggregate([{$group: {"_id": {X: "$X",Y_DT: "$Y_DT",Z: "$Z"},adj: {$sum: "$adj" },bjc: {$sum: "$bjc" },jbc: {$sum: "$jbc" },mnk: {$sum: "$mnk"}}}]) The same query when in ran from Nifi , RunMangoAggregation  throwing out error, though the mongo aggregation query changed to json type query ...{"$group": {"_id": {"X": "$X","Y_DT": "$Y_DT","Z": "$Z"},"adj": {"$sum": "$adj" },"bjc": {"$sum": "$bjc" },"jbc": {"$sum": "$jbc" },"mnk": {"$sum": "$mnk"}}} Getting following error ..error run mongodb aggregation query.: com.fasterxml.jackson.databind.exc.MismatchedInputException:canot deserialize instace of java.util.ArayList' out of START_OBJECT token   Nifi workflow.. Processor(runMangoAggregation) configuration What is the change I need to do in json query which supposed to executed @ runMongoAggregation  processor?

Hello everybody,the title explains pretty much the annoying issue I'm facing. I have to install the JCE policy files, but when I click on the button there is no actual response from Cloudera Manager. It keeps loading the wizard page but it doesn't go on. I know I should be redirected on the wizard page. I tried to restart the cm-server process, the cms, the cm-agent with no result. Also I can't find anything inside the cm-server and cm-agent logs. I'm using Cloudera Express 6.3.0. I faced this problem another time in the past, but in that case it magically autosolved: I was tired of this annoying issue and I started doing some other work and left the tab open. At a certain point it loaded the wizard page. But this time I can't wait too much since I  have some deadlines to deal with and honestly more than three hours have passed now. Any help would be appreciated, thanks.

 I need a Help i have two question please 1-how I can transform the data from apache kafka to hdfs????2-how I can transform the data from apache kafka to SparkStreming???? thank you

Is the namespace field in an AVRO schema required?  Does the namespace first have to be created if you provide it in the schema?  Not sure how this works. Thanks, johndcal

Hi Community members, We have some maintenance on a host. And want to bring down a host for an hour or so.Got the below document:https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cm_mc_host_maint.html So, I want to ask, what is the proper way to proceed with it:- Select the host --> Stop all roles on the host and then bringing down cloudera-scm-agent on the host and after maintenance bring those upor as the document mentions:- Select the host -->Decommission Host(s)-->Take DataNode Offline and after maintenance Recommission Host(s).-->Bring hosts online and start all roles Please suggest what is the difference in these two approaches and which one is best in which scenario?  

Dear Cloudera Community, Before installing the trial version of CDP Data Center,we must make sure :"The Cloudera Manager Server host must have uniform SSH access on the same port to all hosts " What does it means and how to do that ? Regards,BSST