Q&A

Is there any documentation on KafkaConnect?. I see lots of documentation and examples on Confluent. However, they use kafkacat and ksql.Where can I find documentation, examples for Cloudera?

Hi.I'm trying to utilize GetHDFSFIleInfo(not GetHDFS) to start after previous step.Actually, this is what I exactly want to do (https://community.cloudera.com/t5/Support-Questions/NiFi-fetchHDFS-without-ListHDFS/td-p/211708) flowstreamcommand -> GetHDFSFileInfo -> FetchHDFS -> PutSFTP I did make gethdfsfileinfo first, after then tried to get flow file info from previous job at FetchHDFS.However, the result of gethdfsfileinfo was only one attribute and I don't know how to fetch all files at fetchHDFS. This is an attribute of gethdfsfileinfo. I got an attribute as below and want to fetch those all files.filename : faaa~~~~~hdfs.count.dirs : 1hdfs.count.files : 44hdfs.full.tree : {"ojbectName";"gfk",""...., "content":[{"objectName":"Weekly_GfK_02_Merge_F_HP_AT.txt",".....]}hdfs.objectName : gfk....Could you please tell how to use it? or how to solve this problem? 

Hi,Is a CDH Enterprise license transferable from one cluster (deployed just for a proof-of-concept and after e.g. 3 months destroyed), to another cluster (deployed only after the former one was destroyed)?

Hi everyone, I am uploading Navigator logs into Hive for analysis. I used this: ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ("separatorChar" = ",","quoteChar" = '"',"escapeChar" = "\\")STORED AS TEXTFILE Based off: https://community.cloudera.com/t5/Support-Questions/Hive-escaping-field-delimiter-in-column-value/m-p/233346#M195173 to get an external table to handle the commas in queries. I think it's working fine (but will test tomorrow to make sure). However, I then do a move to a managed partitioned table, but it is not handling the commas in queries correctly. I created it as:PARTITIONED BY(event_day INT, event_month INT, event_year INT); How do I handle the commas in queries as part of this move?Does the managed table need the same delimter & escape info as the external table?If that's the case, what's the syntax for it? Schwifty

we have Hadoop cluster with only 2 data nodes machines in HDFS configuration we defined the Block replication to 3 soBlock replication=3is it OK? to defined Block replication=3 , when we have only two data nodes in the cluster? from my understanding when we defined Block replication=3 while we have 2 data nodes machines in HDFS cluster its means that one machine should have 2 replica .  and the other machine one replica , am I correct here?

I have a line by line list in my flowfile, which records the result of some file processing, somthing like the following; C:\Data\client1\order100.txt - SUCCESSC:\Data\client1\order122.txt - ERRORC:\Data\client1\order323.txt - SUCCESS I can't see a way to add this flowfile content to the e-mail MessageBody. I have therefore tried an ExtractText processor with message.body as an attribute and (.*) as the regex. This gets me close, but all the filenames are on the same line, so I end up with this; C:\Data\client1\order100.txt - SUCCESS C:\Data\client1\order122.txt - ERROR C:\Data\client1\order323.txt - SUCCESS Which is not quite the e-mail I'm looking for. Any sensible suggestions welcome.   

1. Setup Nifi 1.11.4 secure using my own certs and kerberos authentication accepting username/password.  All is working fine can log in using kerberos username/password as initial admin user. 2. Setup Nifi Registry 0.6.0 using same certs and kerberos all is fine. Again all is working fine can log in using kerberos username/password as initial admin user.Problem is that when adding registry to secure Nifi it seems they attempt to talk using the username 'anonymous'.If I add the user "anonymous" to Registry giving it permissions to proxy, then all works fine, however registry shows all changes were performed by 'anonymous' Also can't use policies to not allow the anonymous user access to the UI  I can't find any documentation on why I would need to create a user with name 'anonymous'In fact most documentation says when using https that anonymous access it not supported. Q. How does Nifi/Registry identify the identity of the host in standalone mode? If I gene

It told me to be a CDP Data Center customer. But can you tell me how to be ???  

Dear All,I need your suggestion huge number data transfer to Syslog through Apache NiFiMy aim is transferring all IIS logs (located on Azure Blob Storage) to syslog line by line. Therefore I am using split text for parsing lines. Extract Text is transferring split line on the attribute, in this way I can say to syslog processor "Message Body: IISHttp${msg}". I will explain details at the belowActually my problem is bottleneck on the Extract Text. I have to transfer IIS Logs near-real time (less than 1 hour delay). But Extract Text isn't draining the messages (flow files) from the queue in proper time. I tried increasing Thread Number, Run Duration, increasing/reducing Queue size but I couldn't achieve my target. The queue between split text and extract text are always full. I have log gap about 13 hours. I am trying find a way for thatMy flow is that;1. Getting IIS Log files from Azure Blob Storage1.1. List Azure Blob Storage Processor1.2. Route on Attribute Processor (I have date filt

Hello, I have a piece of JSON on which I want to apply an updateRecord processor using some filter capabilities. My JSON looks like this:   "status": [ { "code": "InProgress", "isCurrentStatus": false, "startDateTime": "2020-03-05T20:45:00Z" }, { "code": "Pending", "isCurrentStatus": true, "startDateTime": "2020-03-05T21:20:00Z" } ],   And my RecordPath expression is the following:   /status[0..-1][./isCurrentStatus = true]/code  This expression doesn't work and I have an error in my logs file stating about incorrect syntax:  Unexpected token ']' at line 1 If I put the following syntax:  /status[0..-1][./isCurrentStatus = "true"]/code  it doesn't give anymore error, but of course the filter doesn't work. So, I have the feeling that recordPath filter doesn't support the use of boolean values. My Nifi

noticed below error from users.log  but nothing changed from nifi configs but started noticing this error nifi-user.log:2020-07-02 12:16:13,857 INFO [NiFi Web Server-343673] o.a.n.w.a.c.ResourceNotFoundExceptionMapper org.apache.nifi.web.ResourceNotFoundException: Unable to find user with id '938eb61e-bbc4-383a-8475-aee80541b5a5'.. Returning Not Found response. @MattWho 

Hi, Trying to upload logs into a Hive external table. Some records contain queries which have multiple columns in them, but commas are also the field delimiters. Other questions here recommend using org.apache.hadoop.hive.serde2.OpenCSVSerde. So I used: ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'STORED AS TEXTFILELOCATION 'hdfs://hdfs/path/to/directory/external/2020Jun30'TBLPROPERTIES ('skip.header.line.count'='1'); This didn't deal with the additional non-escaped commas in the log file though. So I tried:ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES (   "separatorChar" = ",",   "quoteChar"     = '"',   "escapeChar"    = "\\")  STORED AS TEXTFILE However, in Hue I get this error when using org.apache.hadoop.hive.serde2.OpenCSVSerde:AnalysisException: Failed to load metadata for table: 'db.ext_table' CAUSED BY: TableLoadingException: F

Hi Team ,  I am having issues while connecting to mysql database using apache nifi.  The error is as followsCannot create PoolableConnectionFactory Please note that Apache Nifi and MySQL is installed on the same Linux server and hence there shouldn't be any problems with the 2 applications talking to each other.  The database connection pool settings is provided below The error when I try extracting the records is provided below  This is something very urgent as we are doing a POC and if this succeeds our Projects will be based on this. Any help in this regard will be much appreciated.  @Former Member @Shu 

Hi, Following the official documentation  I logged in to https://public-IP  Logged in using the Super User admin account. Post that I configured A/D integration and on the next page I added a few users/groups for admin say users1, user2, etc.. Post that it logged me out and asked me to login to the same page  https://<public-IP>  . Now I am trying to login using user1 and corresponding A/D password but I get and error - invalid username/password error. I tried with user2 also which was configured. I also tried using user1@realm.com  . To my surprise, I am not longer able to login using super admin also.  I tried to see the various container logs but no success. Has anyone faced similar issue?What am I missing here?  Thanks in advance.  Regards Tagging @SumitraMenon @Former Member @abajwa 

HI Team, I came across an scenario where HA failed when active namenode server was rebooted.Accidentally the master node which runs active namenode got rebooted.when it got rebooted i believed the standby namenode will become active namenode and will continue the operations but standby namenode did not became active and HDFS was completely down.So my question is1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster2. Whether the HA is expected to work only between the active and standby namenode daemons? Thanking in Advance!!

Environment HDP-3.1.0.0Hive3.0.0.3.1 Context I am trying to configure hive to use LDAP (AD). But hiveserver2 throws errors when restarted. Similar errors are visible when connecting to beeline. Configurationhive-site.xml (truncated)     <configuration xmlns:xi="http://www.w3.org/2001/XInclude"> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.authentication</name> <value>LDAP</value> </property> <property> <name>hive.server2.authentication.ldap.baseDN</name> <value>DC=MYDC,DC=MYDC</value> </property> <property> <name>hive.server2.authentication.ldap.Domain</name> <value>DOMAIN</value> </property> <property> <name>hive.server2.authen

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format./spark2-history/application_1574201808558_1228909.lz4Ambari Version : 2.6.0.0HDP Version : 2.6.3.0-235Spark Version : 2.xThank you.

Hi friends,Im in CDH 6.2 and hence trying to check jps command through cmd prompt.Not sure why it is not showing up hadoop services on issuing it and hence returning with below error [root@cloudera bin]# jpsbash: jps: command not found... Kindly help me out with this.Regards,GTA

I am trying to create a very basic Custom HiveMetastore Event Listener using the following blog. I have successfully created the JAR file and have executed the following steps on beeline but the event listener logs are still not getting displayed in /var/log/hive/hivemetastore.logFollowed Tutorial=> https://towardsdatascience.com/apache-hive-hooks-and-metastore-listeners-a-tale-of-your-metadata-903b751ee99fGit Repository=> https://github.com/mesmacosta/hive-custom-metastore-listenerStep 1: Added JAR to hdfshadoop fs -put custom-metastore-listener.jar /user/data/Step 2: Added JAR from HDFS to the hive beeline> add jar hdfs:///user/data/custom-metastore-listener.jar; Step3: Set event listenerbeeline> set hive.metastore.event.listeners=com.medium.hive.metastore.listerner.CustomListener;Step4: Run create table command and check hivemetastore logs.beeline> CREATE TABLE test(name string) STORED AS ORC;  But the log does not show anything from a custom

Hi,Would appreciate any advice, how to solve a problem with terminal access from a CDSW session. Let me highlight that I can launch a session, however within a session I am unable to access the terminal – please see the attached screenshot with HTTP ERROR 401.The networking requirements are met, in particular:IPv6 is enabledCDSW hosts are within the same subnet as the CDH clusterDNS is configured with the relevant A record for domain name, CNAME record for wildcard domain, and a reverse PTR domain record (please see the enclosed response to ping command, where the DNS resolves the terminal's FQDN to CDSW master node's IP)No iptables rules were enabledSElinux is disabled[cloud-user@cdh-control-01 ~]$ ping -c1 tty-jidv65sd8630btx4.cdsw.<intranetdomain>PING cdsw.<intranetdomain> (10.133.210.200) 56(84) bytes of data.64 bytes from cdsw.<intranetdomain> (10.133.210.200): icmp_seq=1 ttl=60 time=0.884 ms--- cdsw.<intranetdomain> ping statistics ---1 pack

Hi,I am looking for Cloudera CDP, Hive with Kerberos security and encryption to setup a basic development environment. What kind of license policy is required? Thanks,Tulasi

Hi All,   I am trying to generate separate UUID for JSON array of objects. I tried to use ${UUID()} in JOLT specification, but this is generating same UUID for all the elements.   e.g. Input: =============== {  "aliases": [    "Alias1",    " Alias2"   ]}   JOLT Specification ===================== [{"operation": "shift","spec": {"aliases": {"*": "alias[&0].name"}}},{"operation": "modify-default-beta","spec": {"alias": {"*": {"id": "${UUID()}"}}}}]Current Output: ===================== "alias" : [ {"name" : "Alias1","id" : "2948446b-6213-44ba-b81f-c4e5f7dab2eb"}, {"name" : " Alias2","id" : "2948446b-6213-44ba-b81f-c4e5f7dab2eb"} ]   Any pointers will be greatly appreciated.   Thanks, Biswa 

I have designed flow in nifi for XML to JSON converter. And for that, I have used GenerateFlow File and JoltTransformRecord(not JoltTransform JSON). But I am getting null value in the JSON file with the schema(output in screen shot).Please find my jolt specification:[{"operation":"shift","spec":{"to":"Receiver","from":"Sender","heading":"Heading","body":"Message"}}] Jolt Transformation DSL is chain. As I am trying to convert XML to JSON and make a change in element name in one go through JoltTransformRecord. I think JoltTransformRecord accepts any form of payload and then converts it into JSON format and then applies the change in elements of the JSON file. OutputInput file  Please help why this happens. thanks

Hello I installed CDP 7.1.1 because I wanted to try Ozone. I played a little with apache Ozone in the past and wanted to see how it is in Cloudera. After installation, I went ahead and tried to create a volume using this command (which complies with the documentation):ozone sh volume create --quota=1TB --user=hdfs /tests I got this error message:"Service ID or host name must not be omitted when ozone.om.service.ids is defined." I searched the internet for the error and did not find anything useful. I looked for ozone-site.xml file and found that ozone.om.service.id was "ozone1". But I do not know where in the command should I specify it. the command syntax does not take any service id or host name. Why am I getting this error ? how can I specify service id or host name so the volume will be created ? Thanks Guy

I am installing Cloudera 7 Data Center on AWS m5.xlarge (as a single node cluster) and ran into this problem - logs from /var/logs/cloudera-scm-server 020-04-26 16:26:28,355 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 401 URI: https://archive.cloudera.com/p/cdh7/7.0/parcels/manifest.json2020-04-26 16:26:28,355 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Could not retrieve repository info for repo https://archive.cloudera.com/p/cdh7/7.0/parcels/. Got HTTP response code 4012020-04-26 16:26:28,454 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-7.0.3-1.cdh7.0.3.p0.1635019-stretch.parcel2020-04-26 16:26:34,884 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands2020-04-26 16:26:34,884 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThre