Q&A

What is the stable version of CDH6.X ,Not CDH6.3.3.We have cluster with 5.16.2 and planning to upgrade to CDH 6 ,but not CDH 6.3.3. Can someone suggest me which is the stable version of CDH 6 except CDH6.3.3? 

"SecondShot" Cloudera Promotion not working for me, please assist!!

Hi Team, We are using HDP-3.x & HDF-3.x in our environment. Is it possible to do in-place upgrade from HDP-3.x to CDP Private Cloud Base ? Also, how to upgrade HDF-3.x to CDP ? Link to respective documentations would be really helpful. Please suggest ways to do upgrade. Thanks,Bhushan

I am looking to explore join properties in hive.I would like to know if the setting I change in my login/session ,will affect other users too.This is the property I am going to work: set hive.auto.convert.join=true;

Hi, can you please tell me how to add the Flink Service to Cloudera. I can add the Parcel but it's not Distributing and shows service not available.Can you please help me on this.Thanks

Hi there. I would like to concat all the values of a repetitive key and put all of them as the value of one unique key with the same name.The input would be like: { "main": "id;Date;Time;Trigger_time_open;Trigger_time_close;Door;Door_Cycle;Motor_current;Motor_voltage;DLS;DCS;Inclinometer;Temperature" }, { "main": "00000;2020/7/18;04:53:37.211;04:53:37.524;04:53:51.847;01;36688;12.062867;0.000000;-1;-1;12.078014;26.303200" }, { "main": "00001;;04:53:37.212;;;;;12.064088;0.000000;-1;-1;;" }  An the output something like: {"main":"id;Date;Time;Trigger_time_open;Trigger_time_close;Door;Door_Cycle;Motor_current;Motor_voltage;DLS;DCS;Inclinometer;Temperature00000;2020/7/18;04:53:37.211;04:53:37.524;04:53:51.847;01;36688;12.062867;0.000000;-1;-1;12.078014;26.30320000001;;04:53:37.212;;;;;12.064088;0.000000;-1;-1;;"}  I have been trying with join and concat, but I'm completely new to Jolt transformations, and it's been a pain to find the way.&nb

Hello all, I'd like to use the updateRecord processor to create some fields in my JSON. But I'd like to use some conditions to do that. In the recordpath documentation, I can see that there is a "filter" capability, which works like this: /field[filter]/path, but as far as I can see it's not possible to have multiple conditions in this filter. I'd like to do something like: if (field1 == A or field1 == B) {field2 = C} else if (field1 == D and field1 != E) {field2 = G}  The example is silly but this is to give you the idea. By the way, except using a script, I don't know how to do that, except may be extracting the fields, put it as attribute and play with expression language but this would mean working with single-record flows and I don't want to do that. Any idea? Thanks for your suggestions Stéphane 

how to get credentials for accessing :-https://archive.cloudera.com/p/HDP/centos7/3.x/updates/3.1.5.0 ??Note: i am new user and i need credentials to access. 

Good afternoon I am trying to make improvements to the way we make our Nifi flows by implementing Record processing. Historically we:split text > get timestamps using regex > merge on 'corellation_id' (attribute from timestamp: format- yyyy-MM-dd-HH) > extract sourcetype using regex (flow splits at this point for each sourcetype) > merge again on 'correlation_id' > out to HDFS/Splunk etc flow structured like this:Old Split/Merge Flow(each spine is a different sourcetype) I recently watched this series of videoshttps://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWwI was particularly interested in the first one where Record Processing is used in place of split/merge/regex etc.  As a result I have started to re-design the flow above, the logs come into us in CEF format from a syslogListener (TCP). My progress so far in processors: GenerateFlowfile - using some copied data from original flow.SplitText - to split text/plain content i

 Like this we are getting some times HDFS canery good and some times HDFS Canary BadHDFS Canary Good2 Still ConcerningNov 27 12:15:53 PM HDFS Canary BadNov 27 12:15:08 PM DataNode Health ConcerningNov 27 11:58:47 AM DataNode Health BadNov 27 11:58:12 AM DataNode Health ConcerningNov 27 10:07:15 AM DataNode Health BadNov 27 10:07:00 AM DataNode Health ConcerningNov 27 9:29:35 AM DataNode Health BadNov 27 9:29:20 AM DataNode Health ConcerningNov 27 8:45:31 AM DataNode Health BadNov 27 8:45:06 AM DataNode Health ConcerningNov 26 10:03 PM HDFS Canary Good2 Still BadNov 26 10:02:23 PM DataNode Health BadNov 26 10:02:18 PM HDFS Canary BadNov 26 10:01:42 PM HDFS Canary Good2 Still ConcerningNov 26 8:01:53 PM HDFS Canary BadNov 26 8:01:03 PM HDFS Canary Good2 Still ConcerningNov 26 6:16:18 PM HDFS Canary BadNov 26 6:15:38 PM DataNode Health ConcerningNov 26 4:45:01 PM DataNode Health Bad   We are finding this logs in service Monitor   &nbs

Dear All,Currently I am using below SQL query in Execute SQL processor (SQL select query) to join 3 different tables that exist within my PostgreSQL database.   SELECT T1.*,T2.*,T3.* FROM amsterdam.amd_block_info T1 LEFT JOIN amsterdam.amd_ap_info T2 ON T2.ap_track_id = T1.track_idLEFT JOIN amsterdam.amd_track_info T3 ON T2.ap_track_id = T3.track_id;   But I am getting below error response.   ExecuteSQL[id=3c9f01f3-0176-1000-0000-000032d6da19] Unable to execute SQL select query SELECT T1.*,T2.*,T3.* FROM amsterdam.amd_block_info T1 LEFT JOIN amsterdam.amd_ap_info T2 ON T2.ap_track_id = T1.track_idLEFT JOIN amsterdam.amd_track_info T3 ON T2.ap_track_id = T3.track_id; due to org.apache.avro.AvroRuntimeException: Duplicate field track_id in record any.data.amd_block_info: track_id type:UNION pos:11 and track_id type:UNION pos:3.. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.avro.AvroRuntime

Greetings, I work for a large organization that wants to purchase a NIFI system,but before the purchase is made we need to know whether the system is Fault tolerant, in the sense that if a Node / Cluster crashes while consuming / processing / putting the data, it is possible to fully recover the data that was lost, automatically or manually.If the answer is no, is it possible to set NIFI so it won't send Acks to the system from which it consumes the data from, until the moment the data completes the whole flow process and is uploaded to its destination, or at least until the data is uploaded properly to NIFI?thank you!

I'm trying to learn more about s3guard and was attempting to follow along with some of the CLI examples in the CDP documentation. Any command I try results in a warning:WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.propertiesfollowed by an error:java.lang.IllegalStateException: Authentication with IDBroker failed. Please ensure you have a Kerberos token by using kinit. When I try running kinit I see:Client 'cloudbreak@[FQDN]' not found in Kerberos database while getting initial credentialsWhere "FQDN" corresponds to the VM I am ssh into. I've tried a couple machines in both my Data Hub and Data Lake clusters. Does anyone have insight on how to properly interact with my environment's s3guard setup?

Until recently we had no issues with the HUE - Kerberos Ticket Renewer - now this service is stopping/starting and failing all the time.   Does anyone have any helpful links on how to troubleshoot this service?     I don't have much experience with this product in terms of troubleshooting - we are running Cloudera Express 5.12.0  Any suggestions or ideas would be greatly appreciated.

Hello Team, I am facing issue in my cloudera cluster and HDFS used space is growing and "/hbase/oldWALs" is occupied more than 50%. I can confirm that, HBase replication is disabled and TTL is set to 1 Min.hbase master logcleaner ttl = 1 Minhbase replication = false HBase logs i can see below warn's,WARN org apache hadoop hbase master cleaner CleanerChore: A file cleanerhostname 60000.oldLogCleaner is stopped, won't delete any more files in /nameservice1/hbase/oldWALs And checked list of peers in hbase,hbase(main):001:0> list_peersPEER_ID CLUSTER_KEY STATE TABLE_CFS0 row(s) in 0.2360 seconds I dont see anything, Please help me with your comments.  Thanks & Regards,Vinod

I want to copy only csv files from on directory into another, I use the command:hadoop distcp /a/b/*.csv /e/f/ The issue here is, if there's multiple csv files under /a/b/, this command would work, it will copy all csv files into the directory /e/f/, however, if there's only one csv file, it will be copied into /e/f where f is a file not a directory, is there a way to resolve this? Thanks in advance! 

I am using ConvertJSONtoSQL to convert simple JSON to a SQL statement for inserting rows in DB2. I am using ConvertJSONtoSQL and PutSQL to write to DB2, but it is omitting the unmatched columns. How can I add unmatched columns to the SQL? Is there a way to add the matching names in the JSON prior to running ConvertJSONtoSQL? I tried using AttributestoJSON but it didn't work. Can someone please help me?

I have an external table pointing to partitioned parquet data in an AWS S3 bucket. I realized our write-out process was creating too many files within a partition, so I tweaked our code and overwrote the parquet data in that S3 location to be more compact. I then dropped the table and re-ran the `CREATE EXTERNAL TABLE` and `ALTER TABLE ... RECOVER PARTITIONS` statements. The issue I'm running into now is that the table seems to be pointing to both the old and new parquet data. If I run a `SHOW FILES IN` command I see both old and new files listed for the table. This leads to errors when I try to access data in the table, as it's seem to be trying to read from a file that no longer exists in S3. Is there a cache or something similar that needs to be cleared in these types of situations?

when I upgrade to the HDP3.1 and hbase also upgrade to 2.0(ambari server show 2.0.0.3.1,and hbase version shows HBase 2.0.2.3.1.0.0-78),and how can i use hbck2 if I need compile myself,and how I set the hbase version apache,or HDP ,where can I get the HDP pom.xml ,now hbck can't fix my hbase2.0

Hello everyone i am now having a problem connecting tableau to druid on HDP.... Anyone know how to connect tableau to druid ?

Anyone know..How to connect R Studio to Hive in Hortonworks Data Platform 3 ?I am using Hive 3.1.0

Hi.I have a problem with outOfMemory error on impala catalog server.Data in tables about 30TB and about 175k partitiions.I set Java Heap Size of Catalog Server in Bytes=30gb, but still have out_of_memory error in start.How can i calculate how much memory needs for catalog server? And is it have any limit to set ?

Hi, I'm using NIFI EvaluateXPath module.I am trying to extract the value from the first Channel parameter (Microsoft-Windows-Sysmon/Operational) and add it as an attribute - The channel1 attribute is empty when setting the value to  "/Event/System/Channel"    I'm trying to understand what should be the Value in the Channel1 property  XML  <?xml version="1.0" encoding="UTF-8"?><Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385f-c22a-43e0-bf4c-06f5698ffbd9}"/> <EventID>10</EventID> <Version>3</Version> <Level>4</Level> <Task>10</Task> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2020-12-03T14:23:35.660463800Z"/> <EventRecordID>94211</EventRecordID

I am trying to create Ranger hive row-level filter policies using Rest API. I am truying to use this bellow Json format but it is creating just regular hive policy instead of row-level filter policy. Any clue on the best Jason format ? curl -iv -u USERID:PASSWD -H "Content-Type: application/json" -X POST http://HOSTNAME001:6080/service/public/api/policy/ -d '{"policyName": "RESTAPI_TEST","databases": "Test_databse","tables": "Test_table","columns": "*","udfs": "","description": "Hive Row Level Filter Policy","isEnabled": true,"isAuditEnabled": true,"rowFilterPolicyItems" : [ {"rowFilterInfo" : {"filterExpr" : "ROW_DVI = 02"},"delegateAdmin" : false,"accesses" : [ {"type" : "Select","isAllowed" : true} ],"users" : [ "USER_ID123" ]}]}'

I am running into a problem with the NiFi fetchgcsobject processor.. It fails when trying to retrieve files with spaces in the file name... Below is a sanpshot of the error. Interestingly the listgcsbucket works fine and lists the file. fetchgcsobject is where it fails. It also works fine with files that have no spaces in the file name. This works on version 1.10. Is this a bug on 1.12.1? I tried to encode spaces with %20 .. that doesnt help either.. Failed to fetch GCS object due to Blob BlobID(bucket=test-bbbb, name=outbound/test file.pdf) not found