Solved

Hello,I need to reboot my Data node to fix an underlying network issue. OS team needs around 3 hours (approx.) to complete this activity, what would be the best approach:Decommission and recommission host / dn.Move DN to Offline Mode (feature in CM 5.16.x)Any help / guidance is appreciated. RegardsAmn

hi all we are examining the option to installed new Hadoop cluster based on CDH cloudera packages I will appreciate to get feedback's / notes / remarks / else ,  from someone that have the experienced of installing the CDH packages , and how this Hadoop looks after installing the  Cloudera open source ( CDH )    

During an overly ambitious cleanup effort, the IAM user / access key used by our Altus Director server was deleted from AWS and now we can no longer manage our clusters.  I would like to either update the access key used by director, or remove the key and force director to rely on IAM role.  Please advise.

Hi Community,Suppose that i changed one of the values in the node hdfs-site.xml manually, will cloudera manager override this config?example : in /etc/hadoop/conf i want to change the dfs.nameservices in  hdfs-site.xml manually and i want to how cloudera manager can override this. *** I don't want to make such a change but i want to know if cloudera manager can override this.

I'm running Ubuntu 18.04 LTS in my machine.I'm trying to install CDH 6.1.While installing agents I got error message like this below     Please help give soliution. Thank you Regard,solomonchinni

Hi Team, I  have updated nifi to 1.11.4 from 1.5.0 and the ranger policy is not working as expected .I have set up policy with /process-groups/<process group uuid>  and it opens up access to all the process groups which are present in canvas . example :- /process-groups/992fc902-0172-1000-348e-b0bae84f520f.  - i gave read and write to this process group .But i am able to go into all the process groups which are available .   regardsBharad

We are using Livy 0.7.0 with CDH 6.3 version. We are able to submit the job through Livy to CDH Cluster. We wanted to restrict the User access to the Livy session through Livy ACL.  As per the Apache Documentation added the below parameter values and restarted the Livy. But still livy is allowing all the users to access.    Any other settings needs to be enabled for ACL to work.   # Whether to enable Livy server access control, if it is true then all the income requests will# be checked if the requested user has permission.# livy.server.access-control.enabled = false livy.server.access-control.enabled=true # Allowed users to access Livy, by default any user is allowed to access Livy. If user want to# limit who could access Livy, user should list all the permitted users with comma separated.livy.server.access-control.allowed-users = raj   Thanks Raj

from ambari we can capture all version , by click on `Admin` button and the click on `stack and version`finally click on `Versions` , then we get the following details  we want to know how to capture all these version by using ambari rest api?we trycurl -u admin:admin -H 'X-Requested-By:admin' 'http://localhost:8080/api/v1/clusters/HDP/configuratons/service_config_versions'but its not return any info 

Is there any documentation on KafkaConnect?. I see lots of documentation and examples on Confluent. However, they use kafkacat and ksql.Where can I find documentation, examples for Cloudera?

Hi.I'm trying to utilize GetHDFSFIleInfo(not GetHDFS) to start after previous step.Actually, this is what I exactly want to do (https://community.cloudera.com/t5/Support-Questions/NiFi-fetchHDFS-without-ListHDFS/td-p/211708) flowstreamcommand -> GetHDFSFileInfo -> FetchHDFS -> PutSFTP I did make gethdfsfileinfo first, after then tried to get flow file info from previous job at FetchHDFS.However, the result of gethdfsfileinfo was only one attribute and I don't know how to fetch all files at fetchHDFS. This is an attribute of gethdfsfileinfo. I got an attribute as below and want to fetch those all files.filename : faaa~~~~~hdfs.count.dirs : 1hdfs.count.files : 44hdfs.full.tree : {"ojbectName";"gfk",""...., "content":[{"objectName":"Weekly_GfK_02_Merge_F_HP_AT.txt",".....]}hdfs.objectName : gfk....Could you please tell how to use it? or how to solve this problem? 

Hi,Is a CDH Enterprise license transferable from one cluster (deployed just for a proof-of-concept and after e.g. 3 months destroyed), to another cluster (deployed only after the former one was destroyed)?

Hi everyone, I am uploading Navigator logs into Hive for analysis. I used this: ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ("separatorChar" = ",","quoteChar" = '"',"escapeChar" = "\\")STORED AS TEXTFILE Based off: https://community.cloudera.com/t5/Support-Questions/Hive-escaping-field-delimiter-in-column-value/m-p/233346#M195173 to get an external table to handle the commas in queries. I think it's working fine (but will test tomorrow to make sure). However, I then do a move to a managed partitioned table, but it is not handling the commas in queries correctly. I created it as:PARTITIONED BY(event_day INT, event_month INT, event_year INT); How do I handle the commas in queries as part of this move?Does the managed table need the same delimter & escape info as the external table?If that's the case, what's the syntax for it? Schwifty

we have Hadoop cluster with only 2 data nodes machines in HDFS configuration we defined the Block replication to 3 soBlock replication=3is it OK? to defined Block replication=3 , when we have only two data nodes in the cluster? from my understanding when we defined Block replication=3 while we have 2 data nodes machines in HDFS cluster its means that one machine should have 2 replica .  and the other machine one replica , am I correct here?

I have a line by line list in my flowfile, which records the result of some file processing, somthing like the following; C:\Data\client1\order100.txt - SUCCESSC:\Data\client1\order122.txt - ERRORC:\Data\client1\order323.txt - SUCCESS I can't see a way to add this flowfile content to the e-mail MessageBody. I have therefore tried an ExtractText processor with message.body as an attribute and (.*) as the regex. This gets me close, but all the filenames are on the same line, so I end up with this; C:\Data\client1\order100.txt - SUCCESS C:\Data\client1\order122.txt - ERROR C:\Data\client1\order323.txt - SUCCESS Which is not quite the e-mail I'm looking for. Any sensible suggestions welcome.   

1. Setup Nifi 1.11.4 secure using my own certs and kerberos authentication accepting username/password.  All is working fine can log in using kerberos username/password as initial admin user. 2. Setup Nifi Registry 0.6.0 using same certs and kerberos all is fine. Again all is working fine can log in using kerberos username/password as initial admin user.Problem is that when adding registry to secure Nifi it seems they attempt to talk using the username 'anonymous'.If I add the user "anonymous" to Registry giving it permissions to proxy, then all works fine, however registry shows all changes were performed by 'anonymous' Also can't use policies to not allow the anonymous user access to the UI  I can't find any documentation on why I would need to create a user with name 'anonymous'In fact most documentation says when using https that anonymous access it not supported. Q. How does Nifi/Registry identify the identity of the host in standalone mode? If I gene

It told me to be a CDP Data Center customer. But can you tell me how to be ???  

Dear All,I need your suggestion huge number data transfer to Syslog through Apache NiFiMy aim is transferring all IIS logs (located on Azure Blob Storage) to syslog line by line. Therefore I am using split text for parsing lines. Extract Text is transferring split line on the attribute, in this way I can say to syslog processor "Message Body: IISHttp${msg}". I will explain details at the belowActually my problem is bottleneck on the Extract Text. I have to transfer IIS Logs near-real time (less than 1 hour delay). But Extract Text isn't draining the messages (flow files) from the queue in proper time. I tried increasing Thread Number, Run Duration, increasing/reducing Queue size but I couldn't achieve my target. The queue between split text and extract text are always full. I have log gap about 13 hours. I am trying find a way for thatMy flow is that;1. Getting IIS Log files from Azure Blob Storage1.1. List Azure Blob Storage Processor1.2. Route on Attribute Processor (I have date filt

Hello, I have a piece of JSON on which I want to apply an updateRecord processor using some filter capabilities. My JSON looks like this:   "status": [ { "code": "InProgress", "isCurrentStatus": false, "startDateTime": "2020-03-05T20:45:00Z" }, { "code": "Pending", "isCurrentStatus": true, "startDateTime": "2020-03-05T21:20:00Z" } ],   And my RecordPath expression is the following:   /status[0..-1][./isCurrentStatus = true]/code  This expression doesn't work and I have an error in my logs file stating about incorrect syntax:  Unexpected token ']' at line 1 If I put the following syntax:  /status[0..-1][./isCurrentStatus = "true"]/code  it doesn't give anymore error, but of course the filter doesn't work. So, I have the feeling that recordPath filter doesn't support the use of boolean values. My Nifi

noticed below error from users.log  but nothing changed from nifi configs but started noticing this error nifi-user.log:2020-07-02 12:16:13,857 INFO [NiFi Web Server-343673] o.a.n.w.a.c.ResourceNotFoundExceptionMapper org.apache.nifi.web.ResourceNotFoundException: Unable to find user with id '938eb61e-bbc4-383a-8475-aee80541b5a5'.. Returning Not Found response. @MattWho 

Hi, Trying to upload logs into a Hive external table. Some records contain queries which have multiple columns in them, but commas are also the field delimiters. Other questions here recommend using org.apache.hadoop.hive.serde2.OpenCSVSerde. So I used: ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'STORED AS TEXTFILELOCATION 'hdfs://hdfs/path/to/directory/external/2020Jun30'TBLPROPERTIES ('skip.header.line.count'='1'); This didn't deal with the additional non-escaped commas in the log file though. So I tried:ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES (   "separatorChar" = ",",   "quoteChar"     = '"',   "escapeChar"    = "\\")  STORED AS TEXTFILE However, in Hue I get this error when using org.apache.hadoop.hive.serde2.OpenCSVSerde:AnalysisException: Failed to load metadata for table: 'db.ext_table' CAUSED BY: TableLoadingException: F

Hi Team ,  I am having issues while connecting to mysql database using apache nifi.  The error is as followsCannot create PoolableConnectionFactory Please note that Apache Nifi and MySQL is installed on the same Linux server and hence there shouldn't be any problems with the 2 applications talking to each other.  The database connection pool settings is provided below The error when I try extracting the records is provided below  This is something very urgent as we are doing a POC and if this succeeds our Projects will be based on this. Any help in this regard will be much appreciated.  @Former Member @Shu 

Hi, Following the official documentation  I logged in to https://public-IP  Logged in using the Super User admin account. Post that I configured A/D integration and on the next page I added a few users/groups for admin say users1, user2, etc.. Post that it logged me out and asked me to login to the same page  https://<public-IP>  . Now I am trying to login using user1 and corresponding A/D password but I get and error - invalid username/password error. I tried with user2 also which was configured. I also tried using user1@realm.com  . To my surprise, I am not longer able to login using super admin also.  I tried to see the various container logs but no success. Has anyone faced similar issue?What am I missing here?  Thanks in advance.  Regards Tagging @SumitraMenon @Former Member @abajwa 

HI Team, I came across an scenario where HA failed when active namenode server was rebooted.Accidentally the master node which runs active namenode got rebooted.when it got rebooted i believed the standby namenode will become active namenode and will continue the operations but standby namenode did not became active and HDFS was completely down.So my question is1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster2. Whether the HA is expected to work only between the active and standby namenode daemons? Thanking in Advance!!

Environment HDP-3.1.0.0Hive3.0.0.3.1 Context I am trying to configure hive to use LDAP (AD). But hiveserver2 throws errors when restarted. Similar errors are visible when connecting to beeline. Configurationhive-site.xml (truncated)     <configuration xmlns:xi="http://www.w3.org/2001/XInclude"> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.authentication</name> <value>LDAP</value> </property> <property> <name>hive.server2.authentication.ldap.baseDN</name> <value>DC=MYDC,DC=MYDC</value> </property> <property> <name>hive.server2.authentication.ldap.Domain</name> <value>DOMAIN</value> </property> <property> <name>hive.server2.authen

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format./spark2-history/application_1574201808558_1228909.lz4Ambari Version : 2.6.0.0HDP Version : 2.6.3.0-235Spark Version : 2.xThank you.