Q&A

Hi all, I've followed the following tutorial CDH Hadoop Kerberos, NameNode and DataNode are able to start properly and I'm able to see all the DataNode listed on the WebUI (0.0.0.0:50070). But I'm unable to access the Hadoop CLI. I've followed this tutorial Certain Java versions cannot read credentials cache, still I'm unable to use the Hadoop CLI. [root@local9 hduser]# hadoop fs -ls /20/11/03 12:24:32 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]20/11/03 12:24:32 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]20/11/03 12:24:32 WARN security.UserGroupInformation: Priviledge

While trying to create a database through hive in Hue or using jdbc, I reveive the following error.Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.nio.file.AccessDeniedException: [bucket]: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: Missing required credential provisioning for Cloud Access Broker. It is expected that keys and secrets be provisioned as aliases or that Cloud Access Broker be running on a node with an Instance Profile attached.If I ssh into a node and run hdfs dfs -ls s3a://[bucket-name]/ I also see20/10/29 19:42:47 INFO impl.MetricsSystemImpl: s3a-file-system metrics system startedOct 29, 2020 7:42:47 PM org.apache.knox.gateway.shell.KnoxSession createClientINFO: Using default JAAS configuration20/10/29 19:42:48 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS20/10/29 19:42:48 ERROR common.DefaultRequestExecutor: Error executing request: org

Hi all, In the WEB UI NFI, I can find where modifie the policies to have permission to list/empty queue thanks for help.

I have a cluster of 1 Master and 1 Slave that are connected and "probably" communicating, I have followed several guides to install and setup the cluster in which almost all of them are similar, only differences are the memory and cores assigned. Both my master and slave have 8vcores and 32GB each, with around 600GB of SD. On the UI I can see the node is healthy and connected. However when I try to run a hadoop task I get the following message:   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output20/11/03 15:51:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable20/11/03 15:51:35 INFO client.RMProxy: Connecting to ResourceManager at master/master:803220/11/03 15:51:36 INFO input.FileInputFormat: Total input paths to process : 120/11/03 15:51:36 INFO mapreduce.JobSubmitter: number of splits:120/11/03 15:51:36 INFO mapreduce.JobSubmi

IN NiFi I have a flow which fetches files from remote server and post data into HDFS. PUTHDFS  relationship was set to both success and Failure. So, the files which are loaded to fail exists in same processor. Could you please let me know how to view list of files failed other than from data provenance?

Hello, Hope all of you are fine,  I recently started learning Apache Metron and I have been performing the squid tutorial for the last week. (I installed HDP 3.1.4 on Ambari 2.7.3 with HCP 2.0.0.0) The problem I am facing is that my logs aren't showing the the alerts UI.  I have already defined a elasticsearch index template with the two fields "metron_alert" and "alert" Here is the sensor information parser logs are being created in kibana      Below is the index template when you run  GET alert_ui_test_*   Below is the sensor and enrichment config     When I see the logs at /var/log/metron/metron-rest.log I see this error , but not sure about what it is  

I am trying a ParseCEF & a ParseSysLog processor with a text file and it falls down at the point of Parsing wrt only seeing the first line and then stopping. Are there any examples of parsing a text file to JSON and also recognising the multiple lines after the Parse processor so that there are multiple lines but in the same file. (I realise you can do a SplitText processor before the Parsing, but don't want to go this route.

UPDATE: If you ever need to modify a record use this solution: https://community.cloudera.com/t5/Support-Questions/NiFi-UpdateRecord-processor-is-not-updating-JSON-path/m-p/186256 ORIGINAL question:How can I extract a string nested in the value of a JSON? For the status key below, I only want the value Active, instead of MapRecord[{current_status=Active}]. Original JSON { "indicator": "147.183.200.154", "timestamp": 1577742090000, "status": "MapRecord[{current_status=Active}]"}  Desired final JSON { "indicator": "147.183.200.154", "timestamp": 1577742090000, "status": "Active"}  

Greetings,  I wanted to know how to set up a process where one will receive an email when ever a queue reaches its threshold limit , in apache nifi.Thanks in advance.    

Hi, I am trying to see a summary of a query's progress that updates in real-time, Hence I ran 'setLIVE_PROGRESS=1;'. But I am getting the message stating as below: ERROR: User hadmin@HADOOP-GROUP.LOCAL is not authorized to access the runtime profile or execution summary. However, the user hadmin is the authorized user to view that particular database and table. Please suggest what could be the reason and how to fix this, Since I have taken the Kerberos ticket as well.

I want to set logging level of  hiveserver2Interactive.log to DEBUG in HDP-3.1.could anyone help here?

Hello all!I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere? I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!! Thanks.Valentin

Hello all, I've recently decided to enable Kerberos on all the service which are getting used in my company. I've successfully enabled Kerberos on Zookeeper, Kafka, Hadoop, Hbase.When I'm trying to enable Kerberos on Hive-metatore and Impala I'm getting following error:I've followed the following guides:CDH Impala Kerberos CDH Hiveserver2 Security CDH Hive Metastore Security  hive-metastore.log ERROR [pool-4-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TU

I am trying to reference an Avro Schema that already works from a CEF file and take out a part of that file (time as rt) by referencing the Schema, but I am a bit rusty with my SQL so it's not going to plan rn - does anyone have any examples of doing this in NiFi?

Hi there, I have some issue with disable the region replication for a table.HBase version: 2.0.2.3.1.4.0-315 I have region replication enabled on a single table (REGION_REPLICATION => 2), which doubled amount of regions of the table. I keep it for a while and then changed region replication to 1 (via alter 'table', 'cf', {REGION_REPLICATION => '1'}). What I expected to see is the number of region go down. But it wasn't happened. What I tried:run major compactionrestart region serversdisable region_replica_replicationDoes anyone know what is the procedure for disabling a region replication and how to get rid of secondaries regions?

Hi Team, We are using HDP 2.6.3 and ambari 2.6.0. we are facing issue while using sqoop import to MS Parallel Data Warehouse. As per https://community.cloudera.com/t5/Community-Articles/Sqoop-to-MS-PDW-Parallel-Data-Warehouse-issue/tac-p/304591 issue is fixed in HDP 2.5.5 version and above but we are still seeing the same issue in HDP 2.6.3. Do we have any workaround for this.   Thanks in Advance!

Hi sir/mam,Good day!I received my Cloudera Certification. There is a typo error in my name. In Certificate : Needhi Devan Senthi KumarMy Name : Needhi Devan Senthil Kumar There is 'l' missing in the certificate name. What should I do for this name correction. It would be a pleasure if any one replies. Thank you.

Currently we have a Hortonworks HDP cluster version 2.6.5. We are planning to migrate our platform to the new Cloudera. Which is the recommended upgrade path? Upgrade HDP first to version 3.1.5 and later migrate to CDP or is also possible to migrate directly from HDP 2.6.5 to CDP?   Thanks in advance for your inputs and suggestions.

I took the Cloudera CCA 175 (CCA Spark and Hadoop Developer) exam on 12.10.2020 at 10.30 am IST. I was unable to use quotes and double quotes in the test environment. I tried various ways like copied it from spark documentation and used it but I cannot run it in the test environment. It throws the error illegal character '\u201d'. At last I lost my hope after 1.30 hours and requested for reschedule the exam.

Hello,  I have a cluster that currently runs with CDH and Cloudera Manager version 5.15.2 for both and I would like to upgrade only Cloudera Manager to 6.3.3. Is there any issue running the cluster with these different versions of CDH and CM? Thank you, 

hello comunnity. Currently I am going to implement a cluster of 20 hosts in total.The RDBMS is ORACLE, which will house the Cloudera scm, rman, nifi, etc. tables. What should be the minimum requirements for a database server.Oracle versionStorageRAMCPUThank you very much for your valuable help

Hi All - I'm new to Hadoop, recently installed HDP_2.5_virtualbox - trying to login to SQL - it prompts for a password, not sure the password - appreciate any help to login to Mysql. [root@sandbox ~]# Mysql -u root -p Enter password: 

Hello, We are using a beeline or impala-shell to extract data from hive tables as per requirement from the end-user. However, the request for data extraction is for high records for more than 1000 or sometimes more than 3000. It is very tedious to extract using the select query and dump to excel sheet. Is there any alternative way to take the output in a CSV file?. Like the output of select query moves to CSV file.  Please suggest.

While querying one of the Impala tables (out of hundreds of tables under the same database), I got the following exception: select count(*) from impala_table;Query: select count(*) from impala_tableQuery submitted at: 2020-10-23 03:12:42 (Coordinator: https://impalad:25000)ERROR: AnalysisException: Failed to load metadata for table: 'impala_table'CAUSED BY: TableLoadingException: Failed to load metadata for table: impala_db.impala_tableCAUSED BY: RemoteException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1962) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1421) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3055) at org.apac

Hi Community. I am getting an error starting Hive in Cloudera Manager, CDP7. The error reads, "Command failed to run because service Hive has an invalid configuration. Review and correct its configuration. First error: Missing service dependency for Hive: YARN". How do I add the YARN dependency to an existing Hive service? Reinstalling Hive is not a really practical solution. Is there any way to add the dependency without reinstalling Hive? Thanks!