Solved

Hello, I am unable to know the size of the hive table. Actually it is a view specifically which we wanted to know the size of. I have tried by running commands such as hdfs dfs -du -s -h /path/to/table, but the view can't be found in the location. beeline> describe formatted/extended table name; Also does not give the output  I also tried with ANALYZE but no output as well. Can some one tell me how to know the size of the view table. Since we have created view of the different table in database.  And we want to know the size of the particular table.

Have a list of about 100+ SQL  Count Queries to run against a Hive Data Table, Looking for the most efficient way to run these queries. Queries are accessed at runtime as a list of queries stored in another Hive Table, generated by a different process.     Queries like these each with a different where clause, where clauses are complex: 1. Select count(1) as count1 from MyTable where (... complex where clause here...) 2. Select Count(1) as Count1 from MyTable where (... where clause here ...) 3. etc..   Environment: Cloudera CDH 6.2  

I've enabled Auto-TLS on an existing CDP 7.1.3 cluster as instructed in https://blog.cloudera.com/auto-tls-in-cloudera-data-platform-data-center/ option 2b.Initially after completing the wizard and restarting CM agent and servers agent heartbeat was lost due to missing agent certificates from truststore. I was able to fix this by adding hosts to trustore "Cloudera Manager TLS/SSL Client Trust Store File" /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks with Java Keytool. However now I'm observing some oddities in services. - YARN, Kafka seem to report in their Role Log at server startup (Error 403 Broken pipe (Write failed) the server declined access to the page or resource.) - If I try to access any logs in the cluster via CM GUI (Clusters -> Service -> Instances -> Any role -> Log files -> Any log -> Download full log ) I get the following message "HTTP ERROR 403 Problem accessing /cmf/process/all/logs/download. Reason: Receiv

Hi,I have a cluster running Hbase with 1 Master and 6 RS. Recently I noticed that Hbase commands are sort of queuing up and possibly hung(RUNNABLE state) for one table.When checking the state of the table, I an see that the table is disabled. The command executed to enable the table failed when executing from the Hbase shell stating that process id xxx(previous truncate command) is already running. In the HBase UI, I can see both the truncate and enable commands(shown under Procedures) is in RUNNABLE state.I tried the kill procedure command on the truncate command but it returns false, indicating the process cannot be killedI have data on other tables and scan commands on these tables work fine.What might be the issue here, how can I kill the command running at Hbase and get the table back to working?Any Help is much appreciated.Regards,ThomasOn checking the Master log, I can see the following Warnings:2020-01-08 11:03:28,168 WARN  [RegionOpenAndInitThread-buckets_2-10] ipc.Client

Does Major compaction in Hbase lead to regions merging or the regions are merged only manually by applying merge command?Please help @Manoj69 @jsensharma 

Hello!The maven artifacts list for 7.1.3 is available at :https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdppvc-runtime-maven-7.1.3.html Is the same list for 7.1.4 available?  Is it the same list?  I am unable to find this list for CDP 7.1.4. Thanks.

Hello,On our data node, we are increasing getting alerts related to Data Node Pause Duration. So far, this is happening on a single data node out of nine data nodes.Following is the error captured from DN logs 2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms) Current Java Heap Size of Data Node in Bytes is at 6GB CM / CDH – 5.16.x Any help is appreciated. RegardsAmn

Hi all, I've followed the following tutorial CDH Hadoop Kerberos, NameNode and DataNode are able to start properly and I'm able to see all the DataNode listed on the WebUI (0.0.0.0:50070). But I'm unable to access the Hadoop CLI. I've followed this tutorial Certain Java versions cannot read credentials cache, still I'm unable to use the Hadoop CLI. [root@local9 hduser]# hadoop fs -ls /20/11/03 12:24:32 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]20/11/03 12:24:32 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]20/11/03 12:24:32 WARN security.UserGroupInformation: Priviledge

While trying to create a database through hive in Hue or using jdbc, I reveive the following error.Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.nio.file.AccessDeniedException: [bucket]: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: Missing required credential provisioning for Cloud Access Broker. It is expected that keys and secrets be provisioned as aliases or that Cloud Access Broker be running on a node with an Instance Profile attached.If I ssh into a node and run hdfs dfs -ls s3a://[bucket-name]/ I also see20/10/29 19:42:47 INFO impl.MetricsSystemImpl: s3a-file-system metrics system startedOct 29, 2020 7:42:47 PM org.apache.knox.gateway.shell.KnoxSession createClientINFO: Using default JAAS configuration20/10/29 19:42:48 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS20/10/29 19:42:48 ERROR common.DefaultRequestExecutor: Error executing request: org

Hi all, In the WEB UI NFI, I can find where modifie the policies to have permission to list/empty queue thanks for help.

I have a cluster of 1 Master and 1 Slave that are connected and "probably" communicating, I have followed several guides to install and setup the cluster in which almost all of them are similar, only differences are the memory and cores assigned. Both my master and slave have 8vcores and 32GB each, with around 600GB of SD. On the UI I can see the node is healthy and connected. However when I try to run a hadoop task I get the following message:   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output20/11/03 15:51:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable20/11/03 15:51:35 INFO client.RMProxy: Connecting to ResourceManager at master/master:803220/11/03 15:51:36 INFO input.FileInputFormat: Total input paths to process : 120/11/03 15:51:36 INFO mapreduce.JobSubmitter: number of splits:120/11/03 15:51:36 INFO mapreduce.JobSubmi

IN NiFi I have a flow which fetches files from remote server and post data into HDFS. PUTHDFS  relationship was set to both success and Failure. So, the files which are loaded to fail exists in same processor. Could you please let me know how to view list of files failed other than from data provenance?

Hello, Hope all of you are fine,  I recently started learning Apache Metron and I have been performing the squid tutorial for the last week. (I installed HDP 3.1.4 on Ambari 2.7.3 with HCP 2.0.0.0) The problem I am facing is that my logs aren't showing the the alerts UI.  I have already defined a elasticsearch index template with the two fields "metron_alert" and "alert" Here is the sensor information parser logs are being created in kibana      Below is the index template when you run  GET alert_ui_test_*   Below is the sensor and enrichment config     When I see the logs at /var/log/metron/metron-rest.log I see this error , but not sure about what it is  

I am trying a ParseCEF & a ParseSysLog processor with a text file and it falls down at the point of Parsing wrt only seeing the first line and then stopping. Are there any examples of parsing a text file to JSON and also recognising the multiple lines after the Parse processor so that there are multiple lines but in the same file. (I realise you can do a SplitText processor before the Parsing, but don't want to go this route.

UPDATE: If you ever need to modify a record use this solution: https://community.cloudera.com/t5/Support-Questions/NiFi-UpdateRecord-processor-is-not-updating-JSON-path/m-p/186256 ORIGINAL question:How can I extract a string nested in the value of a JSON? For the status key below, I only want the value Active, instead of MapRecord[{current_status=Active}]. Original JSON { "indicator": "147.183.200.154", "timestamp": 1577742090000, "status": "MapRecord[{current_status=Active}]"}  Desired final JSON { "indicator": "147.183.200.154", "timestamp": 1577742090000, "status": "Active"}  

Greetings,  I wanted to know how to set up a process where one will receive an email when ever a queue reaches its threshold limit , in apache nifi.Thanks in advance.    

Hi, I am trying to see a summary of a query's progress that updates in real-time, Hence I ran 'setLIVE_PROGRESS=1;'. But I am getting the message stating as below: ERROR: User hadmin@HADOOP-GROUP.LOCAL is not authorized to access the runtime profile or execution summary. However, the user hadmin is the authorized user to view that particular database and table. Please suggest what could be the reason and how to fix this, Since I have taken the Kerberos ticket as well.

I want to set logging level of  hiveserver2Interactive.log to DEBUG in HDP-3.1.could anyone help here?

Hello all!I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere? I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!! Thanks.Valentin

Hello all, I've recently decided to enable Kerberos on all the service which are getting used in my company. I've successfully enabled Kerberos on Zookeeper, Kafka, Hadoop, Hbase.When I'm trying to enable Kerberos on Hive-metatore and Impala I'm getting following error:I've followed the following guides:CDH Impala Kerberos CDH Hiveserver2 Security CDH Hive Metastore Security  hive-metastore.log ERROR [pool-4-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TU

I am trying to reference an Avro Schema that already works from a CEF file and take out a part of that file (time as rt) by referencing the Schema, but I am a bit rusty with my SQL so it's not going to plan rn - does anyone have any examples of doing this in NiFi?

Hi there, I have some issue with disable the region replication for a table.HBase version: 2.0.2.3.1.4.0-315 I have region replication enabled on a single table (REGION_REPLICATION => 2), which doubled amount of regions of the table. I keep it for a while and then changed region replication to 1 (via alter 'table', 'cf', {REGION_REPLICATION => '1'}). What I expected to see is the number of region go down. But it wasn't happened. What I tried:run major compactionrestart region serversdisable region_replica_replicationDoes anyone know what is the procedure for disabling a region replication and how to get rid of secondaries regions?

Hi Team, We are using HDP 2.6.3 and ambari 2.6.0. we are facing issue while using sqoop import to MS Parallel Data Warehouse. As per https://community.cloudera.com/t5/Community-Articles/Sqoop-to-MS-PDW-Parallel-Data-Warehouse-issue/tac-p/304591 issue is fixed in HDP 2.5.5 version and above but we are still seeing the same issue in HDP 2.6.3. Do we have any workaround for this.   Thanks in Advance!

Hi sir/mam,Good day!I received my Cloudera Certification. There is a typo error in my name. In Certificate : Needhi Devan Senthi KumarMy Name : Needhi Devan Senthil Kumar There is 'l' missing in the certificate name. What should I do for this name correction. It would be a pleasure if any one replies. Thank you.

Currently we have a Hortonworks HDP cluster version 2.6.5. We are planning to migrate our platform to the new Cloudera. Which is the recommended upgrade path? Upgrade HDP first to version 3.1.5 and later migrate to CDP or is also possible to migrate directly from HDP 2.6.5 to CDP?   Thanks in advance for your inputs and suggestions.