Q&A

HDF 3.4.1  NIFI 1.9 - NIFI Provenance Repository filling disc 500GB I have a requirement to retain provenance for 5 days and made necessary changes  - provenance retaining hardly - 2 days and lesscontent_repo - 500GB   utilization 10%Provenence_repo - 500GB Utilization 98% flowfile_repo - 500GB   utilization 10% below are the configs  @MattWho @TimothySpann   Please advice 

Problem Description : Sqoop is pushing data daily from MSSQL to HDFS location in date format like year/month/day (one pull daily for delta of data) and on top of it, created external Hive table. But Past data at source RDBMS is getting updated frequently so sqoop has to pull and merge those changes with already pulled data in HDFS. With that, I want to delete data that is 60 days old. Is there any way to achieve it. Can partitions in Hive resolve it. 

Hi,I tried to get the CDP documentation in pdf using the pdf button on the online document. Its good, but I found out that some of the screenshot image became lot bigger and does not fit within the document(Some is perfectly fine). Is this can be repaired or I'm asking a bit too much.  In this example i used https://docs.cloudera.com/runtime/7.1.1/security-ranger-authorization/security-ranger-authorization.pdf Thank you very much.  the pdf button that i used this is the same page as the 1st picture 

We recently enabled kerberos and ldap to our cloudera 6.3.2 . And able to connect to Hive through JDBC connections successfully. But oozie/sqoop jobs that interact with metastore are failing with below error:-------------------------------------------------------------- javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)at java.security.Ac

Hi folks,  After install Cloudera CDP 7.1.3 we try to access on RANGER CONSOLE --> AUDIT --> ACCESS and we get this error/message"Error running solr query, please check solr configs. Collection not found: ranger_audits"Looking file catalina.out==================================================================================Oct 19, 2020 11:59:39 PM org.apache.ranger.server.tomcat.SolrCollectionBootstrapper connectINFO: Solr is in Cloud modeOct 19, 2020 11:59:39 PM org.apache.ranger.server.tomcat.SolrCollectionBootstrapper uploadConfigurationINFO: Config already exists with name ranger_auditsOct 19, 2020 11:59:41 PM org.apache.ranger.server.tomcat.SolrCollectionBootstrapper createCollection SEVERE: Error while creating collection in solr : org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://XXXXXXXXXXXX:8993/solr: Underlying core creation failed while creating collection: ranger_auditsOct 19, 2020 11:59:41 PM org.apache.range

Hi, Is there is any documentation available for OnPrem HDP to CDP Cloud Migration?Also let me know any document available for onprem HDP to OnPrem CDP migration. ThanksManish

Hello, I want to send and email using PutEmail with czech characters in email html body. I used UTF-8 charset, but it does not work, I'm keep getting emails with question marks instead of correct czech special characters. But subject text shows characters correctly. When checking data provenance, everything is ok.Can you please help me with processor configuration? What am I doing wrong?Screenshots of my setup below. Thanks a lotRgdsLudvik I'm getting message text in incorrect charset    

I have flowfile entries like the following:{\"CommandLine\":\"/bin/sh -c ( LC_MESSAGES=C;export LC_MESSAGES;df -k | awk 'NF == 1 { printf $0; next } { print }' ) > /opt/netiq/netiq/AM/tmp/netiqKykswU\",\"ConfigBuild\":\"1007.8.0010402.9\",\"ConfigStateHash\":\"788983332\",\"Entitlements\":\"15\",\"GID\":\"0\",\"ImageFileName\":\"/usr/bin/bash\",\"MD5HashData\":\"0883bc66ffeba0e89ee0a753ddbc1950\",\"NDRoot\":\"313466589192688920\",\"ParentBaseFileName\":\"agent\",\"ParentProcessId\":\"313467877387179481\",\"ProcessEndTime\":\"1602877133.023\",\"ProcessGroupId\":\"313466594618021306\",\"ProcessStartTime\":\"1602877133.016\",\"RGID\":\"0\",\"RUID\":\"1420\",\"RawProcessId\":\"52404\",\"SHA1HashData\":\"0000000000000000000000000000000000000000\",\"SHA256HashData\":\"c1c73170d33c0319a795bb05b1dbe7884b7582ec9f696d8a795e0ce692cb714e\",\"SVGID\":\"0\",\"SVUID\":\"1420\",\"SessionProcessId\":\"313467877387179481\",\"SourceProcessId\":\"313467877387179481\",\"SourceThreadId\":\"0\",\"TargetPr

I've recently created the following table in Impala....create table hpchistperf.oozie_job_stats(name string,start_time timestamp,end_time timestamp,rows_inserted integer,last_id_loaded integer,primary key(name,start_time))PARTITION BY HASH PARTITIONS 16stored as kudu; I read that there was no problem creating the table in Impala and it could be accessed in Hive since they share the same metadata. The problem is that when I run the query on Hive it gives me the following error:FAILED: RuntimeException java.lang.ClassNotFoundException: org.apache.kudu.mapreduce.KuduTableInputFormat

Hello, we are using Apache NiFi to process CSV data that contains time stamps without a specified time zone.  By default, it seems that NiFi considers timestamps parsed by CSVReader with a parse specification like "yyyy-MM-dd HH:mm:ss" to be in UTC, no matter what the user.timezone property of the JVM or the TZ environment variable is set to.  Is there a way to force the parsing to happen in a specific or the system time zone instead? Thanks,Hans

Hi all, I want to build a flow which brings up information about all versioned processgroup on this system. What I need is:- name of each processgroup- version of each processgroup- timestamp when this version was brought to this system Now I tested with NiFi-REST-API but I can not find the right request to do this.Could someone help me to get this works?Thanks.

Hello Team, We are planning to Impala UDFs and UDAs. To try things out we started by exploring the examples given in the cloudera github repo. After building the .so file using the make utility when I try to create the function using the following create function has_vowels (string) returns boolean location '/user/hive/udfs/libudfsample.so' symbol='HasVowels';     We are getting the following error ERROR: AnalysisException: Could not load binary: /<hdfs_path>/udfs/libudfsample.soUnable to load /var/lib/impala/udfs/libudfsample.9909.1.sodlerror: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /var/lib/impala/udfs/libudfsample.9909.1.so)  Following are the error logs org.apache.impala.common.AnalysisException: Could not load binary: /user/parth.khatwani/udfs/libudfsample.soUnable to load /var/lib/impala/udfs/libudfsample.9909.1.sodlerror: /opt/cloud

Hi, I want to enable an on-demand metadata feature for impala as it gives a lot of improvement.   one of the features is that the cached metadata gets evicted automatically under memory pressure.    I want to know if this feature can be enabled in Cloudera CDH 5.14.4 version.   Has anyone implemented this on prod on CDH 5.14.4 version?

After running Impala query  select distinct(partition_date) parts from mddb_servt; I am getting below error: Execution time 10 seconds2) java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:ExecQueryFInstances rpc query_id=937d334667doe010:4967d4b900000111 failed: Failed to get minimum memory reservation of 68.00 MB on daemon ec2-3-128-13.us-east-2:22000 for query 937d334667doe010:4967d4b900000111 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage:Process: Limit=60.00 GB Total=50.68 GB Peak=50.70 GB  Buffer Pool: Free Buffers: Total=0  Buffer Pool: Clean Pages: Total=2.31 GB  Buffer Pool: Unused Reservation: Total=-2.29 GB  Free Disk IO Buffers: Total=1.10 GB Peak=1.15 GB  RequestPoo

Hi!I am running a scheduled job that consists of an insert-select query in hive 3.0/hdp 3 as the following Insert into table t1 as select * from t2 where timestamp > "predefined timestamp"The job was running flawless until out of sudden it started failing with the following error:Caused by: java.lang.IllegalArgumentException: bucketId out of range: 4147[2020-10-13 06:58:12,214] INFO - at org.apache.hadoop.hive.ql.io.BucketCodec$2.encode(BucketCodec.java:94)[2020-10-13 06:58:12,214] INFO - at org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.<init>(OrcRecordUpdater.java:271)[2020-10-13 06:58:12,214] INFO - at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.getRecordUpdater(OrcOutputFormat.java:278)[2020-10-13 06:58:12,214] INFO - at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordUpdater(HiveFileFormatUtils.java:350)[2020-10-13 06:58:12,214] INFO - at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getAcidRecordUpdater(HiveFileFormatUtils.java:336)[2020-10

So, we recently moved to using spark as HIVE engine instead of mr and we are seeing some significant improvements in certain queries which needs intermediate tables/storage to process. Can someone provide a complete or optimum list of configuration that we should be using that will not cause memory issues but still be able to get the best out of using spark as engine?Nodes : 30Cores: 16Memory: 112GB /nodeHadoop 2.6.0-cdh5.13.0Hive 1.1.0-cdh5.13.0

I am trying running nifi on AWS with a ELB fronting it.  I use ListenHTTP (port 7777)and need to do a health check via ELB.  It seems listenhttp only supports HEAD and POST.  ELB only supports health check via GET.  Any ideas how I can accomplish this to determine health status of port 7777 using ELB health checks?

hi all   We are performing now the change hostname configuration on production cluster according to the document - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_changing_host_names.html The last stage is talking about – "in case NameNode HA enabled , then need to run the following command on one of the name node"  hdfs zkfc -formatZK -force  since we have active name node and standby name node we assume that our namenode is HA enable ?  but we want to understand  what are the risks when doing the following cli on one of the namenode hdfs zkfc -formatZK -force is the below command is safety to run without risks ?

Hi everyone,I am new to Ambari and have just installed Ambar 2.7.4 with HDP stack 3.1.4 with a cluster having 1 node running the Ambari Server, one being the master node of all service components and there is one more. The installation completed with few warnings, all the services on the web UI are running with green light on. The problem is an alert that says.. SmartSense Bundle Capture FailureLast SamrtSense bundle was not successful. Bundle key.unspecified_xxxxxx_0_2020-10-04_02-00-01, status:failedAlso, the Activity Analyzer stops again and again after restarting it. All other services are running.PS: I have just installed this and haven't loaded any data or anything or tested anything.Please provide help!                                        Thankyou!@jsensharma  @MattWho @Shelton @nsabharwal 

I was curious if it was possible to move flows from one bucket to another and keep the version history in Registry. The way we have our environment set up today is we just have one bucket that we use in our DEV environment, as well as in production. It gets messy and we would like to clean it up. Here is my idea: We have a bucket for each, DEV, Prod, & Archive. We keep versions controlled for the development process, if any, and when its ready to be deployed to another environment we can the move the flow versioning info over to the production bucket. And then if a flow is old but we don't want to delete it, we can then move it over to the archive bucket.  The DEV to Prod bucket is not as important because we frequently are updating things that are in prod through DEV so it wouldn't be necessary to always move it back into the dev bucket since prod still tracks the version, however, the archive portion is important to us. If moving flows from bucket to bucket is

I have installed a an HDF v3.3 cluster. everything seems to be working fine except Schema Registry which starts fine but then shuts down immediately giving the following errors:The mysql drivers have been loaded fine in ambari and work for all other modules.java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource at com.zaxxer.hikari.util.PoolUtilities.createInstance(PoolUtilities.java:141) at com.zaxxer.hikari.util.PoolUtilities.initializeDataSource(PoolUtilities.java:153) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:146) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:113) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnect

From this document it seems the query results can be redacted however I don't see the same. Is there a known bug?https://www.hadoopandcloud.com/hadoop/enableconfigure-log-and-query-redaction/ 0: jdbc:hive2://calingita.pheonix.co> select * from pheonix_global.redaction;INFO : Compiling command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103): select * from pheonix_global.redaction;INFO : Semantic Analysis CompletedINFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:redaction.id, type:int, comment:null), FieldSchema(name:redaction.email, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103); Time taken: 0.322 secondsINFO : Executing command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103): select * from pheonix_global.redactionINFO : Completed executing command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103); Time taken: 0

Hello all, I tried to search Hive related topics but did not find the issue I face so I wanted to ask here. This is CDH 6.3.2 (Parcels) on Linux (CentOS) I have Kafka and Hive and I am able to create external Kafka tables after putting hive-exec-3.1.3000.7.1.3.0-100.jar and kafka-handler-3.1.3000.7.1.3.0-100.jar in Hive under Hive Auxiliary JARs Directory However, when I try to select data from the external table I get this error: 0: jdbc:hive2://localhost:10000/testdb> select * from kafka_table;Error: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;JJ)V (state=,code=0) After looking for this error I've found that it is most likely related to issues with different versions of guava in Hadoop and Hive: https://issues.apache.org/jira/browse/HIVE-22915 I see that Hadoop is using  /opt/

I'm trying to identify why I am seeing the error given in the subject of this question. The error appears nightly on each host running cloudera-scm-agent (6.2).   I suspect the following: When cron.daily is run, the host monitor appears to start monitoring the disk_usage of /run/user/0. Due to the temporary nature of /run/user/*, when the directory is removed the cm-agent throws 'No such file or directory' errors into /var/log/messages.   Has anyone else encountered these errors? What's the rationale of monitoring /run/user/*? Could this be handled better?   Thanks, Deaks

I'm testing the CDH upgrade from v5.16.1 to v6.3.3. Cloudera manager server v6.3.3 upgrade was done successfully. But I have a trouble in Upgrading the Cloudera Manager Agents section. https://docs.cloudera.com/cdp-private-cloud/latest/upgrade-cdh/topics/ug_cm_upgrade_agent.html I can see The Upgrade Cloudera Manager Agent Packages page. But the Install agents fails due to Failed to copy installation files. I tried both repository location of Cloudera repository and custom repository but the both was failed.   I think inter hosts ssh communications are no problem because I tested ssh login b/w them and I confirmed the /tmp/scm_prepare_node.XXXXXXXX was made on target agent host. I suspects key handling is the cause but I'm not sure how to confirm it. Here's the cloudera-scm-server.log. (modified URL because it's raw username and password)   2020-10-06 14:54:10,500 INFO NodeConfiguratorThread-12-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prep