Q&A

We recently upgraded Kafka to 2.3.0 after that one of our connector is failing with the below error.  org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.connect.errors.SchemaProjectorException: Switch between schema-based and schema-less data is not Any ideas anyone ?I have already looked and implemented this https://issues.redhat.com/browse/DBZ-235 but no luck. Nothing has changed in the schema. We dont think its a tombstone issue. I was able to check where the connector stopped and looked at the events and its not a tombstone. 

Hello community,   I'm writing a Python script, and connect through the 64-bit odbc driver to Hive and Impala. I'm interested in creating an external table using the Hive connection, and then run some faster-than-hive queries using an Impala connection. After table creation, I am able to see and query the external tables in both hive and impala editors in HUE. However, these newly created tables are not visible nor available through the Impala connection.   Tried the    invalidate metadata <table_name>   but it's not working either.   Any pointers would be greatly appreciated.   Luis

Hi All, I have implement almost 300 lines of hive query. The query template is as below. Insert into targetTableSelect col1,col2...........cols....(Almost 300 lines in select query with lots of case when statemnet)From source table My observation is that it works without giving any error in apache zeppeline. But when i trying to execute it through beeline it's failing with  transport exception response code 500. Other observation,when i remove few number of columns from select query in that  also it works fine. I do not understand why query is failing ? I need to know the reason behind failure? Thanks in advance  

Hi All, I have a flow setup where I am fetching some data from an Oracle DB. Some of the columns of the table are in Oracles Number type, and I need to put them into a Postgres DB as floats. If I look at the Schema that is being created on read from Oracle it looks like it's the data that's being defined as a type of "Bytes" with a logical type of "decimal" that I am talking about. Currently I am using the following flow as I need to add some additional update onConflict logic to the Postgres insert to handle updates. Oracle -> Convert JSON -> Split Record -> Convert JSON to SQL -> Replace Text -> Execute SQL But at the moment this throws an error when trying to do the insert into Postgres, which seems to be related to the number not being coerced into an actual number and rather been empty in the SQL. Anyone got any ideas/suggestion on how I can force the type of the numerical types to be a float in the Avro and then a number type once

I am using spark 2.3.2 and i am trying to read tables from database. I established spark connection. But i am unable to read database tables from HUE cloudera and unable to query them in pyspark as well.   Here is my code, import findsparkfindspark.init('C:\spark-2.3.2-bin-hadoop2.7\spark-2.3.2-bin-hadoop2.7')import pysparkfrom pyspark.sql import SparkSessionspark = SparkSession.builder.config("hive.metastore.uris", "thrift://10.1.1.70:8888").enableHiveSupport().getOrCreate()#spark.catalog.listTables("tp_policy_operation")import pandas as pdsc = spark.sparkContextsc from pyspark import SparkContextfrom pyspark.sql import SQLContextsql_sc = SQLContext(sc)SparkContext.setSystemProperty("hive.metastore.uris", "thrift://10.1.1.70:8888")spark.sql("SELECT * FROM tp_policy_operation")   ######The error i am getting Traceback (most recent call last): File "<ipython-input-4-8f0aa5852b01>", line 16, in <module>spark.sql("SELECT * FROM tp_policy_operation")## Database ? File

Hi Team, We are unable to start Hbase.   We are using Hadoop 3.0.1 on Isilon One FS Error Message :   Creating: Resource [source=null, target=/apps/hbase/data, type=directory, action=create, owner=hbase-xxxxx, group=null, mode=null, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystemException occurred, Reason: Unexpected error: status: STATUS_MEDIA_WRITE_PROTECTED = 0xC00000A2 with path="/apps/hbase/data", username=hbase-xxxxx, groupname=org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected error: status: STATUS_MEDIA_WRITE_PROTECTED = 0xC00000A2 with path="/apps/hbase/data", username=hbase-xxxxx, groupname= at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)

Hi,Do we need to stop or put all hadoop services in maintenance, during our DB patching activity or just putting Cloudera Manager in maintenance mode will suffice? ThanksAnm

Hello guys,I am extracting attributes from the flow file and putting it to the PostgreSQL.2017-04-27 09:15:04.794761 10.69.1.138 -> 10.69.1.100 <4>Apr 27 03:44:58 kernel: [ 542.579606] TS|242268|00:14:5a:03:29:72|RSSI_BCN|56 2017-04-27 09:47:59.351047 10.69.1.138 -> 10.69.1.100 <4>Apr 27 04:17:43 kernel: [ 2517.159922] TS|2216852|00:14:5a:03:23:01|RSSI_BCN|53 I am trying to extract integer value 56,53... end of the line and put it to the column<col> of PostgreSQL table<tab> where the <col>datatype is integer.  I used GetFile->SplitText->ExtractText ->putSQL in my process flow. I used (\d*)\z regular expression in ExtractText processor and its seems working well for extracting the value 56.  But after a certain time the putSQL processor shows error ERROR: invalid input syntx for integer:"". If I change the datatype from integer -> text in the <col> of my PostgreSQL table then I don't face this problem but I need to store

I have installed Hue 3.8.1 from tarball and it's running on centos 7. I am not sure how to start hue.    Below is the directory structure.   [root@ds hue-3.8.1]# ls -ltotal 56drwxrwxr-x. 21 sitadmin sitadmin 4096 May 4 2015 appsdrwxr-xr-x. 3 root root 17 Mar 10 23:14 builddrwxrwxr-x. 5 sitadmin sitadmin 58 May 4 2015 desktopdrwxrwxr-x. 6 sitadmin sitadmin 142 May 4 2015 docsdrwxrwxr-x. 3 sitadmin sitadmin 24 May 4 2015 ext-rw-rw-r--. 1 sitadmin sitadmin 11358 May 4 2015 LICENSE.txt-rw-rw-r--. 1 sitadmin sitadmin 4715 May 4 2015 Makefile-rw-rw-r--. 1 sitadmin sitadmin 8505 May 4 2015 Makefile.sdk-rw-rw-r--. 1 sitadmin sitadmin 3498 May 4 2015 Makefile.vars-rw-rw-r--. 1 sitadmin sitadmin 2192 May 4 2015 Makefile.vars.privdrwxrwxr-x. 2 sitadmin sitadmin 21 May 4 2015 maven-rw-rw-r--. 1 sitadmin sitadmin 801 May 4 2015 NOTICE.txt-rw-rw-r--. 1 sitadmin sitadmin 1562 May 4 2015 READMEdrwxrwxr-x. 5 sitadmin sita

How to solved the recurring Heap size issue? we just add size on the heap size but do we have any other solution? is it manageable now in 6.3 version of CDH?

Hi May I know please if what are the versions of the components like Spark, Hbase etc under the new 6.3 cloudera enterprise?

Used case :  Need to transfer entire folder of files based on date to S3. Ex :  Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv             S3    : /Users/datastore/20200312/gyyy.csv   Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder

Hi,   I'm running Cloudera 5.16.1 on CentOS 7 and OpenJDK8 I enabled TLS/SSL on the Cloudera Manager (level 1 - level3) with Self-Signed certificate and then moved to enable Kerberos with AD. I followed the docs but enabling Kerberos is failing when trying to import credentials.   /usr/share/cmf/bin/import_credentials.sh failed with exit code 1 and output of <<+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf1018223695564634823.keytab+ USER=cdhadmin@<XXXXX>.COM+ PASSWD=REDACTED+ KVNO=1+ SLEEP=0+ RHEL_FILE=/etc/redhat-release+ '[' -f /etc/redhat-release ']'+ set +e+ grep Tikanga /etc/redhat-release+ '[' 1 -eq 0 ']'+ '[' 0 -eq 0 ']'+ grep 'CentOS release 5' /etc/redhat-release+ '[' 1 -eq 0 ']'+ '[' 0 -eq 0 ']'+ gr

Hi, I am trying to obtain data ingestion lineage of Nifi via Cloudera Navigator. I have not come across any solution or documentation for the same. Has anyone experimented with this successfully?  Is there an existing solution for this?

Hi, I tried using the Data Warehouse service in my CDP account. What i came across is that Data Warehouse in the backend creates a EKS cluster in your AWS environment. But i was unable to access this EKS cluster using kubectl. Is this use case blocked or I am missing something. I tried accessing this EKS cluster with admin credentials of my AWS account but was still unable to access it.  Can anyone help me with this.

Small and empty files are recurring on our current version of CDH Cluster. Does is still exist on 6.3.3 version?

I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3.  While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive.

Good night people Is there an easier way to install the hadoop ecosystem using the cloudera solution? , I am trying to install version 6.3 for a proof of concept, but I am having difficulty, I remember that before I had a VM that made installation a lot easier, but no longer, could someone help me?

I have this flowchart in nifi, where json files (ConsumerKafka) are consumed and then pass them in mongodb   the first Json file is:   {"_id": "Eems_20200312", "Name": "Eems", "Date": "2020-03-12 23:14:02", "Volatility ": "5,328", "Value risk": "12,395"}{"_id": " Esprinet_20200312", "Name": " Esprinet ", "Date": "2020-03-12 23:14:02", "Volatility ": "5,654", "Value risk": "12,765"}{"_id": " Eurotech_20200312", "Name": " Eurotech ", "Date": "2020-03-12 23:14:02", "Volatility ": "5,534", "Value risk": "12,832"}   the second Json file is:   {"_id": "Eems_20200312", "Name": "Eems", "Last": "0,066", "%": "-9,04", "Hour": "17:26", "Min": "0,061", "Max": "0,069", "Open": "0,068", "Date": "2020-03-12 23:14:02"}{"_id": "Esprinet_20200312", "Name": "Esprinet", "Last": "3,495", "%": "-14,86", "Hour": "17:35", "Min": "3,47", "Max": "3,89", "Open": "3,89", "Date": "2020-03-12 23:14:02"}{"_id": "Eurotech_20200312", "Name": "Eurotech", "Last": "5,05", "%": "-12,17", "Hour": "17:36

Below is the text and I need to write the regular expression so that I can extract attribute from this field.  2017-04-27 09:07:20.517342 10.69.1.138 -> 10.69.1.100 <4>Apr 27 03:37:13 kernel: [ 78.294310] TS|4294745280|00:14:5a:03:29:33|RSSI_BCN|34   timestamp:2017-04-27 09:07:20.517342  source_ip:10.69.1.138 dest_ip:10.69.1.100 mac:00:14:5a:03:29:33 obm:RSSI_BCN dbm:34 How can I write the regex sothat I can collect each of the field from the text?

Hi Team,   We have 12 nodes cluster hosted on premise in 2 different regions. The question is how can we keep the data in 2 data centers in sync and what will be the latency.   Appreciate your help.   Thanks & Regards  

Hi All,   When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following: Role Log File Stdout Stderr Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error: HTTP ERROR 403 Problem accessing /cmf/process/335/logs. Reason: Unexpected end of file from serverThe server declined access to the page or resource.   Powered by Jetty:// 9.4.14.v20181114   Same error is observed when attempted fetching logs using CM API.   Environment: CDH 6.2 OS Redhat 7.7  

Dear community,   After downgrading CDH 6.1 from Enterprise to Express edition, is it possible to upgrade again to Enterprise using same license file used in Enterprise edition?   Our problem is that we want to add additional data nodes to cluster, which will result in exceeding valid licensed node count. So we are thinking to downgrade to Express, add nodes, and wait for procurement for licenses 🙂  

Hello, I am at below screen where I have to give host names.   http://localhost:7180/cmf/express-wizard/hosts   I have installed VirtualBox on Mac OS and using Ubuntu 16.   Have installed below cloudera-manager. wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin   I am confused what should i give in hostname in the current screen (http://localhost:7180/cmf/express-wizard/hosts) I tried 127.0.0.1 / localhost/ and ip addres which i got from ifconfiug but none of them is working Your help will be highly appreciated !!!!

I have a column in a table and value is (1;13004211,13004211_02_13004212,4000000003378605589,1105,2000,)need output like   Column 1 Column 2 Column 3 Column 4  Column 5 13004211 133004211_02_13004212 4000000003378605589 1105 2000   Note: I don't  want the first 1;