Q&A

I am trying to install cloudera 6.3 using postgresql 10 and openjdk 11.0.6When I start the cloudera-scm-server, it fails and i get the below error in the cloudera-scm-server.out. Any ideas?  JAVA_HOME=/usr/lib/jvm/java-openjdkOpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.Unrecognized VM option 'UseParNewGC'Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.  Here is the java version  [root@dex cloudera-scm-server]# java -versionopenjdk version "11.0.6" 2020-01-14 LTSOpenJDK Runtime Environment 18.9 (build 11.0.6+10-LTS)OpenJDK 64-Bit Server VM 18.9 (build 11.0.6+10-LTS, mixed mode, sharing) 

Hi All , I am trying to make a directory as a snapshot directory using curl API ,when i run the curl for Allowsnapshot it throws below error  Any help is Appreciated..Thanks in Advance ..!!  curl -i -X PUT "http://internal:50070/webhdfs/v1/dummy/?op=DISALLOWSNAPSHOT"HTTP/1.1 400 Bad RequestCache-Control: no-cacheExpires: Tue, 14 Apr 2020 03:26:30 GMTDate: Tue, 14 Apr 2020 03:26:30 GMTPragma: no-cacheExpires: Tue, 14 Apr 2020 03:26:30 GMTDate: Tue, 14 Apr 2020 03:26:30 GMTPragma: no-cacheContent-Type: application/jsonX-FRAME-OPTIONS: SAMEORIGINTransfer-Encoding: chunkedServer: Jetty(6.1.26.hwx){"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid value for webhdfs parameter \"op\": No enum constant org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.DISALLOWSNAPSHOT"}}[

Is there something about configuring Nifi that needs to be done when setting up?   I ask because I've worked on a few different systems and the one I just set up seems to be having issues with Regex, prepend, and other items. 

Hi Team, While using the envelope tool, I'm getting below error while using the tool to write into hive table:20/04/08 02:54:32 ERROR run.Runner: Pipeline exception occurred: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;;20/04/08 02:54:32 ERROR yarn.ApplicationMaster: User class threw exception: java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;;java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;; Can you please provide any suggestion to overcome this. Regards,Arvind

I am trying into ingest one table from MS SQL Server to hdfs through sqoop and I get following error.        cause:java.io.IOException: Sqoop does not have the splitter for the given SQL data type. Please use either different split column (argument --split-by) or lower the number of mappers to 1. Unknown SQL data type: -15119/10/30 12:42:37 ERROR tool.ImportTool: Import failed: java.io.IOException: Sqoop does not have the splitter for the given SQL data type. Please use either different split column (argument --split-by) or lower the number of mappers to 1. Unknown SQL data type: -151at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:194)at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:305)at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)at org.apache.hadoop.mapreduce.Job$10.run(Job.

Team ,   I have installed and configured HUE .but i see below error when i try to access HDSF .   Cannot access: /user/admin. HTTPSConnectionPool(host='catl0dlbd00028.corp.cox.com', port=50070): Max retries exceeded with url: /webhdfs/v1/user/admin?op=GETFILESTATUS&user.name=hue&doas=admin (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)",),))     I have logged in to HUE using default admin password . DO i have to configure LDAP before accessing HUE ? . I see its going to /user/admin wheer that folder might not be available in hdfs .

Hello, I want to transfer files from hdfs to kudu.  I tried through talend fabric and its components but I have an error : Cannot run anywhere due to node and executor blacklist.   Can you help me please? Thanks a lot.

Hi, we are facing a strange issue with an ORC external Hive table, some lines can't be retrieved. The context is HDP 3.1.0 with ORC files generated with a Spark 2.3.2 job into an HDFS partition with an external table mapped on this HDFS directory. Table partitions were added. When trying to filtering this table with a specific column value, there's no result;but when accessing these files from the Spark interpreter of a Zeppelin notebook, the filter shows the expected lines. Hive LLAP is not used for this issue and have another behaviour (data is retrieved).   The SQL query (JDBC tool or beeline command) that don't return any result : select * from tracabilite.tg1pivot where sscc="330232926251080606" and `application`="LOGUSI" and month="04" and day="01"; Another SQL query that returns 2 results (within the same partition): select * from tracabilite.tg1pivot where sscc="330232926636794272" and `application`="LOGUSI" and month="04" and day="01"; The Zeppelin notebook sample : val l

Hi, I'm attempting to replace values in an ASCII file that im importing from a website. I've added the header with replace text but, I'm having a hard time replacing the spaces with commas so, I can convert the file into csv and later to json. Here is what I'm dealing with.    Edit: I should add that sometimes there is one space other times there are two spaces. As of now I have two spaces in the search attribute and a comma in the replace attribute works for part of this problem.     Value     Search Value       Replacement Value   ,   Character Set   UTF-8   Maximum Buffer Size   1 MB   Replacement Strategy   Regex Replace   Evaluation Mode   Line-by-Line   Line-by-Line Evaluation Mode   All   Before: Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_v

Team ,   I have installed HUE using the below steps .   https://github.com/steven-dfheinz/HDP2-Hue4-Service.   I was able to install HUE successfully ,but it fails to start . below is the error i am getting .   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/hue_server.py", line 76, in <module> HueServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/hue_server.py", line 26, in start import params File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/params.py", line 194, in <module> webhdfs_url = format('http://' + dfs_namenode_http_address + '/webhdfs/v1') File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration p

Hello,   I'm facing an issue with the display and storage of special charactere in hive. I'm using spark for doing a WriteStream like this in Hive,     // Write result in hive val query = trimmedDF.writeStream //.format("console") .format("com.hortonworks.spark.sql.hive.llap.streaming.HiveStreamingDataSource") .outputMode("append") .option("metastoreUri", metastoreUri) .option("database", "dwh_prod") .option("table", "res_idos_0") .option("checkpointLocation", "/tmp/idos_LVD_060420_0") .queryName("test_final") .option("truncate", "false") .option("encoding", "UTF-8") .start() query.awaitTermination()      but when I have a special charactere Hive doesn't display it. I have already fixe encoding UTF8 in the hive table :     select distinct(analyte) from res_idos_0;+--------------------------------------------+| analyte |+---------------------------

Hi May I know what is the difference between CDP 7.0 product and Cloudera Enterprise 6.3? what are the features missing and what are the advantages?

I'm running the latest version of HDP sandbox using Docker. I'm following the instructions here: Getting Started with HDP Sandbox   When I create a table using the DAS UI, the operation completes successfully, and I am able to use the query tool to query the database. However, if I go to the "database" tab, is says there are no tables in the database. If I try to create a new database, it also doesn't appear on the database dropdown.   Also, only the first query I execute is added to the query log, even though subsequent queries complete successfully.   How can I configure sandbox so that I can view tables and databases properly in DAS?   Thanks.

does hive version 1.1.0-cdh5.16.1 supports acid transaction?

Hello!   In my Envelope pipeline, I need to compare two Hive tables. Instead of hardcoding the tables in the .conf file, I would like to pass which tables I'm going to compare. I tried using spark.yarn.appMaster.varName but it doesn't seem to work. I'm running CDH 5.13.3 with Java 1.8 on a Centos VM.   This is what the script that runs the spark job looks like: #!bin/bash sudo -u hdfs spark2-submit \ --master yarn \ --deploy-mode client \ --conf spark.yarn.appMaster.Env.tableA=dbA.tableA \ --conf spark.yarn.appMaster.Env.tableB=dbB.tableB \ envelope-0.7.2.jar comparison.conf    Part of my .conf file: application{name = comparison} steps{ tableA{ type = hive table = ${tableA} } tableB{ type = hive table = ${tableB}} }    

Hi All,   I install Cloudera in the AWS EC2 ubuntu 16.4 version. successfully installed, but not able to access in the browser. can you help me. how to open a browser. Are there any commands to check in Ubuntu, whether it's is running or not.   Regards, BVNP

Until when does Cloudera can cater and support the 6.3 version? until what year or what is the life span? and what is the most extended life span can Cloudera support it?

Hi, I have a Linux box which i configured HDFS client to connect and able to put files in the directories. Note that the machine is not part of the cluster.   1. Is it possible to run hive queries through hive CLI from this client machine? If so what all configuration i need to to.  2. Need one understanding on pushing file to hdfs. If i am pushing the file to hive table under below location and if it is a clustered ENV. will there be any side effect? As of now mine is a single node cluster so using the below. hdfs dfs -rm -r /user/hive/warehouse/schema.db/tablehdfs dfs -copyFromLocal -f /scratch/mano/table.dat /user/hive/warehouse/schema.db/table   Thanks in advance. Manoj

May I know the difference of Parcel and Packages upgrade? and How do i know what method i need to use? Do I based it on how the CDH installed previously?

Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here. Bootstrap log file :2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running2020-04-04

how do i get authentication credentials (username and password) for Ambari repositories ? 

Hi All   In my CDP environment,  Hive Server 2 is failing to start,Below is the log for the same     0-03-25 08:33:56: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!   of the failure,i am unable to figure out whats causing the failure of the service.

I am trying to get the InvokeHTTP processor to download a csv, the URL is http://www.mambiente.madrid.es/opendata/horario.csv . Problem is, no matter what the options are in the processor, the response is always a 403 error. If I set the "Always Output Response" to true, the response body is: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>403 Forbidden</title></head><body><h1>Forbidden</h1><p>You don't have permission to access /opendata/horario.csvon this server.</p><hr><address>Apache/2.2.22 (Ubuntu) Server at www.mambiente.madrid.es Port 80</address></body></html>   Thing is, a simple "curl --request GET http://www.mambiente.madrid.es/opendata/horario.csv" works just fine, so I have no idea what the problem is.

Hello,   I have recently Kerberized our Hadoop cluster using Apache Ambari and HDFS seems to be working fine. However, YARN ResourceManager and TimelinseServer seem to not be able to communicate:   yarn-yarn-resourcemanager-XXXXXX.log:     2020-03-16 22:58:27,382 ERROR metrics.SystemMetricsPublisher (SystemMetricsPublisher.java:putEntity(549)) - Error when publishing entity [YARN_APPLICATION,application_1581534326709_0003]com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: <a href="<a href="http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn" target="_blank">http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a>" target="_blank"><a href="http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a" target="_blank">http://w.x.y.z:8188/ws/v1/timeline/?user.name=yarn</a</a>>, status: 403, message: org.apache.hadoop.secu

i'm new to NiFi and have developed simple dataflow which would read 3 files from on-prem and upload into S3 bucket. i am using ListFile, FetchFile and PutS3 processor groups for this case. When i trigger dataflow, CPU consumption is spiking >100% and NiFi gets crashed.After few minutes. it automatically comes down to normal. This is happening frequently. My onprem server is Linux and has 8 CPUs. Any idea what's going on  with this.