Solved

Hive's execution engine supports MapReduce & Tez. Does it supports Spark in HDP 3.x. https://issues.apache.org/jira/browse/HIVE-7292

Hi, I have an issue with distcp authentication between a kerberos secured cluster (HDP2.6.1.0-129) and an unsecured cluster. (HDP3.0.1.0-187) The compatibility matrix for the versions says they should interoperate I am running the following from the secure cluster:   hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true swebhdfs://FQDN(secure cluster):50470/tmp/test.sh swebhdfs://FQDN(insecure cluster):50470/tmp/test.sh   Both ends have TLS enabled.I have a truststore configured and working.  I can see packets outbound from the client and arriving at the server using tcpdump   The error returned by the distcp command above is: 19/10/15 09:58:48 ERROR tools.DistCp: Exception encounteredorg.apache.hadoop.security.AccessControlException: Authentication requiredat org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:460)at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:114)at org.apache.hadoop

I installed successfully CDSW 1.6 on 1 Master-Node and 2 Worker-Nodes. Unfortunately I get on both of the Worker-Nodes these error messages when I check the nodes with cdsw status:(On the Master-Node there are no error messages) Failed to run CDSW secrets check.Failed to run CDSW persistent volumes check.Failed to run CDSW persistent volumes claims check.Failed to run CDSW Ingresses check.Checking web at url: http://workbench.company.comOK: HTTP port checkCloudera Data Science Workbench is not ready yetI couldn't see any drawbacks till this time and everything seems working quite nice. But I'm interested why I get these error messages and what they mean.Can someone help please?Thanks!

Hello, I am using Sandbox HDP 3.0, I am having a Datanote error not running (I started it up, it did not report anything but then stopped immediately)I have attached the image file and the log file. Thanks to everyone for watching and helping me fix the mistake. Thank you very much!   File log ************************************************************/2019-10-13 09:56:41,017 INFO datanode.DataNode (LogAdapter.java:info(51)) - STARTUP_MSG: /************************************************************STARTUP_MSG: Starting DataNodeSTARTUP_MSG: host = sandbox-hdp.hortonworks.com/172.18.0.2STARTUP_MSG: args = []STARTUP_MSG: version = 3.1.1.3.0.1.0-187STARTUP_MSG: classpath = /usr/hdp/3.0.1.0-187/hadoop/conf:/usr/hdp/3.0.1.0-187/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/stax2-api-3.1.4.jar:/usr/hd

i am trying connected mysql java connector to my mariadb using this docs https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/prepare_cm_database.html. I am getting this error messege       [root@cloudera java-1.6.0]# /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm my_passwordJAVA_HOME=/usr/java/jdk1.6.0_31Verifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/jdk1.6.0_31/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/util/PSQLException : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.d

We recently upgraded from hdp 2.6.4 to 2.6.5. After the upgrade i am not able to execute this command -  show partitions `<database_name>.<table_name>`   The command works fine with 2.6.4 version. Any help would be appreciated. Thanks!

Good Morning.I am doing a small example with HIVE and MANAGED TABLE. My environment is Oracle VM virtual Box 6.0Hive 1.1.0-cdh5.13.0 I indicate the steps I have taken:1st)CREATE DATABASE scenariox 2nd) USE scenariox CREATE TABLE scenariox(smallint order;name string;)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; 3rd) I load the table with my csv file that is stored in HDFS HDFS> /scenariox/scenariox.csv LOAD DATA LOCAL INPATH '/scenariox/scenariox.csv' INTO TABLE scenariox The table loads correctly. I can run: '' Select * from scenariox ''. I recover data correctly. The table has been loaded successfully.The directories in hive are created correctly: /user/hive/warehouse/scenariox.db/scenariox/scenariox.csv I execute the sentence: SELECT COUNT (1) FROM scenariox;MAP / REDUCE launched   My problem is: I can't find the output file that produces the map / reduce. What directory is it stored in? I thought that a file of type '00000_0' was generated and stored in the HDFS. Can some

I am trying install cloudera 5.16.2(single mode). In cluster instalasion  i am get this error.   Spoiler (Highlight to read) Spoiler (Highlight to read) /tmp/scm_prepare_node.ZzLnmqNEusing SSH_CLIENT to get the SCM hostname: 10.0.0.4 34302 22opening logging file descriptorStarting installation script...Acquiring installation lock...BEGIN flock 4END (0)Detecting root privileges...effective UID is 0Detecting distribution...BEGIN grep Tikanga /etc/redhat-releaseEND (1)BEGIN grep 'CentOS release 5' /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 5' /etc/redhat-releaseEND (1)BEGIN grep Santiago /etc/redhat-releaseEND (1)BEGIN grep 'CentOS Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep 'CentOS release 6' /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep Maipo /etc/redhat-releaseEND (1)BEGIN grep 'CentOS Linux release 7' /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)END (0)/etc/redhat-release ==>

Hi, Below is the logs   Host checks started.Check execute list: db_connection_checkDB connection check started.WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded beforeWARNING: File /var/lib/ambari-agent/cache/mysql-connector-java.jar already exists, assuming it was downloaded beforecall['/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-agent/cache/DBConnectionVerification.jar:/var/lib/ambari-agent/cache/mysql-connector-java.jar -Djava.library.path=/var/lib/ambari-agent/cache org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://host.com/metastore" "hive" [PROTECTED] com.mysql.jdbc.Driver'] {}call returned (1, 'ERROR: Unable to connect to the DB. Please check DB connection properties.\ncom.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.')DB co

Hello,I would like to implement this logic on nifi:I have a json array:[{"att1": "1","att2": "aaa"},{"att1": "2","att2": "bbb"},{"att1": "3","att2": "aaa"},] I would like that, according to the attribute, the json was processed to the processA or to the processB. For example, if att2 is equal "aaa", it was processed to the processA, else it was processed to the processB.I tried to follow the suggestion of the following solution https://community.cloudera.com/t5/Support-Questions/Nifi-processor-route-json-array-into-multiple-json-arrays/td-p/128722 but it didn't work. Can you give me an alternative? Thank you.

Hi, We have  a requirement to retrieve files from SFTP server and place into to different prefixes in a S3 bucket.   Mapping is something like this:   ftp path1 -> file pattern match1 -> s3 prefix1 ftp path1 -> file pattern match2 -> s3 prefix2 ftp path2 -> file pattern match3 -> s3 prefix3 ftp path3 -> match4 -> s3 prefix4   ListSFTP -> FetchSFTP -> RouteOnAttribute  RouteOnAttribute - Match1 (${filename:indexOf('pattern1'):gt(-1)}) - putS3Object1 RouteOnAttribute - Match2 (${filename:indexOf('pattern2'):gt(-1)}) - putS3Object2   We are using the ListSFTP, FetchSFTP and RouteOnAttribute to route it to the a corresponding putS3Object processor with different variables for different patterns.   We initially had only 2 patterns which grew into 10 - 15 patterns (Match1, Match2, .. Match N) to be routed to the corresponding paths. So if we follow the same approach we have to use 10 -15 putS3Object. Is there a way to avoid thi

Hi   I'm new to Nifi and I wanted to orchestrate a data pipeline with micro-services that I have using Nifi. I have two micro-services and I'm using InvokeHTTP and ListenHTTP for them. I have no issue with the invokeHTTP but I'm getting error with the ListenHTTP processor which says "Failed to properly initialize the processor". Can someone help me? Can I use Nifi to orchestrate my data pipeline that has multiple micro-services. Which processor is used to listen to the microservices and how Do i connect them

Hi,I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated. Error starting NodeManagerorg.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)Caused by: org.fus

Hello, Good afternoon.I am new in cluster. I am installing CDH 5.16.2-1.cdh5.16.2.p0.8. I am stuck in hereI am check scm-server log using "tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log"I get this logID '798f1ee7-be79-475c-bc30-28573451693e' that is not recognized2019-10-03 11:48:43,218 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: (27 skipped) Found files KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel,CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel,CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel,CDH-6.2.0-1.cdh6.2.0.p0.967373-bionic.parcel under /opt/cloudera/parcel-repo2019-10-03 11:48:52,417 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel2019-10-03 11:48:52,417 WARN ParcelUpdateService:com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable: Error while executing CmfEntityManager taskjava.lang.IllegalArgumentException: Unknown Distro suffix: bionicat co

I am trying to enable Kerberos on Ambari 2.6.2.2 on CentOS 7.  Below are the errors*********************** 500 status code received on POST method for API: /api/v1/clusters/hdp265/requestsError message: An internal system exception occurred: Failed to execute the command: Broken pipe *********************************** Below is my krb5.conf file nano /etc/krb5.conf # Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/ [logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log [libdefaults]dns_lookup_realm = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = truerdns = falsepkinit_anchors = /etc/pki/tls/certs/ca-bundle.crtdefault_realm = HADOOPSECURITY.COMdefault_ccache_name = KEYRING:persistent:%{uid} [realms]# EXAMPLE.COM = {# kdc = kerberos.example.com# admin_server = kerberos.example.com# }HADOOPSECURITY.COM = {kdc = p1.bigdata.comadmin_server = p1.bigdata.com} [domain_

Hi Team, I want to rename/modify the filename of the flowfile with the timestamp added to it. Let's say the original flowfile name is file1.txt, I want to change the name of file to be file1_<timestamp>.txt. I used updateAttribute processor to do it. The expression which I gave is   ${filename:substringBeforeLast(‘.’)}_${now():format("yyyy-MM-dd HH:mm:ss.SSS’Z’”)}.${filename:substringAfter('.')}   This Giving me error. I'm bad at Expression language. could anyone help me in this?Thanks. 

Hi Team, I have a zipped file which contains so many text files. I used unpackContent processor to unzip the files. I want to store those unzipped files in another location. But the file names should be replaced with file1.txt, file2.txt, file3.txt and so on. How can I give file names dynamically? I thought updateAttribute can be used in this. But I'm not sure at the expression language to get this done. Can anyone help me to write a expression language in updateAttribute processor or any alternative processor/solution for this.   Thanks.

Hello,I am using HDP3.0, Ambari 2.7.1. My MapReduce2 is not running, it reports the following error:   Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 126, in <module>HistoryServer().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 94, in startskip=params.sysprep_skip_copy_tarballs_hdfs)File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/copy_tarball.py", line 516, in copy_to_hdfsreplace_existing_files=replace_existing_files,File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in runself.run_action(resource, action)File "/usr/lib/ambari-agent/lib/resource_management

I am using cloudera VB. I am trying to login to beeline. I tried the username and passwod (empty) but it's not working :-!connect jdbc:hive2://localhost:10000/Connecting to jdbc:hive2://localhost:10000/Enter username for jdbc:hive2://localhost:10000/:Enter password for jdbc:hive2://localhost:10000/:Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)0: jdbc:hive2://localhost:10000/ (closed)>@  [cloudera@quickstart ~]$ beeline2019-10-09 17:06:46,217 WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.Beeline version 1.1.0-cdh5.7.0 by Apache Hivebeeline> !connect jdbc:hive2://localhost:10000/scan complete in 5msConnecting to jdbc:hive2://localhost:10000/Enter username for jdbc:hive2://localhost:10000/: hadoopEnter password for jdbc:hive2://localhost:10000/:Error: Could not open client tra

I am using cloudera virtual box. while creating partitions, it is creating all the partitions whether they are unique or not create table product_order1(id int,user_id int,amount int,product string, city string, txn_date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';LOAD DATA LOCAL INPATH 'txn' INTO TABLE product_order1;Loading data to table oct19.product_order1Table oct19.product_order1 stats: [numFiles=1, totalSize=303]OK Time taken: 0.426 secondshive> > set hive.exec.dynamic.partition = true;hive> > set hive.exec.dynamic.partition.mode = true;hive> > create table dyn_part(id int,user_id int,amount int,product string,city string) PARTITIONED BY(txn_date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';OK Time taken: 0.14 secondshive >INSERT OVERWRITE TABLE dyn_part PARTITION(txn_date) select id,user_id,amount,product,city,txn_date from product_order1;Result which i have received :-Loading data to table oct19.dyn_part partition (txn_date=null)

Aloha!Before, I was able to connect to the web UI via host IP and port 7180. After setting up a Static IP for the hosts in my Cluster, I seem to have broken Cloudera Manager. Should I uninstall and reinstall, or is there a simple solution that I can implement to get things working again?

Hi,   I try to replace text with space using ReplaceTextWithMapping processor. I have file with multiple lines   "action":"account sing up""action":"account sign in""action":"account sign out"   And I have a mapping file with \t separating the search value and replace value     account sing up asuaccount sign in asiaccount sign out aso     Regular expressions "action":"(.*)" Matching group: 1   Result   "action":"account sing up""action":"account sign in""action":"account sign out"   Expected:   "action":"asu""action":"asi""action":"aso"     Does anyone know what I did wrong?   Thanks David

HI Guys,   I upgrade to the latest CDSW 1.6 over the weekend. The upgrade was successful, however i am having trouble lunching a CLI terminal once i have created a project. Every time i do, i get the below error.      This page isn’t working tty-8c51gbduvszddduehl.domain.co.za is currently unable to handle this request. HTTP ERROR 500   My cluster is fully TLS'ed and only having this problem since the upgrade. 

Hello. I've been following this tutorial (https://www.batchiq.com/nifi-configuring-ssl-auth.html) to the best of my ability, yet the web UI still does not want to start. I get the error message ERR_CONNECTION_REFUSED" when trying to access the nifi server through Google Chrome. It has also been this way when trying to access through Firefox, so I suppose this is more of a server error than anything else. The following is a relevant section of my conf file...   # web properties #nifi.web.war.directory=./lib#nifi.web.http.host=#nifi.web.http.port=8080nifi.web.http.network.interface.default=nifi.web.https.host=nifi.web.https.port=8443nifi.web.https.network.interface.default=nifi.web.jetty.working.directory=./work/jettynifi.web.jetty.threads=200nifi.web.max.header.size=16 KBnifi.web.proxy.context.path=nifi.web.proxy.host=     # security properties #nifi.sensitive.props.key=SyoerSecretFlyingPasswordnifi.sensitive.props.key.protected=SuperSecretFlyingPasswordnifi.sensitive.pro

Why isn't the root user showing up on ranger? I intend to give permission to the root user on Hbase but I'm getting the following error:     hbase(main):001:0> grant 'root','RWXCA'ERROR: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 400 Error: root is Not Found at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1261) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1072) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:10023) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10187) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8065) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.ja