Solved

Hi community,what is the reason for hive dependency on pig during installation process in ambari ?Br,Dennis

i have json array like the following: [{"id":"one"}, {"id,"two"}] Any idea how could i "package" it into json "root" like the following: { [{"id":"one"}, {"id,"two"}] }I am trying to use JolttransformJSON but couldn't find any clues how to approach operation like this....Thank you

Here is my Hbase log : 2019-04-26 10:39:09,736 ERROR [Thread-16] master.HMaster: Failed to become active masterorg.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=READ, inode="/apps/hbase/data/hbase.version":root:hdfs:-rw-------at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:261)at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:589)at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:377)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1857)at org.apache.hadoop.hdfs.server.namenode.FSDirectory

I fetched the data from hbase using fetchHbaseRow and the response is in json format. but the value for the attribute is array of objects and it is having escaped characters in there.A sample response is posted below:{"someValues":"[{\"a\":\"6f7f8ffe-4837-4fc1-a5b9-45bc35727b69\",\"b\":\"da_states\",\"doc\":\"unknown\",\"idNew\":\"2846e8e1-5b17-490c-a99e-df1f1e3fa009\",\"status\":\"Approved\",\"scope\":\"Global\",\"dateModified\":\"2019-04-19T11:26:02Z\"}]"}I need to split the objects in array and because of escape characters I am not able to.@Matt Burgess

I'm trying to use regex to match the line:# DOG CAT BIRDNiFi regex seems to treat # as a comment character, whereas other regexes do not.How do I match the literal character # in a string with regex?The doc page https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escaping doesn't seem to help me out here.

I can't find a definitive answer (yet) to this question:Is "Partitioned By" compatible within the context of a CTAS type query?  I want to create a partitioned table using a CTAS type query using the Map Reduce engine.  I could use Spark but typically use Map Reduce because of the volume of data.  I do not have Impala available to me.  This documentation appears to suggest that I can (https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_create_table.html), but I can't get it to work (the documentation may pertain only to Impala which is not installed in my shop).  See specific documentation snippet, below. I'm running CDH 5.10.  See additional details (far) below. Thanks,  Jim CREATE TABLE AS SELECT:CREATE [EXTERNAL] TABLE [IF NOT EXISTS] db_name.]table_name [PARTITIONED BY (col_name[, ...])] [COMMENT 'table_comment'] [WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)] [ [ROW FORMAT row_f

Hi , I am upgrading from Spark 1.6.0 to Spark 2.1 on CDH 5.10 platform. I am trying to run spark2-submit command for python implementation and it is failing giving below error. Looks like it is expecting some path property while initilization and creating SparkContext object which is not happening. Error details are as below. Please suggest if any specific configuration is missing or required for spark2. sc = SparkContext(conf=conf)File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_initFile "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_contextFile "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/p

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).Regards

hi allI use the following API , to stop the SPARK2_THRIFTSERVER , on specific node - node2 curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://node2:8080/api/v1/clusters/HDP/hosts/node1/host_components/SPARK2_THRIFTSERVERafter the API performedhow to know if service stooped ?

Can we put data into HDFS from the nifi which in not part of any clusterI have HDP-Ambari with 4 node cluster, and I have installed nifi as standalone server in the master node machineCan get the data from my local machine and store them in HDFS ?

Will you please please help me to configure new database.By default the spark-sql is connecting to the hive, so i want to configure new database like postgreSQL instead of hiveit will be very help full to rectify for me

Hello, I am trying to install Spark 2.3.0 release 4, but after placing csd (jar file) to /opt/cloudera/csd i cannot add it from service. I checked on this url http:xyz:7180/cmf/csd/refreshand got the following error: placeholderHandlers":[],"invalidCsds":{"/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar":["Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist"]},"repoEnabled":true}}I followed these steps:a. Download the spark 2.3.0 jar b. Upload the CSD to /opt/cloudera/csd in the Cloudera Manager server.c. Change the owner and group for the JARd. restart cloudera scm server and management service Can anyone help ? I have CDH release 5.16.1-1.cdh5.16.1.p0.3

Hi Team,We upgraded our spark to 2.3.0.2.6.5.0-292. When I export the spark_major_version=2, and if I type spark-shell then it will be connecting to 2.3.0.2.6.5.0-292. I want to use spark 2.1.0.2.6.0.23-2 version, how can I do that. Please help us on this.

I have a Cloudera CDH cluster currently running on Oracle JDK, and I need to know the proper process to migrate the entire cluster over to OpenJDK.

Hi Team, Greetings! There was a need to move the kudu installation from my current cluster of AWS machines to new cluster of  AWS Machines.So I did the following steps.Took the back/sanpshot of disk haivng the kudu data Directory.Installed CDH and kudu on the new AWS cluster and attached the Disk from the older cluster to the newer one.After the installation I checked for kudu health in Cloudera Manager, kudu cluster was healthy and all new tablet servers were running fine along with master.when I ran the ksck command all except few the tables were in CONSENSUS_MISMATCH stateLogs suggest the that master is still trying to refer to the tablet servers which is having IP of older machines.Can anybody help in pointing out why this is happening.  RegardsParth 

How can I trigger sample Apache Spark and PySpark jobs using Apache Oozie?

Changes that i tried but did not help:set home directory for javachanged the warehouse root directorycreated the znode for hiveserver2 manually (but it returns empty set)reinstalled zookeeper and hive againstderr: /var/lib/ambari-agent/data/errors-252.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module> HiveServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 53, in start hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service wait_for_znode() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrappe

Hi , We are using 12 Datanodes in our cluster. And we have 3 replication factor. Currently we are doing cost optimization and our total cluster is 35% used. We want to remove one datanode from our cluster, but we want to see first how our cluster performs on 11 datanodes. if it doesn't perform well, we need to restart that datanode again.To experiment this- Do I need to Stop the datanode or Decomission datanode?Please need suggestion.

I am fetching data from pastel(pervasive) database.But every other day, i used to get error--->LNA session timeoutThen i have to go the connection setting and when i disable the connection and then enable it, everything works fine.Is there any way that i can automate the process of disable then enable the connection.

Hi All, does the "Balancer" role itself use any resources?  Does it matter which host the role resides (e.g. NameNode, Edge Node, etc)? Thank you.

hello folks , the nodeManager has suddently stopped in a instance (while stille running for other nodes/intances ).so when I try to start/restart it -via cloudera manager - , an error is shown in the first step : Failed to start role.and I'm using CentOS release 6.10 (Final)please what do you suggest me to look or check in order to resolve this problem ?here's my stdout log : Tue Apr 23 10:18:56 PDT 2019JAVA_HOME=/usr/java/jdk.1.8.0_144using /usr/java/jdk.1.8.0_144 as JAVA_HOMEusing 5 as CDH_VERSIONusing /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-yarn as CDH_YARN_HOMEusing /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-mapreduce as CDH_MR2_HOMEusing /var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGER as CONF_DIRCONF_DIR=/var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGERCMF_CONF_DIR=/etc/cloudera-scm-agent## A fatal error has been detected by the Java Runtime Environment:## SIGBUS (0x7) at pc=0x00007f8c1fde51a1, pid=300

Installed kafka broker in a node using ambari blueprint with hdp 2.6.5 with ranger-kafka plugin enabled. Created topics using kafka-topics.sh command. Getting TOPIC_AUTHORIZATION_FAILED warning and unable to produce messages. By going through this document (https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-Authorizingtopiccreation ), got to know that ranger-kafka plugin authorization works only for autocreated topics. Tried the same producer command by auto-created topics but still getting the same error. Could anyone please help me to resolve this.

Hi,I would like to use the yarn node label functionality. For this I need to create a yarn node label as user yarn.When I try 'su yarn' it is asking me for a password.How is it possible to change the yarn password within HDP ?Br,Dennis

We have Hadoop cluster based on ambari Since thrift server have poor performance , we decided to replace it with presto Our current Hadoop cluster have the following machines 960 data node machines ( based on redhat 7 OS )Few words about the presto- Presto (or PrestoDB) is an open source, distributed SQL query engine, designed from the ground up for fast analytic queries against data of any size. It supports both non-relational sources, such as the Hadoop Distributed File System (HDFS), We installed the new presto server as the following First we installed the OS ( redhat 7 ) , total 13 machines 1 machine for the presto coordinator And 12 machines for presto workersAfter installing the OS We installed successfully the presto ( presto coordinator + presto workers ) Now we are stuck about how to do the integration between presto cluster to the Hadoop clusterI will give short example about hive connector ( hive.properties )we have the following variable hive.config.resources=/etc/hado

Hi all,I had built up a HDF 3.3.1.0 with ambari, but I got NoClassDefFoundError when I'm building a data processing kafka to druid by Streaming Analytics Manager, how to fix it?My environment:Storm:1.2.1SAM: 0.6.0Kafka: 1.0.0.3.1Druid: 0.12.1Below is the logs in storm worker.java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingExceptionat org.apache.storm.kafka.spout.KafkaSpout.open(KafkaSpout.java:142) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.3.1.0-10.jar.1555487788000:1.2.1.3.3.1.0-10]at org.apache.storm.daemon.executor$fn__10149$fn__10164.invoke(executor.clj:615) ~[storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) [storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingExceptionat java.net.URLClass