Q&A

Hello,   had someone a Solr (Clouder Search) configuration running that index all of the Cloudera logs? Is this possible? I would like to have a better log analyzing and alerting than manual checking in Cloudera Manager...   Regards Oliver

I double that, but I check here anyway. Our system is on CDH-5.15.2. The resource manager and job scheduler is YARN:$ yarn versionHadoop 2.6.0-cdh5.15.2Subversion http://github.com/cloudera/hadoop -r c97bcbf0cba923467d45f5519b1953f436c64f12Compiled by jenkins on 2018-11-13T13:53ZCompiled with protoc 2.5.0From source with checksum 9d2d5b887383c7d4b811372f867c6440This command was run using /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/jars/hadoop-common-2.6.0-cdh5.15.2.jar There are two users, they want isolated environment to run some experiments. I wonder if we can reserve two nodes for their use.  Any suggestions and discussion are very welcome.  Best Regards,Vincent

Hive partition – I have set 10 partition in a table, how can I reduce it to 6 partition without dropping table from HDFS?

Hi, I am trying to figure out how to run a sample Apache Hive query using Apache Oozie, but I'm not sure how to set it up or what the best approach is. Could someone guide me on how to do this?

How can I configure and setup a load balancer for my HUE and OOZIE services?

We have a data flow wherein standardized transaction logs are generated as xml files in a particular directory in VM. We want to publish these logs to a Kafka topic. Following NiFi/MiNiFi flow is what we are using:ListFile ---> FetchFile (completion strategy - Move File) ---> PublishKafkaEach transaction translates to single xml file, at peak hours, over a million files are getting generated per hour, i.e. approximately 300 files per second. Our goal is achieve the above flow using MiNiFi.ListFile uses two listing strategies, tracking timestamps and tracking entities. Initially data flow was created with by selecting tracking timestamps option. Here we observed that several files were not picked by NiFi / MiNiFi. The files that were picked were eventually moved to a different directory but some files were not picked. To put a number to this, 2% - 5% of files were not picked (thus not published to Kafka topic). The behavior was observed in both NiFi and corresponding YAML file in

Good Afternoon,I am creating an apache nifi flow in which I want to flow files from Oracle DB to IBM/DB2, I have completed that part of the flow. I need help with configuring a way to reconcile failed files, I want to know exactly which files failed and the error which caused it. I'm considering using a PUTHDFS to put the failed files into a HIVE TABLE that way I can capture the flow file error. I would then re-enter the failure files back into the flow via GETHDFS.So, I'm curious is this possible? Also is there a mechanism in NiFi to identify the singular file that fails in a batch?

Hello,I am new in Sandbox HDP, I am trying to connect to Hive using JDBC, I used the JDBC URL given by Ambari, and here is my code:private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";         public static void main(String[] args) throws SQLException {        try {            Class.forName(driverName);        } catch (ClassNotFoundException e) {            e.printStackTrace();            System.out.println(e.getMessage());        }        Connection con = DriverManager.getConnection(            "jdbc:hive2://sandbox-hdp.hortonworks.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2");}However, I got this error:Exception in thread "main" java.sql.SQLException: Invalid URL: jdbc:hi

  Dear all,   I got the following error in my HDFS datanodes. However , kinit with the keytab files that generated by the CM is fine and java security policy has been applied.   Is there anyway to diagnose such a problem?   For more information: no LDAP is set in the cluster nodes. cloudera manager and CDH version 5.13.2.   javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757) at java.security.AccessCont

Hi everyone, please help!I'm using ListSFTP processor to get files and process them. In the first execution the processor take the files normally, but when you start the processor the second time it doesn't. I know this is the correct behavior for this processor. Is there any way to start and initialize the processor via NIFI API in such a way that It takes the files again? I need this approach because I need to do testing. Thanks!

Hi everyone,    Hoping someone can help.    I'm getting this error when impalad connects:    RPC error: Client for <hostname>:23000 hits an unexpected exception authorize: cannot authorize peer, type: N6apache6thrift9transport13TSSLException   I get a similar error when SSL is disabled.    the connect/disconnect cycle repeats about every 1/2 second forever.   The log does include a message that the cert was successfully loaded so I don't think it's a problem with that, but I'm open to any suggestions.   Is the type: noted above literally referring to a connection to an HBase thrift server?    Anyone know more about this specific error message?    Thanks!    

I have created a NIFI template. After that I have made few changes in template, now im not understanding hwo to save the template with changes included.

Accessing hive metastore from pyspark on a kerberized cluster throws an error ....Py4JJavaError: An error occurred while calling o27.sql.: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException; at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106) at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114) at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102) at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141) at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136) at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55) at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$

Hi community,what is the reason for hive dependency on pig during installation process in ambari ?Br,Dennis

i have json array like the following: [{"id":"one"}, {"id,"two"}] Any idea how could i "package" it into json "root" like the following: { [{"id":"one"}, {"id,"two"}] }I am trying to use JolttransformJSON but couldn't find any clues how to approach operation like this....Thank you

Here is my Hbase log : 2019-04-26 10:39:09,736 ERROR [Thread-16] master.HMaster: Failed to become active masterorg.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=READ, inode="/apps/hbase/data/hbase.version":root:hdfs:-rw-------at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:261)at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:589)at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:377)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1857)at org.apache.hadoop.hdfs.server.namenode.FSDirectory

I fetched the data from hbase using fetchHbaseRow and the response is in json format. but the value for the attribute is array of objects and it is having escaped characters in there.A sample response is posted below:{"someValues":"[{\"a\":\"6f7f8ffe-4837-4fc1-a5b9-45bc35727b69\",\"b\":\"da_states\",\"doc\":\"unknown\",\"idNew\":\"2846e8e1-5b17-490c-a99e-df1f1e3fa009\",\"status\":\"Approved\",\"scope\":\"Global\",\"dateModified\":\"2019-04-19T11:26:02Z\"}]"}I need to split the objects in array and because of escape characters I am not able to.@Matt Burgess

I'm trying to use regex to match the line:# DOG CAT BIRDNiFi regex seems to treat # as a comment character, whereas other regexes do not.How do I match the literal character # in a string with regex?The doc page https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escaping doesn't seem to help me out here.

I can't find a definitive answer (yet) to this question:Is "Partitioned By" compatible within the context of a CTAS type query?  I want to create a partitioned table using a CTAS type query using the Map Reduce engine.  I could use Spark but typically use Map Reduce because of the volume of data.  I do not have Impala available to me.  This documentation appears to suggest that I can (https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_create_table.html), but I can't get it to work (the documentation may pertain only to Impala which is not installed in my shop).  See specific documentation snippet, below. I'm running CDH 5.10.  See additional details (far) below. Thanks,  Jim CREATE TABLE AS SELECT:CREATE [EXTERNAL] TABLE [IF NOT EXISTS] db_name.]table_name [PARTITIONED BY (col_name[, ...])] [COMMENT 'table_comment'] [WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)] [ [ROW FORMAT row_f

Hi , I am upgrading from Spark 1.6.0 to Spark 2.1 on CDH 5.10 platform. I am trying to run spark2-submit command for python implementation and it is failing giving below error. Looks like it is expecting some path property while initilization and creating SparkContext object which is not happening. Error details are as below. Please suggest if any specific configuration is missing or required for spark2. sc = SparkContext(conf=conf)File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__File "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_initFile "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_contextFile "/apps/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/p

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).Regards

hi allI use the following API , to stop the SPARK2_THRIFTSERVER , on specific node - node2 curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://node2:8080/api/v1/clusters/HDP/hosts/node1/host_components/SPARK2_THRIFTSERVERafter the API performedhow to know if service stooped ?

Can we put data into HDFS from the nifi which in not part of any clusterI have HDP-Ambari with 4 node cluster, and I have installed nifi as standalone server in the master node machineCan get the data from my local machine and store them in HDFS ?

Will you please please help me to configure new database.By default the spark-sql is connecting to the hive, so i want to configure new database like postgreSQL instead of hiveit will be very help full to rectify for me

Hello, I am trying to install Spark 2.3.0 release 4, but after placing csd (jar file) to /opt/cloudera/csd i cannot add it from service. I checked on this url http:xyz:7180/cmf/csd/refreshand got the following error: placeholderHandlers":[],"invalidCsds":{"/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar":["Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar, Error: CSD bundle data directory or Jar file doesn't exist"]},"repoEnabled":true}}I followed these steps:a. Download the spark 2.3.0 jar b. Upload the CSD to /opt/cloudera/csd in the Cloudera Manager server.c. Change the owner and group for the JARd. restart cloudera scm server and management service Can anyone help ? I have CDH release 5.16.1-1.cdh5.16.1.p0.3