Find answers, ask questions, and share your expertise
Recently active
How do you configure a PutMongo NiFi Processor with ID as the primary key used to select the document and updating 2-3 fields inside the document with user defined attributes? What I have so far and it errors out : Update Query--> {my_id:${my_id},$set:{['a']['b']['exists_in_file']:"True"}}
we need to know if nifi is responsible for the file below: /run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE/supervisor.conf because this file contains information that we are not finding in the nifi settings in the CDM
Hi,I have successfully installed CDP private cloud trial version on AWS using below instructions.https://www.cloudera.com/tutorials/how-to-create-a-cdp-private-cloud-base-development-cluster.htmlQuestion:How to access the CDP components (spark, hive, etc) from my local laptop through CLI?As part of installation, 4 hosts were created in aws, but I do not know how to access those hosts through CLI from my laptop. I am using ubuntu on my laptop.Also I am able to access all the 4 aws hosts through ssh using below command, but after getting into the aws host I am not able to invoke pyspark because I am getting below error.ssh command:ssh -i '/home/<localuser>/.ssh/cdp-trial-key.pem' centos@<aws_ip_address>Error:org.apache.hadoop.security.AccessControlException: Permission denied: user=centos, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xMy objective:I need to learn all components in CDP, write few lines of code using python and spark, access HDFS, etc. But I am new to t
Here is the code.create external table testtable1 (code string, codesystem string)ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'WITH SERDEPROPERTIES ( "input.regex" = "(.{27)(.{50})" )LOCATION '/data/raw/testtable1';The error message is:ParseException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'org.apache.hadoop.hiv... ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax errorIt looks like Impala table only accepts "Row Format Delimited".Then how can I create an hive table with fixed width layout? Should I just do it outside Impala, bu through Hive, and then do other data operation on this table via Impala?Thanks.
can we delete the files that exist in the directory configured in the "Heap Dump Directory" option of the services in CDH/CM?
Hello,I receive a json from a HTTP Request. It looks like this : { "accessToken" : "token", "expiresIn" : "12600", "refreshToken" : "token", "tokenType" : "Bearer", "usagePointsId" : "22516914714270, 22516914714270, 22516914714270"} I have the usagePointsId key containing a string with multiple values. I will need each values to include them in different API calls.I would therefore like to be able to separate the string at the comma level to extract each value.Then the idea is to create a new unique flow file for each value. This value would be associated with a "usagePointId" attribute which would be used in my API calls. Also, the others json key would be added as Attributes for each flow filesI don't know if it's possible to create multiple differents flow file based on a split attributes. I tried to use the ExecuteScript (with ECMASCRIPT) to split the string and create Flow Files with something like this : flowFile = session.get()if (!flowFi
we are having a problem with hbase, the following error is displayed: INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=11, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2982)at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1018)at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304), details=, see https://s.apache.org/timeout hbase and zookeeper have already been restarted several times and still the above error is presented the zooke
we need to know which is the responsible file that keeps the zookeeper settings, which one would it be?we need to validate if the "maxClientCnxns" option is the same as what was defined by the CDM in the zookeeper service.
Hello everyoneI cant see pending containers count on Yarn resource pool. How can enable or solve this problem? any suggestions?thanks
Hello,Is it possible to send dynamic values using some external commands like "curl" so that values will be used in the process?For example we will be send date dynamically then it can be used in SQL query.
I am facing the following error on the cmd: command: grunt> A = LOAD 'C:\data\myfile.txt'; or grunt> a = load 'c:\data\myfile.txt' using PigStorage(' ') as (f1:int,f2:int, f3:int); error:<line 1, column 14> Unexpected character 'D'2020-03-15 20:56:50,153 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 1, column 14> Unexpected character 'D'Details at logfile: C:\deploy\logs\pig_1584281243336.log the error trace is as follows- Pig Stack Trace---------------ERROR 1200: <line 1, column 14> Unexpected character 'D' Failed to parse: <line 1, column 14> Unexpected character 'D' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:246) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:182) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1792) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1765
hi, is it possible to add workload user as host user ? (just need it for rstudio server) thanks
I have been searching all over and can't find any solution that works. I use Kudu and Parquet(hdfs) all searchable via Impala. I want to use Grafana to visualize these data sources, but all I'm finding is performance metrics of the Cloudera components, not how to get the the massive amounts of data in the lake it self. Any suggestions?
We are running rebalance on HDFS and we are getting the error below: "WARN balancer.Dispatcher: Failed to move blk_1314197946_240487461 with size=134217728 from 10.10.10.109:1019:DISK to 10.10.10.183:1019:DISK through 10.10.10.99:1019: Got error, status message opReplaceBlock BP-1707938289-10.10.10.96-1520510791093:blk_1314197946_240487461 received exception java.io.IOException: Got error, status message Not able to copy block 1314197946 to /10.10.10.183:51650 because threads quota is exceeded., copy block BP-1707938289-10.10.10.96-1520510791093:blk_1314197946_240487461 from /10.10.10.99:1019, block move is failed" we would like to know why we are getting these errors? would it be because the network interfaces are 1GB? if so, how can we resolve these errors? increasing the speed of network interfaces? PS: The hosts have 1GB network interfacesPS: We are using Ambari Server version 2.6.2.2.
Previous issue was Py4JJavaError: An error occurred while calling oxxxx.collectToPython.
Hi,Nifi is not capable of spinning up a new node if one goes down, some of the threads in community says, autoscaling is possible in AWS . AWS Auto Scaling monitors your applications and automatically adjusts capacity to maintain a steady, predictable predefined number of a node if a specifically configured alert is triggered eg Bad health or lost heartbeat. We have a setup in our 3 fixed node cluster. To established authorised nodes communication within the cluster we need to provide host names in authorizers.xml file. So I need to know how we can autoscale our NIFI clusters, where we need to update our authorizers.xml file in each and every instance. Any help would be highly appreciated. Thanks Abhishek
When I am connecting via Cloudera ODBC Driver the process fails due to following error: [Cloudera][Support] (50170) Creation of new thread failed. Does anyone know what the issue is and can help here? I did not find anything concerning this error.
I'm having a problem trying to create a nifi cluster with kubernetes.Error displayed: "The specified run.as user ec2-user does not exist. Exiting."The error appears when I use /opt/nifi/nifi-current/conf folder mappingIf I remove this mapping from the nifi folder, the cluster is created without problems.But I need this folder to keep the data flows if the container dies, come back with the datathe folder in the instance is in /home/ec2-user/user/1.16I already gave permission 777 on the folder to test and it still didn't workI already changed the user to upload with root user and nifi (pid 0 and 1000) and it didn't work
Hello Community, I'm running into a problem when I submit a job in Hue editor, from Hue home page I navigate to Query > Editor > Java. I point to the path of my jar file and then add the class name called Hortonwork.SparkTutorial.Main. Then after I hit run I get this error "Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf". I imagine it has to with not having the jar file SparkConf, I'm using IntelliJ for this project and I've added some jars to my project, I've taken some screen shots of what the jar files look like and have attached them to the post. Curious if any of these jars look familiar to anyone, I may not even be adding the right ones. Thank you all for the taking the time to read my post and for your help. Full list error is: >>> Invoking Main class now >>>INFO: loading log4j config file log4j.properties.INFO: log4j config file log4j.properties loaded successfully.Setting [
Tried to install 7.4.4 on Ubuntu 2004 on a VMWe do have a license for private useOn starting server (Base) we have follwing log 2022-03-31 16:25:44,839 INFO main:com.cloudera.server.cmf.Main: ================================================================================2022-03-31 16:25:44,852 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -Dcmf.log.dir=/var/log/cloudera-scm-server, -Dcmf.log.file=cloudera-scm-server.log, -Dcmf.jetty.threshhold=WARN, -Dcmf.schema.dir=/opt/cloudera/cm/schema, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dpython.home=/opt/cloudera/cm/python, -XX:+HeapDumpOnOutOfMemoryError, -Xmx4G, -XX:MaxPermSize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp, -XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, -XX:OnOutOfMemoryError={CMF_PATH}/bin/killparent-cm-server], Args: [], Version: 7.4.4 (
I have literally used the VM out of the box. Trying to train on materials and can't get anything working. Been trying to resolve this issue for a week. Just finished a Codementor session and they couldn't get it either. if the password is not default (and I literally haven't touched anything) what is next? Hive access denied. Someone suggested changing something in ranger. I don't know if that's true or not. Asked a guy on Codementor. Just trying to get data into the table I created.Delete this VM and try again or? I have logged in and ran hive as root, maria_dev, hive and hdfs. (btw, don't know what I am doing, not being trained properly, have zero tech support and am expected to learn this for work clients and am not supposed to have any problems that need solving. (shrugs)
Hi, I am trying to perform a scripted start / stop of HDP services and its components. I need to get this done service by service, because there are few components like Hive Interactive / Spark Thrift server / Kafka broker, which does not get started / stopped in proper time. Most of the times it is observed that even if HSInteractive is still starting (visible in the background operations panel on Ambari), command moves to next service to start / stop and ultimately HSI fails. Hence, I also want to ensure that previous service is completely stopped / started, before attempting to stop / start next service in the list. To achieve, below is the script I have written (this is to stop, I have similar for start). However, when it reaches to the point for Hive or Spark or Kafka services, even though the the internal components like Hive Interactive or Spark Thrift server or Kafka broker are not started / stopped, it moves to the next service. Most of the times the start of Spark T
Hi, I am currently new to NIFI and have been working on a project that requires communicating to a secured RabbitMQ server to consume data. To make an SSL handshake I am using StandardSSLContextService controller service by adding the required Truststore information. And the Truststore actually contains the client certificate to communicate to RabbitMQ server from NIFI. But I am facing the below issue while running the ConsumeAMQP processor. 17:36:40 UTC ERROR nifi.svc.cluster.local:9443 ConsumeAMQP[id=76fe890c-4c44-3a3e-933b-4f4e5c4e4539] Failed to initialize AMQP client: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_requiredcauses: com.rabbitmq.client.ShutdownSignalException: connection errorcauses: java.io.IOExceptioncauses: java.lang.IllegalStateException: Failed to establish connection with AMQP Broker: com.rabbitmq.client.ConnectionFactory@72bc529717:36:40 UTCERROR76fe890c-4c44-3a3e-933b-4f4e5c4e4539 nifi.svc.cluster.local:94
Hi Everyone! I've been receiving the following error on my Cloudera Manager. "603.1 MiB of swap memory is being used by this role's process. Warning threshold: 200.0 MiB." I have tried removing Solr collections by going to the following location and running the following query. http://<HOSTNAME>:8993/solr/#/ranger_audits/documents<delete><query>evtTime:[* TO NOW-0DAYS]</query></delete> But it's not having any effect on the Swap memory usage. P.S. Swappiness is set to 1 as when I run cat /proc/sys/vm/swappiness The output generated is 1. Cluster Details : Solr: 8.4.1 Hadoop 3.1.1.7.1.6.0-297 CDH-7.1.6
Hi everyone I want to add 2 servers with KTS and 2 servers with KMS to my cluster. The question is : Do I need licenses for each of this additional servers? Thanks
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.