Find answers, ask questions, and share your expertise
Recently active
Good morning, I was hoping someone would know if there is a supported method for installing additional libraries onto the NiFi nodes from Cloudera Manager. For example, I need to include the Oracle JDBC driver JAR on each server so it can be referenced in the controller services. I know I can load files onto the servers via server deployment tools, but I was hoping there was a way to deploy additional library files to the cluster via the manager tool as part of the configuration. (Especially as these files may need to be updated over time, and I would rather not run the scripts which impact the server at a lower level.) Thanks.
Updating statistics using "COMPUTE INCREMENTAL STATS" produced the following error. Server version: impalad version 3.2.0-cdh6.3.2 RELEASE (build 1bb9836227301b839a32c6bc230e35439d5984ac)Query: COMPUTE INCREMENTAL STATS hoge_db.huga_tbl PARTITION ( dt >= "20221015" )ERROR: AnalysisException: Incremental stats size estimate exceeds 200.00MB. Please try COMPUTE STATS instead. In the error message, it says "Please try COMPUTE STATS instead.".I don't understand why. Wouldn't "COMPUTE STATS" give the same result?
Hello all, NiFi Jolt questionDoes anyone know if it's possible to convert timestamp formats using Jolt using 'field.value' the way you can using update Record? I'm aware that I can use an attribute value in Jolt to convert a field, like:"date" : "${timestamp:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()}" but I was wondering if this function can be done using Jolt on incoming 'field.values', have tried using the default spec as below but it doesn't change the format. Thanks in advance. Andy
I was testing if mapreduce is working properly on my cloudera 6.3, CDH 6.3.2 cluster.I have installed cloudera using one-click-installer Proof-of-concept one! while i excecuted the below command as it is in this linkhttps://docs.cloudera.com/documentation/other/tutorial/CDH5/topics/ht_usage.html Command:sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100 i Got below error: WARNING: Use "yarn jar" to launch YARN applications.Number of Maps = 10Samples per Map = 100Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Wrote input for Map #4Wrote input for Map #5Wrote input for Map #6Wrote input for Map #7Wrote input for Map #8Wrote input for Map #9Starting Job20/01/02 15:18:01 INFO client.RMProxy: Connecting to ResourceManager at ABC.xyz.net/192.168.xx.xx:8032java.io.FileNotFoundException: File does not exist: hdfs://ABC.xyz.net:8020/user/yarn/mapreduce/mr-framework/3.0.
Hello!I`m newbie in spark and all cloud-data workflow, but I have a problem on my new job where I need to work with PySpark and Hadoop.In my spark-history some applications are "incompleted" for week now. I've tried to kill them, close sparkContext(), kill main .py process, but nothing helped.For example, yarn application -status <id> shows: ...State: FINISHEDFinal-State: SUCCEDED...Log Aggregation Status: TIME_OUT... But in Spark-History I still see it in incomplete section of my applications. If I open this application there, I can see 1 Active job with 1 Alive executor, but they are doing nothing for all week. This seems like a logging bug, but as I know this problem is only with me, other coworkers doesn't have this problem.This thread doesn't helped me, because I dont have access to start-history-server.sh.I suppose this is because of Log Aggregation Status: TIME_OUT because my "completed" app
Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.
I have a Nifi process where I create a Hive table and add a single row to the table in a separate PutHiveQL process. The 'create table' process works most of the time, but the 'insert' almost always fails with the below error. This even though I direct the issue to the RetryFlowFile processor and repeat it 5 times. Note that I can run the same insert statement in Hue without any issues. What could be the problem? Is there anything I can tweak to make this work? PutHiveQL[id=27ab3890-0b98-1ba9-8499-5c4985902ebf] Failed to update Hive for StandardFlowFileRecord[uuid=91a0e7ea-5f2d-4530-bcd8-6e650597f6c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1666794893345-1, container=default, section=1], offset=5660, length=128],offset=0,name=91a0e7ea-5f2d-4530-bcd8-6e650597f6c7,size=128] due to Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask; it is possible that re
also tried on web shell client gettiing server not yet listening on http port 8080 after 50 seconds Exiting.Tried all passwords plz help?
Hello allI have a slightly annoying problem in NiFi, I have some JSON records that have a field 'date' nested under 'sources' that I want to turn to Epoch time. Using updateRecord I am able to do this using the following config:As you can see where this field is present it works, however sometimes this field is not present, in these cases the record value is blank:To do deal with these blank field I did some searching online and found the following 'isBlank' could be used in updateRecord property to deal with blank fields:The outcome of this fixes the blank field issue... but, doesn't convert the present field values to epoch, see below:Does anyone know how I can get these two actions to perform at the same time? Thanks in advance. Andy
Hello and thank you for reading I Have created a randomRead test and it took 40 minutes while randomWrite took 5 minutes is that normal ? below is the output for Write and read operations Write test output hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite 1 2022-10-26 13:32:53,776 INFO [main] hbase.PerformanceEvaluation: RandomWriteTest test run options={"addColumns":true,"autoFlush":false,"blockEncoding":"NONE","blockSize":65536,"bloomType":"ROW","bufferSize":2097152,"caching":30,"cmdName":"randomWrite","columns":1,"compression":"NONE","cycles":1,"families":1,"filterAll":false,"flushCommits":true,"inMemoryCF":false,"inMemoryCompaction":"NONE","measureAfter":0,"multiGet":0,"noOfTags":1,"nomapred":true,"numClientThreads":1,"oneCon":false,"perClientRunRows":1048576,"period":104857,"presplitRegions":0,"randomSleep":0,"replicas":1,"reportLatency":false,"sampleRate":1.0,"size":1.0,"splitPolicy":null,"
I have to do some custom preprocessing tasks on a huge data file (~200GB). currently, its works as below way.select * from tablepreprocessing line by linereturn a new single flow fileso I decided to convert the above approach to the below way.get the row count from the user (let's assume the user gives 1000)execute select * query as resultSetread the results line by line (rs.next())when the line count reaches 1000 return the flow file and continues to other linesSo my approach is as below onTrigger public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { logger = getLogger(); FlowFile flowFile = session.get(); if (flowFile == null) { return; } try { final Long rowLimit = context.getProperty(ProcessorUtils.MAX_RECORD).evaluateAttributeExpressions(flowFile).asLong(); Connection conn = DriverManager.getConnection( // db connection properties
Hi guys, Anyone know scripts to remove hdfs file more than 1 years? Thanks,Mazz
I have a requirement to notify once the files on the destination is processed. The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage. Each state has a different count of files. Once the files are sent to cloud storage, notify the end user using email for each state. I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state.
Hi everyone, I'm geting the next error in NiFi: InvokeHTTP[] Routing to Failure due to exception: IOException thrown from InvokeHTTP[]: IOException -> okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR: org.apache.nifi.processor.exception.ProcessException: IOException thrown from InvokeHTTP[]:okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR I already teste it in Postman and works fine! Any help to try to fix it? Thanks
Hi Team, Using QueryRecord to filter the incoming array of objects where each object has an array field.The field can be an empty array or it contains some objects in it but it is not null. Tried a lot of SQL queries but didn't succeed. Also, I am trying to consider Record based approach as I want to work on multiple flow files instead of single ones In Actual flow wanted to use ForkRecord on this array but it outputs an empty array if any object contains an empty array in the path so had to use QueryRecord before ForkRecord to filter those empty arrays. Not sure if it's the bug/expected behavior in ForkRecord Flow: GenerateFlowFile:Custom Text: [ { "fields": [ { "field": "field1" }, { "field": "field1" } ] }, { "fields": [] }, { "fields": [ { "field": "field2" }, { "field": "field2" } ] }, { "fields": [ { "field": "field3" }, { "field": "field3" } ] }] Expected output: To filter out the 2nd object whose fie
I am using the importPoliciesFromFile API call but see it deletes and creates the policies. Will this cause any impact on authorization if someone hits the api at the same time or is this API synchronous? By synchronous I mean, does it stop all other code execution and user interaction until the call returns?
Our vulnerability scanning found these two vulnerabilities on our CDP Private Cloud 7.1.7-SP1, CVE-2022-22970 and CVE-2022-22971. There are several versions of spring-core in the parcel, none of which are the recommended version:./jars/spring-core-4.3.29.RELEASE.jar./jars/spring-core-5.2.18.RELEASE.jar./jars/spring-core-5.3.10.jar./jars/spring-core-5.3.12.jar./jars/spring-core-5.3.13.jar./jars/spring-core-5.3.4.jarIs CDP vulnerable to these vulnerabilities? https://tanzu.vmware.com/security/cve-2022-22970https://tanzu.vmware.com/security/cve-2022-22971
Hola compañeros, Requiero de una ayuda y es que he tratado de conectarme a Impala desde Tableau, pero no me permite. Revise la solución recomendada https://kb.tableau.com/articles/Issue/error-code-b19090e0-impala-connector-not-connecting-via-http-transport?lang=es&_ga=2.9066673.509334502.1666737880-398800890.1659558110 por Tableau pero no me funciona, ¿Qué más podría revisar? Les agradezco si me pueden ayudar :).
Hi Team, We have total 46 nodes cluster.9(2 Master & 7 slave ) nodes on Baremetal and 37 nodes on VCP(VM's). Can we set any limit/quota on CDH side by restrict using maximum bandwidth from N/W switches??
I have 2 process that I need to execute on parallelo and I have a 3rd process what I wanto to start when firsts boths process will been finished, so my question is: There is a some box processor that I can use in order to implement this logic or somebody has a some idea to develop it. Thank a lot and I am waiting for your answersBest regards
I am a beginner. I submitted the HQL(create table bds_hive.test_ft19 (name string)) using spark3-shell and experienced three permission errors. These errors indicate that I do not have permissions for the corresponding paths on HDFS. According to the error message, I configured the corresponding path permission in Ranger, but it did not work. In addition, there is no denied record in the audit log. I hope guys can help me analyze and provide solutions. Hadoop version:3.1.1.7.1.7.1000-141Hive version:3.1.3000.7.1.7.1000-141Spark version:3.2.1.3.2.7171000.1-1
When I am going to install hive and hive on tez following issue is reflecting again and again in my cluster "hive metastore canary failed to create catalog" can anyone please have a solution for it.
Hello everyone, I have a NiFi cluster (1.12.1) with 3 secure nodes with SSL and 3 more external zookeeper nodes, I need to update this guy to version (1.17.0). I think about uploading a new environment with 1.17.0 and migrating the flow files? Would work?
we have set up 4 node cluster and all other 3 nodes start without any exceptions. and in NIFI uiit shows our down node is in connecting state.but our down NIFI node is not running.its logs as below.2022-10-17 19:04:51,860 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 114432022-10-17 19:04:51,950 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinatorjava.lang.IllegalStateException: Already closed or has not been startedat com.google.common.base.Preconditions.checkState(Preconditions.java:173)at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:165)at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1120)at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:349)at org.apache.nifi.we
Hi, I am new to NiIf and need your help please.I have a HandleHttpRequest processor and want to get the URL parameters into JSON.The URL looks similar to this:http://mydomain.de/getsomedata?key1=value1&key2=value2 but may can also be: http://mydomain.de/getsomedata?myVar1=myValue1&myVar2=value2&lang=DESo No. of parameter and parameter names and values can change of course.Now I need a processor that extracts any of key/value pairs from http.query.string into:{ params: { "key1":"value1", "key2":"value2" }} or for 2nd example:{ params: { "myVar1":"value1", "myVar2":"value2", "lang":"DE" }} How can I achieve that? Is there any processor I can use or should I use ExecuteScript processor?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.