Q&A

Hello,I'm running a groovy script which handles SFTPMy Apache NiFi version  is 1.5.0.3.1.2.0-7 My script starts with this:import com.jcraft.jsch.*def flowFile = session.get()if(!flowFile) {return}.... When I receive the file to process and move I see them in the queue but also the following is appear in the in the nifi-app.log2020-01-25 19:00:00,041 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=56c17bab-81ba-313c-9a00-aa82d48c3ea0] ExecuteScript[id=56c17bab-81ba-313c-9a00-aa82d48c3ea0] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:Script10025.groovy: 32: unable to resolve class JSch@ line 32, column 6.JSch ssh = new JSch()^Script10025.groovy: 32: unable to resolve class JSch@ line 32, column 12.JSch ssh = new JSch()^Script10025.groovy: 33: unable to resolve class Session@ line 33, c

Hello,   I'm trying to install Infra Solr from Ambari. After installation, when i restart the Infra Solr, the status of the service is red and it gives error below:   Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 51, in _ensure_metadata _user_entity = pwd.getpwnam(user)KeyError: 'getpwnam(): name not found: infra-solr'The above exception was the cause of the following exception:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 171, in <module> InfraSolr().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 54, in start self.configure(env) File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.

Hi, I have a scenarios getting the flow file from NLPProcessor as jason object, then passing to PublishKafka, the PublishKafka throwing out following error TimeoutException error, what is wrong in the configurations.     PublishKafka configurations...       The issue looks to be configuration, any help really appreciated.   Thank you, --Murali

Hi folks, i have problem in parsing json textfile. what i have done is, I sent a text file which is in a json architecture to PutMongoRecord To parse that file into json but it is not getting parsed. i have attached the configuration done by me. please help me in this problem. schemaJsonTreeReaderputmongo's configurationthe error which we gottextfile sent

Hi. I'm kerberizing an EDH 6.1.1 on SLES on Google GCE. Kerberos is  up and running. I've created a cloudera-scm/admin@REALM and gained its ticket with kinit, so I suppose everything is fine. I try to enable Kerberos using the CM wizard. When I reach the Setup KDC Account I set the cloudera-scm@REALM principal and the Continue button is enabled. I try to click on it to proceed but nothing happens. I took a look at krb5 log files and to cloudera-scm-server log files but nothing seems to be logged. I've doublechecked NTP, Reverse DNS and krb5.conf file but found nothing useful. Any help ? Thanks Stefano

I created 12 metrics alarms and policy configurations based on these alarms (6 for scale out and 6 for scale in). But in cloudbreak logs I can see that only several alarms are checked and then "No scaling activity required".Why doesn't cloudbreak check all metrics alarms? What the number of checked alarms are depend on? How can I change this behavior?

I am using SelectHive3QL processor in my Nifi flow and it is throwing a below Kerberos authentication exception. Initially it is working fine for some time and after some Idle time, getting a below error. My requirement is "SelectHive3QL" should always up and running without any issues.   Exception: SelectHive3QL[id=47753131-0b9a-39c4-b0e7-2a92ac1042c6] Issue processing SQL select * from egress_test where service_name = 'NDC' due to java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: GSS initiate failed.: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: GSS initiate failed   Cc: @MattWho , @arunak , @stevenmatison 

We currently have 2 ReplaceText processors (and we'll potentially add more in the future) that each contain a list of file names in their Replacement Value attribute, such that the corresponding flowfiles contain this list of file names in their content. We want to know if there are recommended strategies to merge these flowfiles coming from different streams, but I need the merging to happen at the same time because the output will be used to count the total number of files, and the latter will be passed to a Wait processor for some other functionality of the flow that uses the Wait/Notify pattern (the total number of files is assigned to "Target Signal Count" in the Wait processor). The main problem we're facing is that we don't know in advance which ReplaceText processors will run in the first place (could be one or the other, or both), so we don't know which flowfiles we have to merge, but we need a way to merge the flowfiles of the ReplaceText processors that are running. One way

We are still using cdh 5.10 and impala 2.7 and there is a startup option `inc_stats_size_limit_bytes` which is described in https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_perf_stats.html section "Maximum Serialized Stats Size" (which is identical to 5.10 docs) but it does not really describe what this setting means and how to predict that this is a problem. > The inc_stats_size_limit_bytes limit is set as a safety check, to prevent Impala from hitting the maximum limit for the table metadata. Note that this limit is only one part of the entire table's metadata all of which together must be below 2 GB. With pretty big tables and a lot of live partitions we had to increase catalogd memory to be able to cope with the amount of metadata. Now additionally singular large tables were not loaded during compute incremental stats because they reached inc_stats_size_limit_bytes which is far less than catalogd memory. But what does that

The  Apache NiFi registry installation guide is not mentioned support ability for windows, could any one tried installing it on Windows?

Hi, I have a flowfile with this format: <td>Item1</td><td class="dest">50.3421</td><td class="dest">20.5547</td> I need to write a query with the parameters, so I need to extract the numbers, with this for example: (\d{2}.\d{4}) And put the 2 resultos in: Insert into table(a,b) VALUES($1,$2) How can I do this? Thanks!   I tryed with replaceText but it's not what im looking for

The node certificates on my cluster are expiring soon so I have installed new ones, including on the node that has ambari-server. However, after restarting ambari server, ambari agent, and even the node itself, the old certificate still shows. I've tried also clearing cache and cookies for all time on my browser, but it doesn't work and the old cert even shows up on IE. I've tried the same methodology for other nodes in the cluster and it has worked, so why isn't it working for the ambari node? (ambari-server is set up through an https port) 

this is the link for download hdp sandboxhttps://www.cloudera.com/downloads/hortonworks-sandbox/hdf.htmlAfter select the installation type and click 'let's go'. A page for accepting terms and conditions comes out.  Select accept. After this is done, there should be a page to select the hdp sandbox version. Unfortunately, no thing comes out!

I'm trying to execute python code to start processor group and getting below error   Error: Traceback (most recent call last):400 Client Error: Bad Request for url: http://localhost:8080/nifi-api/process-groups/bee62419-016f-1000-f131-5c589af8faffFile "/Users/xxxx/Documents/Project/BU/pythonPractice/nifiTest.py", line 19, in <module>r.raise_for_status()File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_statusraise HTTPError(http_error_msg, response=self)requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://localhost:8080/nifi-api/process-groups/bee62419-016f-1000-f131-5c589af8faff     Code: newUrl="http://localhost:8080/nifi-api/process-groups/bee62419-016f-1000-f131-5c589af8faff"data1={'id':'bee62419-016f-1000-f131-5c589af8faff','state':'RUNNING'}try: r = requests.put(newUrl, data=data1,headers={'Content-Type': 'application/json','Accept': 'application/json','Aut

"Yarn Container" widget shows 0/0/0 when no containers are allocated,pending or reserved, but if any of the 3 metrics changed, the widget shows n/a, wonder if it is a bug or not?

After Cloudera CM Server Service restart for CFM installation, CM Server failed to start with the following error in log file: Caused by: java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:246)at com.cloudera.enterprise.dbutil.DbUtil$1SchemaVersionWork.execute(DbUtil.java:185)at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2289)at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2286)at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:314)at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2308)at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2293)at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:197)at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFa

Hello how are you..?   I have a problem with Zeppelin     Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Error validating the loginat org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:244)... 27 more

What is the most efficient way to get count of records meeting different search criteria from a Hive table. 1. count all records where column-a = Null 2. count all records where column-b in [1, 3, 5] 3. count all records where column-c = 'xxx' etc. there are a couple hundred of these counts, in groups of 3 or 4.    

I have removed files from HDFS by skipping trash but still it shows same disk useage

Very new to Nifi and regex. I have a test txt file with a mock log file in.Format along the lines of: srcip=10.10.10.10 timestamp=152532431 action="denied" What I need is to match against the word and then return everything after the '=' until the next space character. Any help would be appreciated.

I am wanted to split this txt file into separate flow files at the spaces.I am very new to nifi and regex and am unsure of which processor and properties I will need. For example: srcip=10.10.10.10 timestamp=152532431 action="denied" logver=12 tz="UTC+0" logid="0000012" dstip=12.12.12.12This snippet is part of a long line that I need to split up at the spacesSo here I would have 6 separate flow files. Any help appreciated.

[hdfs@xyz ~]$ beelineBeeline version 1.2.1000.2.6.3.0-235 by Apache Hivebeeline> !connect jdbc:hive2://xyz:10000/default;Connecting to jdbc:hive2://xyz:10000/default;Enter username for jdbc:hive2://xyz:10000/default;: *************Enter password for jdbc:hive2://xyz:10000/default;: *************20/01/08 19:08:10 [main]: WARN jdbc.HiveConnection: Failed to connect to xyz:10000Error: Could not open client transport with JDBC Uri: jdbc:hive2://xyz:10000/default;: Peer indicated failure: Error validating the login (state=08S01,code=0)0: jdbc:hive2://xyz:10000/default (closed)>

"Map 0% reduce 0%" is not displayed when I execute Sqoop And Tracking URL is  UNASSIGNED ---------- --- Virtual Machine: Cloudera Quickstart Virtual Machine (Using Oracle VM Virtualbox, download URL: https://www.cloudera.com/downloads/quickstart_vms/5-13.html)(1) Version: 5.13.0-0(2) Number of processors: 2(3) Controller: IDE Controller(4) Memory space: 11400 MB(5) Hard drive: 64.00 GB storage, solid state drive(6) OS: Red Hat, 64-bit(7) MySQL 5.1.73(8) HBase 1.2.0-cdh5.13.0(9) Hadoop 2.6.0-cdh5.13.0(10) Sqoop 1.4.6-cdh5.13.0 ---------- ...20/01/16 13:13:08 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hadoop/conf:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar:/usr/lib/had

Hello,    i have a problem to start hue after enabling AUTO_TLS Feature. Mysql Connection seems to be blocked and flush hosts doenot help.    Can you help me?   Thanks,   Simon

Hi, im trying to upload a CSV to a RedShift database using GetFile->PutDatabaseRecord i'm getting this error: PutDatabaseRecord:CSVReader (my file got headers, this way and not hardcoding the schema makes everything more easily scalable):Null string: NULL CSV:A,B,C,D,E,F,G,H,I2020-01-15,AX,COD245,NO,PASS,R,,,2020-01-15,AX,COD235,YES,PASS,R,,, Table with same fields names.Any idea what i'm doing wrong? Is there any way to look for the query it's making, that will make this more easy to debug.Thank you!