Solved

Hi all,   New in NiFi. Hence need guidance on achieving the desired result. Scenario: 1. Multiple .txt log files  2. each .txt log file contains many lines    Requirement: 1. Read each .txt log file and extract only those lines that has "Three.Link resp:". Below snippet for example is from abc.txt     09/10/18 20:06:07.581 DEBUG: (Radio) One.Link DONE09/10/18 20:06:07.963 34, 0, 0, 0, 0, -9909/10/18 20:06:08.591 DEBUG: (Radio) Two.Link request09/10/18 20:06:08.601 DEBUG: (Radio) Three.Link resp: -1 abc-34664 0 10-Sep 12:06:02 0 .00 .00 .00 .00 0 0 0 -99 -9909/10/18 20:06:08.601 DEBUG: (Radio) One.Link DONE09/10/18 20:06:08.983 34, 0, 0, 0, 0, -9909/10/18 20:06:09.600 DEBUG: (Radio) Two.Link request09/10/18 20:06:09.611 DEBUG: (Radio) Three.Link resp: -1 abc-34664 0 10-Sep 12:06:03 0 .00 .00 .00 .00 0 0 0 -99 -9909/10/18 20:06:09.611 DEBUG: (Radio) One.Link DONE     What I am trying to do is to extract only those lines from abc.txt which contains

Hi, guys:   I'm useing CDH6.1.1. When I query impala like "select sum([field]) from [table]", the impala daemon is correct,and the result is correct too. But when I change sql to "select sum([field]) from [table] group by", the impala daemon unexpected exit，and the ERROR log has nothing, but the "impala.FATAL" throw "F1222 llvm-codegen.cc:137] LLVM hit fatal error: Cannot select: intrinsic %llvm.x86.sse42.crc32.64.64". Thanks everyone for your help！log

If you are using distcp command for transferring data from one cluster to another cluster on regular basis in this scenario only new data will be copied on daily basis so how distcp keep tracks on it?

I am trying to repeat the work of Change Data Capture (CDC) with Apache NiFi. My testing environment is : Ubuntu 16.04;  MySQL and NiFi are installed in the OS system without using Docker, can access to MySQL via the terminal from another Windows PC in the same local network. However, when I just use the processor CaptureChangeMySQL and connect to the MySQL, it shows me the error:    The settings of the NiFi processors are: CDC MapCache: CaptureChangeMySQL:   Ubuntu 16.04:  Could anyone help to address the issues? Thank you very much.    

Hello   I am trying to add an ExecuteGroovyScript processor. A simple example test works fine, but when I try to add my script it does not initiate. Exception (on hoover over) is really brief: InstantiationException thrown by groovy invokehelper. Script uses some external jars. As far as I can see I added those to the classpath.    I am probably doing something stupid but I was wondering how to debug this issue? The log files do not seem to contain anything wrt this error which makes troubleshooting very difficult   All tips warmly welcomed!   Tx   Peter

Hello,   Iwant to execute .py from nifi, I am able to do that but when saving data in mongo from the script (using pymongo) I got an error: Executable command python3 ended in an error: Traceback (most recent call last): 'MYpath/MY.py', in line 104, in <module>  mycol.insert(document)   If I execute this .py in command is working fine.   Command Arguments Strategy   Command Arguments Property   Command Arguments   MYpath/MY.py;1000000000   Command Path   python3   Ignore STDIN   true   Working Directory   path   Argument Delimiter   ;   Output Destination Attribute   No value set   Max Attribute Length   200

Hi community,   Beyond the ConvertExcelToCSVProcessor, does Nifi offer other processors specific to MS Excel (.xls and/or .xlsx) like for example a specific processor to parse MS Excel files?   I know that is possible to do (and I already did) it with scripting (using for example the xlrd module of Python). However, here, I ask about the existence of pre-configured Nifi processors for MS Excel format.    Thanks in advance! 

I'm passing a csv flowfile to ExecuteStreamCommand. How can I zip it with password and pass it to the next processor?   I don't know how to zip file from stdin and pass it to stdout. I'm using python

Raw data will be transferred from the source through an  HandleHttpRequest. How can I encrypt in motion this traffic?

I am logged into Ambari with 'admin' user.   Then I navigated to File View >> 'user' directory  >> Create a New Folder >> Enter name of folder and click on 'Add' button. But I am getting a server message: -   Permission denied: user=admin, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x   How can I provide write permission on user directory to write something or add a folder to it. Request to community to help me. Thanks.

All, while installing cdsw tcp-ingress controller and ds_reconsiler pods are failing due to connection failure with Grps server.. from below web service logs i see grps server status failing with port binding (description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,"grpc_status":14,"os_error":"invalid argument"}]}]}), any suggestions how to fix this...   [..] reated":"@1568398264.632135163","description":"No address added out of total 1 resolved","file":"../deps/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":349,"referenced_errors":[{"created":"@1568398264.632132593","description":"Failed to add port to server","file":"../deps/grpc/src/core/lib/iomgr/tcp_server_custom.cc","file_line":402,"referenced_errors":[{"created":"@1568398264.632131142","description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,"grpc_status":14,"os_error":"invalid argument"}]}]}E0913 18:11:0

Hi,   I tried Hortonworks HDP 3.0 image using oracle virutalbox. But the root/hadoop is not working for SSH access or browser ssh access? Need help in getting correct password or help in resetting it.   Thanks, Kalyan

Hi , I am running free trial cloudera in pseudo-distributed mod (One host, the server and agent are the same).i am trying now to enable tls/ssl for my cluster -Level 1 succedeed https://datastore.xxx.fr:7183 tuns correctly with no problem    When i configure level N2 of tls/ssl ,cloudera manager agent fails to connect !. In CM server fails to have heartbeat: when i checked :tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log I got ERROR : File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 292, in connect_sslreturn m2.ssl_connect(self.ssl, self._timeout)SSLError: certificate verify failed   The agent config files : server_host=datastore.xxx.fr server_port=7182   # Use TLS and certificate validation when connecting to the CM server.use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/x509/cmhost.pem   ---the cmhost.pem get my certifcate correctly ---when i run :   sudo openssl s_client -connect dat

Hello, how about everything I have a problem in hive at the moment of executing a query in hive with Hortonwork the next error I have is   SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskSQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskError while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask  

I have an EC2 instance running with the IPv4 Public IP 11.111.111.111.  The first thing I tried was to get NiFi running.     docker run --name nifi-standalone\ -p 8080:8080 \ -d \ apache/nifi:latest      At 11.111.111.111.111:8080/nifi/ I can reach NiFi. Great.   Now I wanted to setup a secure NiFi instance. Step 1: Create certificates with the toolkit.     ./bin/tls-toolkit.sh standalone -n '11.111.111.111' -C 'CN=admin,OU=nifi' -B SuperSecretPassword -o './standalone'       Step 2: Move keystore.jks and truststore.jks into a specific folder (here /home/ec2-user/project/nifi-standalone/certs).   Scenario 1: set NIFI_WEB_HTTP_HOST  I run the following docker command.     docker run --name nifi-ssl \ -v /home/ec2-user/project/nifi-standalone/certs:/opt/certs \ -v /home/ec2-user/project/nifi-standalone/conf:/opt/conf \ -p 8443:8443 \ -e NIFI_WEB_HTTPS_HOST=11.111.111.111 \ -e AUTH=tls \ -e KEYSTORE_P

Hi,   If I try and get the logs for an application like this: yarn logs -applicationId application_1575531060741_10424 The command fails because I am not running it as the application owner.  I need to run it like this: yarn logs -applicationId application_1575531060741_10424 -appOwner hive   The problem is I want to write out all the yarn logs to the os so I can ingest them into splunk.  If I try and figure out the appOwner for each application then this is awkward and time consuming even in a script.     Is there a better way to dump all the yarn logs to the os ?   Thanks      

Hi, We were trying to enable TLS on our cluster.We have followed cloudera steps ,but i think missed something.After enabling TLS,we were able to login to port 7183,but all  cloudera management services were disabled.   So we reverted back TLS changes from cloudera and removed all keys and certificates.After that given restart to the cloudera manager.   After this process,port 7180 started and login page came.But when i am trying to login then it is not redirecting to the GUI. when i have check the logs. then these exceptions are showing.   2019-12-17 13:24:29,320 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.2019-12-17 13:24:30,850 WARN ActionablesProvider-0:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORINGcom.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running   2019-12-17 13:24:50,238 WARN E

I have installed the HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) using Docker and Kerberized the cluster.   This works, i can kinit myuser and make an API request to Atlas:     curl --negotiate -u : sandbox-hdp.hortonworks.com:21000/api/atlas/admin/version       Now I would like to create a Ranger policy that includes my Kerberos user, however the autocomplete for users does not show any of my Kerberos users... I did read this post (https://community.cloudera.com/t5/Support-Questions/Autocompletion-of-names-not-working-in-Ranger/td-p/109988), however this problem seems to be slightly different.   What do I need to do to make Kerberos users show up in the Ranger policy creation UI?

Hello, I have some problems with NIFII made a simple dataflow, but it doesn't work ( getfile takes file it's okwhen update attribute  it's ok too and PutDataBaseRecord doesn't take any IN information what is the reason???I made a table in my postgresql database with the same attributes names   lately i began to take errors like that 

Hi Team, Trying to follow the Tutorial in Cloudera Search deployment for  CDH 6.2   Everything works until this command: solrctl collection --create test_collection -s 2 -c test_collection_config   we get the exception: { "responseHeader":{ "status":0, "QTime":33610}, "failure":{ "gis18.example.com:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://gis18.example.com:8983/solr: Error CREATEing SolrCore 'test_collection_shard1_replica_n1': Unable to create core [test_collection_shard1_replica_n1] Caused by: Permission denied: user=solr, access=WRITE, inode=\"/\":root:supergroup:drwxr-xr-x\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)\n\tat or

Hi,   I want to add headers to a csv file. I have used Replace text processor for this purpose. But final output just append headers to the first line. eg: Sample : 2-Dec-15,Mumbai,120 Expected output : date,city,temp2-Dec-15,Mumbai,120 Output : date,city,temp2-Dec-15,Mumbai,120   I have tried both ways by adding '\n' and 'shift+enter' after the header line   id,name,site,score\n   id,name,site,scoreshift+enter as the replacement value, but no luck instead \n and shift+enter was taken as a value.   Could anyone help to resolve this issue.

Hi, I'm newbie in NIFI and also in java services  yesterday I downloaded nifi-1.10.0 bin version  i run run-nifi.bin  and after i started to work with NIFI  I made a connection to postgree database but it doesn't work   also my colleague made the same thing and everything is ok. what can goes wrong with my NI FI ?   When I use this password in DBeaver  everything is OK. what have I do to make it right way?

https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.html doesn't seems to be working with ambari 2.7.1, stuck at step 3)No exception in ambari server or agent logs@Jay Kumar SenSharma

I am running following command in Zeppelin.First created hive context with following code - val hiveContext = new org.apache.spark.sql.SparkSession.Builder().getOrCreate() then I tried to load a file from HDFS with following code -val riskFactorDataFrame = spark.read.format("csv").option("header", "true").load("hdfs:///tmp/data/riskfactor1.csv")but I am getting following error message "org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://sandbox-hdp.hortonworks.com:8020/tmp/data/riskfactor1.csv;"I am quite new in Hadoop. Please help me figure out what wrong I am doing. 

Disclaimer: crosspost fromhttps://stackoverflow.com/questions/59359158/apache-nifi-1-10-how-to-execute-a-python-script-as-a-processor   In NiFi <= v1.9 I used the following NiFi setup: `GetTwitter` generates a JSON for every tweet gathered and attached to it there used to be a `ExecuteStreamCommand` that has the following parameters: This would allow me to get the flowfile as input, and I could analyse the content and do something (in this specific case I would put the content on Elasticsearch) - You could reach the same result if you do something like this `cat flowfile.json | python <scriptname>.py` But now in >= v1.10 it seems I cannot do the same. it gives me some error about parameters (it seems it wants to use the flowfile as parameter on calling the command/script instead, as it used to be, pass the flowfile as input.It gives me the following error:     2019-12-16 14:28:44,953 ERROR [Timer-Driven Process Thread-2] o.a.n.p.standard.ExecuteStreamCommand