Hello, I am trying to replay the example of https://de.hortonworks.com/blog/hdf-3-1-blog-series-part-6-introducing-nifi-atlas-integration/.Therefore, I created the following data flow in nifi: In Atlas, I can see the "NiFi Flow Structure", but not the "NiFi Data Lineage". My "ReportLineageToAtlas" Reporting Task looks like this: Thank you for your help! Kind regards, Jan ... View more

Hello, I am trying to replay the example of https://de.hortonworks.com/blog/hdf-3-1-blog-series-part-6-introducing-nifi-atlas-integration/.Therefore, I created the following data flow in nifi: In Atlas, I can see the "NiFi Flow Structure", but not the "NiFi Data Lineage". My "ReportLineageToAtlas" Reporting Task looks like this: Thank you for your help! Kind regards, Jan ... View more

Hello, I am trying to replay the example of https://de.hortonworks.com/blog/hdf-3-1-blog-series-part-6-introducing-nifi-atlas-integration/. Therefore, one data flow in Nifi. In Atlas, I can see the "NiFi Flow Structure", but not the "NiFi Data Lineage". My "ReportLineageToAtlas" Reporting Task looks like this: Thank you for your help! Kind regards, Jan ... View more

Hello, I am using the ExecuteScript processor to execute a shell command. Therefore I tried several functions of the subprocess library (like check_output, check_call, call, Popen etc.), the os library (os.system(cmd)) and the commands library. But they all have the same behavior: It seems that the ExecuteScript Processor does not wait for the subprocess to finish the command, even if I added the .wait() function to do so... Here is my code: import subprocess import os import time import commands import cmd flowFile = session.get() if (flowFile != None): filename = flowFile.getAttribute('filename') file_id = filename.split('/')[-1] route= flowFile.getAttribute('RouteOnAttribute.Route') storePath = '/home/user/input/' + route log.info('Filename: ' + filename) command= 'aws s3 cp s3://<s3_bucket>/'+filename+ ' ' + storePath flowFile = session.putAttribute(flowFile,'cmd',command) # I tried the following commands: #ret = commands.getstatusoutput(cmd) #ret = subprocess.Popen(cmd,env=os.environ) # p = subprocess.check_output([c for c in cmd.split(' ')]) # p = subprocess.call([c for c in cmd.split(' ')]) #ret.wait() p = subprocess.check_call([c for c in cmd.split(' ')]) flowFile = session.putAttribute(flowFile,'storePath',storePath + '/' +file_id) #flowFile = session.putAttribute(flowFile,'returncode',str(ret[0])) #flowFile = session.putAttribute(flowFile,'ouput',str(ret[1])) if os.path.isfile(storePath + '/' + file_id): session.transfer(flowFile, REL_SUCCESS) else: session.transfer(flowFile, REL_FAILURE) Is there a way to explicity wait for the subprocess to finish its task? I also copied the the 'cmd' attribute value and executed it on the shell without any problems. Kind regards, Jan ... View more

Hi, I want to use the PutHiveQL to execute the following INSERT (Content of the Incoming FlowFile: INSERT INTO sensehat VALUES ('2018-01-24 05:26:21', 29.84, 30.78, 25.26, '192.168.16.3'); However, I get the following error: 2018-02-01 15:27:37,256 ERROR [Timer-Driven Process Thread-10] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=286bb204-0161-1000-9486-e320fcd50082] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=b9e4155e-d7ef-4817-9835-721d7f108584,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1516880313990-13, container=default, section=13], offset=717915, length=89],offset=0,name=3026792823336151,size=89] due to java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask: {} org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=b9e4155e-d7ef-4817-9835-721d7f108584,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1516880313990-13, container=default, section=13], offset=717915, length=89],offset=0,name=3026792823336151,size=89] due to java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnGroupError$2(ExceptionHandler.java:226) at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnError$1(ExceptionHandler.java:179) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199) at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101) at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296) at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98) at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) ... 18 common frames omitted My DataFlow looks as follows: The SelectHiveQL is receiving data from the table "sensehat", but the insert statement is not working. I am using NiFi 1.4. and HDP 2.6. Kind regards, Jan ... View more

Hi, I want to connect my pi with nifi as described in https://de.hortonworks.com/tutorial/analyze-iot-weather-station-data-via-connected-data-architecture/section/3/#build-nifi-flow-store-minifi-data-to-hdfs-3. I secured nifi with certificate authentification and generated two truststores and keystores as @Matt Clarke suggested: "I would execute the 'hostname -f" command on both the server where NiFi is installed and the server wheer MiNiFi is/will be installed. Then use those full hostnames in the tls-tookit.sh script to create two keystores for use on these two instances ./tls-toolkit.sh standalone -n '<nifi.hostname>,<minifi.hostname>' -C 'CN=<nifiuser>, OU=NIFI' -P <truststore passsword> -S <keystorepassword> You will end up with a directory for each server being created that contains the needed TLS configuration info/files. If you want to include SAN to your new certificates, you will need to create each one at a time: ./tls-toolkit.sh standalone -n '<nifi.hostname>' -C 'CN=<nifiuser>, OU=NIFI' -P <truststore passsword> -S <keystorepassword> --subjectAlternativeNames '<nifi-ip-address>,<etc>' ./tls-toolkit.sh standalone -n '<minifi.hostname>' -P <truststore passsword> -S <keystorepassword> --subjectAlternativeNames '<minifi-ip-address>,<etc>' You should not need to any SAN entry if you generate your keystores using the actual hostnames assigned to your servers. Also make sure that the following properties have been set to the appropariate hostnames as well in the nifi.properties file: nifi.remote.input.host=nifi.web.https.host= Once your NiFi server is using the new keystore and truststore files, you can use the nifi server hostname in the RPG. Thanks, Matt" I copied the files in the conf folders of the NiFi and MiNiFi servers. The yml file of MiNiFi looks like: MiNiFi Config Version: 3 Flow Controller: name: MiNiFi comment: '' Core Properties: flow controller graceful shutdown period: 10 sec flow service write delay interval: 500 ms administrative yield duration: 30 sec bored yield duration: 10 millis max concurrent threads: 1 variable registry properties: '' FlowFile Repository: partitions: 256 checkpoint interval: 2 mins always sync: false Swap: threshold: 20000 in period: 5 sec in threads: 1 out period: 5 sec out threads: 4 Content Repository: content claim max appendable size: 10 MB content claim max flow files: 100 always sync: false Provenance Repository: provenance rollover time: 1 min implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository Component Status Repository: buffer size: 1440 snapshot frequency: 1 min Security Properties: keystore: './conf/keystore.jks' keystore type: 'jks' keystore password: 'xxxxxx' key password: 'xxxxxx' truststore: './conf/truststore.jks' truststore type: 'jks' truststore password: 'xxxxx' ssl protocol: TLS Sensitive Props: key: algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL provider: BC Processors: - id: ab316f87-2c97-3fe4-0000-000000000000 name: ExecuteProcess class: org.apache.nifi.processors.standard.ExecuteProcess max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Argument Delimiter: ' ' Batch Duration: 5 sec Command: python Command Arguments: /home/pi/Documents/sensehat.py Redirect Error Stream: 'false' Controller Services: [] Process Groups: [] Input Ports: [] Output Ports: [] Funnels: [] Connections: - id: 7aba8512-3f8c-3dd9-0000-000000000000 name: ExecuteProcess/success/1e4831a5-0161-1000-53c4-30f108c20272 source id: ab316f87-2c97-3fe4-0000-000000000000 source relationship names: - success destination id: 1e4831a5-0161-1000-53c4-30f108c20272 max work queue size: 10000 max work queue data size: 1 GB flowfile expiration: 0 sec queue prioritizer class: '' Remote Process Groups: - id: db36c66b-4e7b-316b-0000-000000000000 name: '' url: https://<nifi hostname>:8011/nifi comment: '' timeout: 30 sec yield period: 10 sec transport protocol: RAW proxy host: '' proxy port: '' proxy user: '' proxy password: '' local network interface: '' Input Ports: - id: 1e4831a5-0161-1000-53c4-30f108c20272 name: MiNiFi comment: '' max concurrent tasks: 1 use compression: false Output Ports: [] NiFi Properties Overrides: {} But I get the following error: 2018-01-23 18:30:46,323 ERROR [Timer-Driven Process Thread-3] o.a.n.c.t.ContinuallyRunConnectableTask RemoteGroupPort[name=MiNiFi,targets=https://gcvhdp01.ad.german-mgmt.de:8011/nifi] failed to process session due to java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder; 2018-01-23 18:30:46,325 ERROR [Timer-Driven Process Thread-3] o.a.n.c.t.ContinuallyRunConnectableTask java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder; at org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:257) at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:81) at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder; at org.apache.nifi.remote.util.SiteToSiteRestApiClient.setupClient(SiteToSiteRestApiClient.java:278) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getHttpClient(SiteToSiteRestApiClient.java:219) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1189) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1237) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.fetchController(SiteToSiteRestApiClient.java:419) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:394) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:361) at org.apache.nifi.remote.client.SiteInfoProvider.refreshRemoteInfo(SiteInfoProvider.java:69) at org.apache.nifi.remote.client.SiteInfoProvider.getActiveClusterUrl(SiteInfoProvider.java:247) at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:163) at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:127) at org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:238) at org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250) ... 10 common frames omitted Where do I need to change something? Kind regards Jan ... View more