Member since
09-02-2016
56
Posts
6
Kudos Received
0
Solutions
02-27-2018
12:24 PM
Hi, Since we can mention only one prefix in ListS3 processor I am trying to access AWS S3 using Python boto3 in NiFi ExecuteScript processor. If this succeeds, I can send a list of folder paths to the python script to get files from various folders under S3 bucket. I provided the path of boto3-1.6.0.tar.gz in ModuleDirectory but I get the below error. @Matt Burgess @Bryan Bende @Matt Foley Please let me know how to fix this and achieve use case. ExecuteScript Processor : Error: ExecuteScript[id=d6cf51e8-0161-1000-32de-7748af781842] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: ImportError: No module named boto3 in <script> at line number 1: javax.script.ScriptException: ImportError: No module named boto3 in <script> at line number 1
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2018
10:15 AM
Hi @Andrew Lim Thanks for the article. I tried to execute this using NiFi 1.5 but I get an error "already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed" in QueryRecord processor. PFb excerpt from logs - ErrorHandlingInputStream[FlowFile=StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179]] for StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] because the session was rolled back without the input stream being closed.
2018-02-23 15:10:17,160 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=aff8e1fe-ff3c-3d77-9fcf-dcd3c1cf6ed9] QueryRecord[id=aff8e1fe-ff3c-3d77-9fcf-dcd3c1cf6ed9] failed to process session due to java.lang.IllegalStateException: StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed: {}
java.lang.IllegalStateException: StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed<br> Can you please tell what's wrong?
... View more
02-22-2018
06:24 PM
Hi @Bryan Bende How to get more than one column value from lookup file? PFB details. InputFile ID, Name
1,Elon
2,Mark
3,Jeff Lookup File ID,Company,Domain,HQ
1,Tesla,Automobile,Palo Alto
2,Facebook,SocialMedia,Menlo Park
3,Amazon,ECommerce,Seattle Output ID,Name,Company,Domain,HQ
1,Elon,Tesla,Automobile,Palo Alto
2,Mark,Facebook,SocialMedia,Menlo Park
3,Jeff,Amazon,ECommerce,Seattle Result RecordPath allows only one value. Any tips to get this kind of output?
... View more
02-22-2018
06:04 PM
Thanks for replying @Abdelkrim Hadjidj @Matt Burgess. Now I have two different schemas for input and output CSV for LookupRecord. I modified Result RecordPath to /COMPANY. When I start LookupRecord it takes in 6 flow files but doesn't return anything.
... View more
02-22-2018
04:45 PM
Hi There, I am trying to lookup records from a CSV file using LookupRecord and SimpleCsvFileLookupService but I am unable to get the final record with both values. Please find below details. Input File ID, Name
1,Elon
2,Mark
3,Jeff Lookup File ID,Company
1,Tesla
2,Facebook
3,Amazon Expected Output ID,Name,Company
1,Elon,Tesla
2,Mark,Facebook
3,Jeff,Amazon AvroSchema {
"name": "ceodata",
"namespace": "nifi",
"type": "record",
"fields": [
{ "name": "ID", "type": "string" },
{ "name": "NAME", "type": "string" },
{ "name": "COMPANY", "type": "string" }
]
} LookupRecord Processor Config CSVReader Config CSVWriter Config LookupFromCSVFileService Please let me know what's wrong in these configurations. @jfrazee @Bryan Bende @Abdelkrim Hadjidj @Timothy Spann @Aldrin Piri
... View more
Labels:
- Labels:
-
Apache NiFi
02-09-2018
01:54 PM
1 Kudo
Hi Andrew, Is there any way to automate clearing state from ListS3 processor when the flow is scheduled? I cannot clear the state every time I run the flow.
... View more
12-21-2017
10:22 AM
In a NiFi Cluster If a node goes down, it is known that the flow can be directed to another node, but what about the data that is queued in the failed node? Will have to wait until the node comes back up or it is taken care by another node? Please explain how it handles this scenario.
... View more
Labels:
- Labels:
-
Apache NiFi
12-13-2017
09:28 AM
Hello What are the different frameworks to automate Hadoop jobs? Which is the best? Thanks
... View more
Labels:
- Labels:
-
Apache Hadoop
11-22-2017
03:23 PM
Hi guys, I have a requirement to execute a script automatically whenever a hive table is updated. Please suggest the best option. Thanks
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
10-12-2017
11:48 AM
Hi slachterman thanks for the post. I get below error while running PutHDFS processor. I didn't find the SNAPSHOT jars in HDInsight, May I know the HDI version you used? Can you help me fixing this error? Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration; Thanks Vinay
... View more
10-12-2017
02:55 AM
Hello Team, Is it possible to ingest data to Azure SQL DB directly using NiFi? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
10-12-2017
02:05 AM
1 Kudo
Hello Team, I am trying to ingest data to Azure Data Lake Store using NiFi from my laptop. I followed this article (https://community.hortonworks.com/articles/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html) and built a template but I get the below error when I run it. PutHDFS[id=015f1026-453e-1bf8-2a30-9b94c2e08930] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task ........................... Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration;" Please suggest what's wrong. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
05-02-2017
12:13 PM
Thanks, @Wynner.
We are going with AWS cloud instance to setup NiFi Cloud, I wanted to know any specifications required for the cloud server. Thanks.
... View more
05-01-2017
08:24 AM
Hello Everyone, May I please know server configuration to setup NiFi cloud instance? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
04-27-2017
11:53 AM
Hello everyone, We have Ambari enabled HDF 2.0.1 with NiFi 1.0.0.2.0.10-12. I would like to upgrade NiFi to 1.1.2 in Ambari enabled HDF. Please suggest how we can do this. Thanks
... View more
Labels:
04-24-2017
06:14 PM
Thanks @Timothy Spann Can you please share more information on this. Steps or link to any tutorial would be helpful.
... View more
04-24-2017
06:13 PM
Thanks @Edgar Orendain
... View more
04-15-2017
10:31 AM
Hello @Binu Mathew Thanks for the tutorial. I need your help with Kite API. I get "Descriptor location not found" when I try to use below dataset dataset:hdfs://ip-172-31-2-101.us-west-2.compute.internal:8020/tmp/sample_data/parquet . And I get an exception "java.lang.noclassdeffounderror org/apache/hadoop/hive/conf/HiveConf" when I execute below similar command in hadoop server. ./kite-dataset create dataset:hdfs://ip-172-31-2-101.us-west-2.compute.internal:8020/tmp/sample_data/parquet --schema sample.avsc --format parquet
Can you please suggest the configuration I am missing here? Please suggest Thanks
... View more
04-06-2017
03:46 PM
1 Kudo
Hello Everyone! May I please know how do we make use of the below protocols in HDF 2.0: a) Websockets
b) AMQP Please share the information in detail and links if any. Thanks in advance
... View more
Labels:
- Labels:
-
Apache NiFi
03-13-2017
04:43 PM
Yes Raj, I went through all related questions here in HCC. I implemented tail file with nifi-app.log file but we need to process each row from this log as a flowfile and route it based on error/warn/exception keywords in it. And parse it for pushing it to each column of the Postgres table. Moreover in Ambari enabled NiFi nifi-app.log doesn't contain error messages but only heartbeat log, I couldn't figure out which log file have the errors as of now. To avoid the above-mentioned way I wanted to make use of Rest-API using which we can pick up the bulletins. This seems much easier way.
... View more
03-13-2017
08:31 AM
@Sunile Manjee Yes Sunile I have created a scenario to get an error and I see the error bulletin. PFB. When I start the above invoke-http processor I get an HTML file without error log, I couldn't infer anything from it. PFB. And I don't see the error in Ambari solr infra webpage which I see in bulletin OR in /var/log/ambari-infra-solr . Can you please suggest how to grab errors/warnings from ambar-infra? Thanks.
... View more
03-11-2017
02:32 AM
I would like to capture error messages from bulletin in NiFi and store them in Postgres for audit purposes. I tried with InvokeHTTP processor and NiFi Rest-API end points available but couldn't fetch any error message. Am I missing any step here? Suggest how to get this done.
... View more
Labels:
- Labels:
-
Apache NiFi
03-11-2017
02:18 AM
@Constantin Stanca The queue was piling up without pushing data to Postgres so I stopped Setup SQL Query and the queue stopped at 9900. I see 10000 in the queue if I don't stop it. I don't get any error bulletin. Earlier I could see error/warn logs in nifi-app.log file. Now I have Ambari installed NiFi, I could only see heartbeat messages in nifi-app.log. I also didn't see any error in Ambari LogSearch.
... View more
03-10-2017
06:21 PM
@Constantin Stanca Thank you for your response for my another question. I want to understand why the data is being queued when the PutSQL processor is running? Even though all other processors are stopped the flow files are still in the queue. I don't think what you said would solve my issue.
... View more
03-09-2017
10:35 AM
1 Kudo
I am trying to push data to Postgres using NiFi. I can see the data being populated in the insert query. Data is queued up on
to the PutSQL processor but unable to push to the table. Please suggest
what's wrong here. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
03-08-2017
01:52 PM
Even though nifi-app.log contains every log, the bulletin message is small and precise. The idea is to capture the error message from bulletin and store it in table/file for auditing purposes. We have new versions since a year, hoping we have some way for this. Please share if there is any way to capture the error messages from the bulletin.
... View more
03-08-2017
11:42 AM
This link shares the code of it, I wanted to know how to configure nifi processors (flow) and where/how to use the endpoints in those processors to capture the bulletin.
... View more
03-08-2017
10:14 AM
May I know which NiFi processor should we use here? and In which fields we need to add these endpoints. Please share an example template if you have. Thanks
... View more
03-06-2017
12:17 PM
1 Kudo
Two servers A and B are in NiFi Cluster (HDF 1.0) . I installed Ambari server with mpack in A to have enterprise features ( HDF 2.0 ) . Installed ambari agent and had setup password less SSH setup in A and B servers. I am trying to register A and B hosts in ambari (running on A) but I get "Host checks were skipped on 2 hosts that failed to register." 1. Anything wrong in what I did above? Please guide me if something went wrong. 2. I couldn't figure out why I get above error message even after setting up ipassword less SSH and installed ambari agent in both the servers. thanks
... View more
Labels:
03-03-2017
11:57 AM
How to achieve process failover, processgroup failover and node level failover in NiFi?
... View more
Labels:
- Labels:
-
Apache NiFi