@ravi_sh_DS The solution you are looking for is UpdateRecord.   Here is a great article with full info:   https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven ... View more

@Gubbi    The solution you are looking for is ConvertRecord + ParquetRecordReader + CSVRecordWriter.  The ParquetRecordReader is included in NiFi 1.10 and up.  If you have an older nifi, here is a post where I talk about adding the required jar files to nifi 1.9 (older version doesn't have parquet):   https://community.cloudera.com/t5/Support-Questions/Can-I-put-the-NiFi-1-10-Parquet-Record-Reader-in-NiFi-1-9/td-p/286465   Another suggestion: If you are working with nifi and hadoop/hdfs/hive,  you could store the raw parquet, create external hive table on parquet, then select results and insert them into similar table of csv format.  Then you select the csv table results and create csv file.      Also in order to validate/inspect your parquet, or to read the schema (if you need it for controller services) you use parquet tools:   https://community.cloudera.com/t5/Community-Articles/Build-and-use-Parquet-tools-to-read-parquet-files/ta-p/248629     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

I think the most straight forward would be to drop the infer schema into your version of NiFi.   The procedure is not that hard, you just have to be surgically careful.  The process is explained a bit here in reference to adding parquet jars from new version, into older version.   Be sure to read all the comments:   https://community.cloudera.com/t5/Support-Questions/Can-I-put-the-NiFi-1-10-Parquet-Record-Reader-in-NiFi-1-9/td-p/286465   ... View more

@mike_bronson7  oddly enough, the top post recommendations is this same question you asked 2 years ago:   https://community.cloudera.com/t5/Support-Questions/how-to-change-hostnames-on-all-machines-in-ambari-cluster/m-p/217984#M179885   To confirm, the hostname and the domain name are one in the same.  The "domain.com" is just part of the hostname.  The first part is the subdomain.  So the procedure to change "domain.com" is the same as subdomain.    HOWEVER,  I would highly recommend testing this in a dev cluster before production, especially if you have ssl/kerberos.   This is a major change that affects ambari and all components.  If the ambari command agains the new json mapping fails, or there is any incomplete action within any agent/component,  you are going to bomb the whole cluster.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

Yes.  Auto-termination is how you drop the whole flowfile and all attributes.       If for example, you have a giant attribute of SCHEMA or JSON,  using updateAttribute could be used to empty the value.   However if you do not even need the flowfile anymore, auto-terminate, versus updateAttribute and retain flowfile. ... View more

@Muffex    My recommendation for ingestion process is to always use staging/temporary tables which are managed separately from the master table the data needs to arrive in.   This allows you operate on the staging tables before or after those results are added to the master table w/o effecting the master table.    In your use case, your ingestion process would sqoop to temp,  insert from temp to master table, then drop temp location.    In some of my past implementations of this manner, the temp tables were organized hourly, and they stay active for at least 7 days before a decoupled cleanup job removes anything 7 days old.   This idea was done for auditing purposes, but normally I would create and destroy the data during the ingestion procedure.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@P_Rat98 Creating an API with NiFi using HandleHttpRequest and HandleHttpResponse is something I have done quite a few times for Hortonworks and NiFI Customers.  This is a great use case for NiFi and sending receiving JSON to NiFi, processing JSON, and completing actions downstream is super easy.   I have created a basic template for you which includes the HandleHttpRequest (inbound port 80 call) a process group for doing something with the JSON, and HandleHttpResponse (provides 200 response code) to respond to inbound call.    This is an API in the simplest form with NiFi.   Depending on your use case you can build out Process Api Request Process Group to suit your needs.   Out of the box you should be able to send to import template, add/start the StandHttpContextMap Controller Service, Start the flow, send a call to http://yourhost:80 and have JSON sitting in the bottom of the flow Success Queue.   You can find the template here: https://github.com/steven-matison/NiFi-Templates/blob/master/NiFi_API_with_HandleHttpRequest_Demo.xml     Some API suggestions: Be sure to take a look at both HandleHttp Processors for the properties you can configure.  Ports, hostname, acceptable methods, ssl, authentication, and more. If your API call does not care if the Process API Request finishes,  you can put HandleHttpResponse right after HandleHttpRequest, and let all the downstream work happen after the request/response is completed.  This is common when I expect my API to be only giving inbound data, and doesn't care what the response is (other than just 200 to know it was received).  In this case I accept the payload, return 200, and rest of the flow is decoupled from the connection.     If my processing time is lengthy I usually do this so the system initiating the api call is not left waiting. Once you have the basic framework built,  consider handling errors, and or returning different status codes as a variable (created before the response) in the Status Code for Handle Http Response.  Sometimes I even have different HandleHttpResponse at end of different flow branches.  For example:  if someone sends invalid JSON,  I return maybe 302 or 404 with the invalid error as the content body. Have fun with it.       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more