@surajnag This is a great question.  Here are some of my ideas which I have used in development and production nifi flows.   During development route all flow exceptions to an output port.  I call these EOL, or End of Line and increment them with #s, like EOL1, EOL2, EOL3.  I use these to hold failure, retry, original, no-retry, etc type outbound connections during testing.  As I move to production, some of these may be auto terminated, and some may remain. In production, I route ALL major points of failure which are not auto terminated to an output port called ERROR.  In my flows I sometimes have multiple ERROR output ports for difference purposes.   Depending on the use case, the error is sent to some type of event notification.  For Example:  send an email, or post to a slack channel.  In other cases these are routed to a process group that is in a stopped/disabled state.  Based on the error, I may make some change in the flow, and then enable/start the flow to create a method to REPLAY that flowfile. In reference to successful end of flow, I may just auto terminate, as I want to assume the flow is always on, and everything has succeeded if it is not routed as above in #1 or #2.  In other cases the bottom or end of the flow is also routed to some type of Event Notification.  However, be careful here, as you could create tons of notifications depending on your flow. I hope some of these ideas are helpful.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven ... View more

@wenfeng  Check that the user hue is able to write to the target directory.  You may need to create a hue user directory with proper permissions if the hue user cannot write to the hive user directory.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven ... View more

@alex15   I suspect the issue you have is that nifi use is not able to execute the script. Make sure the user ownership of the file is correct, also confirm read/write permissions.  In unix/linux these are chown and chmod.        If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven ... View more

1. Check on the user permission of the jar file you added to your class path. Make sure jar file is in right place. 2. When you add driver manually also add this to sqoop command  --driver =  org . postgresql . Driver  ... View more

@lukolas The example you provide sees to be REGEX not expression language.  You would need to test some kind of expression language in that Topic Name(s) property.   Another suggestion would be to have a file, or for example generateflowfile, which contains a list of topics.  Then split/extract that list into attributes, then send that attribute to the topic name.     Having a ton of topics going to single processors can become a bottleneck which creates tons of downstream flowfiles from each topic.  So be careful with tuning and concurrency in reference to number and topics and messages per topic.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@CSRabbit  Installing the PostgreSQL JDBC Driver Download the PostgreSQL JDBC driver from http://jdbc.postgresql.org/download.html and copy it to the/var/lib/sqoop/ directory. For example: $ curl -L 'http://jdbc.postgresql.org/download/postgresql-9.2-1002.jdbc4.jar' -o postgresql-9.2-1002.jdbc4.jar $ sudo cp postgresql-9.2-1002.jdbc4.jar /var/lib/sqoop/  Snippet from (reference) for cloudera 5.9x but should be similar for other versions, ambari/hdp, or native sqoop.  Just make sure you have the right download url and path for sqoop.      If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

I suspect you have not completed a step, or missing something.  The cacerts works for me in all cases if the cert is publicly trusted (standard public cert from public CA) which it should be.    You should share info on the configurations you tried and what if any errors you got from that.  The bare minimum settings you need for that are keystore (file location), password, key type (jks), and TLS version.   Assuming you copied your java cacert file to all nodes as /nifi/ssl/cacerts the controller service properties should look like:       If cacerts doesnt work, then you must create keystores and/or trust stores with the public cert.   Use the openssl command to get the cert.  That command looks like:   openssl s_client -connect https://secure.domain.com   You can also get it from the browser when you visit the elk interface; for example cluster health, or indexes.   Double click cert lock icon in the browser then use the browser's interface to see/view/download public certificate.  You need the .cer or .crt file.   Then you use the cert to create the keystore with keytool commands.  An example is:   keytool -import -trustcacerts -alias ambari -file cert.cer -keystore keystore.jks   Once you have created a keystore/truststore file you need to copy it to all nifi nodes, ensure the correct ownership, and make sure all the details are correct in the SSL Context Service.  Lastly you may need to modify the TLS type until testing works.   Here is working example of getting the cert and using it with keytool from a recent use case:   echo -n|openssl s_client -connect https://secure.domain.com | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > publiccert.crt keytool -import -file publiccert.crt -alias astra -keystore keyStore.jks -storepass password -noprompt   keytool -import -file publiccert.crt -alias astra -keystore trustStore.jks -storepass password -noprompt   mkdir -p /etc/nifi/ssl/ cp *.jks /etc/nifi/ssl chown -R nifi:nifi /etc/nifi/ssl/     ... View more

@HarshR  You need to configure your SSLContextService with a keystore/truststore built with the cert you get from the Elasticsearch Cluster.   You can also try cacerts that is included with your Java.  This is usually easier to do.  More details here for cacerts:   https://community.cloudera.com/t5/Support-Questions/Connecting-to-DataSift-HTTPS-API-using-NiFi-GetHTTP/td-p/102276     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@Francesco_Fa   If you do not want to use embedded sqlite, the options available for this are: Mysql Postgres Oracle https://docs.gethue.com/administrator/administration/database/       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@Francesco_Fa    You will need to configure hue with credentials and information for hbase:   [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname. If hbase.thrift.ssl.enabled in hbase-site is set to true, https will be used otherwise it will use http # If using Kerberos we assume GSSAPI SASL, not PLAIN. ## hbase_clusters=(Cluster|localhost:9090) # HBase configuration directory, where hbase-site.xml is located. ## hbase_conf_dir=/etc/hbase/conf # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # Should come from hbase-site.xml, do not set. 'framed' is used to chunk up responses, used with the nonblocking server in Thrift but is not supported in Hue. # 'buffered' used to be the default of the HBase Thrift Server. Default is buffered when not set in hbase-site.xml. ## thrift_transport=buffered # Choose whether Hue should validate certificates received from the server. ## ssl_cert_ca_verify=true   Once you have the configuration working, you should be able to view hbase within the UI.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@DanMcCray1 Once you have the content from Kafka as a flowfile, your options are not just limited to ExecuteScript.   Depending on the type of content you can use the following ideas:   EvaluateJsonPath - if the content is a single json, and you need one or more values inside the object then this is an easy way to get those values to attributes. ExtractText - if the content is text or some raw format, extractText allows you to regex match against the content to get values to attributes. QueryRecord w/ Record Readers & Record Writer - this is the most recommended method.  Assuming your data has structure (text,csv,json,etc) and/or multiple rows/objects you can define a reader, with schema, output format (record writer), and query the results very effectively.   If you indeed want to work with Execute Script you should start here:   https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922 https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018 https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@P_Rat98 The error above is saying there is an issue with the Schema Name in your record reader or writer.   When inside the properties for Convert Record, click the --> arrow through to the reader/writer and make sure they are configured correctly.   You will need to provide the correct schema name (if it is already an existing attribute) or provide the schema text.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@ujay The solution you are looking for is the DetectDuplicate processor:   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html   This processor is used with Distributed Map Cache Client and Server (Controller Services) to deduplicate a flow based on your criteria.  I have a template demo here:   https://github.com/steven-matison/NiFi-Templates/blob/master/DetectDuplicate_DistributedMapCache_Demo.xml     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@P_Rat98 You need to set the filename (Object Key) of each parquet file uniquely to save different S3 files.   If that processor is configure to just ${filename} then it will over write additional executions.   For the second option, if you have split in your data flow,  the split parts should have key/value pair for the split and total splits.  Inspect your queue and list attributes on split flowfiles for these attributes.  You use these attributes with MergeContent to remerge everything back together into a single flowfile.   You need to do this before converting to parquet, not after.     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

@Mike in Austin Thanks for sharing that link.   I had lost the one I book marked last year.   Since you guys are both Cloudera Employees, can you comment on enterprise customers with HDP 2.6.5 which cannot or will not be able to upgrade or replace HDP for many years?    That December 2020 date is right around the corner.    Is Cloudera going to truly stop providing support or are these customers just going to get support elsewhere?   I know some customers on 2.6.5 that could still be on that platform as legacy 5+ years from now due to just how long it takes in the government sector to replace/upgrade technology. ... View more