About DennisJaheruddi

DennisJaheruddi · ‎12-24-2019

This is more a Python question than a Nifi one, perhaps this will help: import pyminizip pyminizip.compress("myfile.txt", "myzippedfile.zip", "mypassword", compression_level) I would recommend you to first see if you can get the python part working, and then worry about how to call it from Nifi.

DennisJaheruddi · ‎12-04-2019

I understand this was hard to find. One point that is not relevant now, but may help with the general understanding: If you make a typo in the field names, avro will just ignore the actions of the right field and instead process the new field, which will be empty. This is technically quite a different problem than when you actually make a typo in the Type, as those are possible to check against the list of possible types. So detection of typos in field names is hard, and despite what the name suggests it seems LogicalType is actually a field name from Nifi perspective.

DennisJaheruddi · ‎11-14-2019

The main point that is always communicated is that running our version in a Data Flow platform, brings you direct integration with other components. Both functional and nonfunctional (e.g. with Ranger and Atlas). That being said, as you can see there are also improvements to the product itself, but I am not aware of a clear public list with this, because it fluctuates a lot over time (First we discover points, and then we commit many of these back to the apache project eventually). Of course having the latest Apache Nifi can have some benefits, but overall I recommend using the latest Cloudera version of Nifi as that one has seen more testing and is less likely to see surprises.

Raj_B · ‎09-25-2019

The question posted is not a hypothetical one, it is a real use case. fyi, here is another thread related to partial file consumption; - https://stackoverflow.com/questions/45379729/nifi-how-to-avoid-copying-file-that-are-partially-written that thread does not suggest the OS automatically takes care of this. The solution proposed there is to add a time wait between ListFile and FetchFile, but in our case, the requirement is to wait for an indicator file before we start file ingestion;

DennisJaheruddi · ‎09-25-2019

This is slightly confusing, but if you have a Mysql query to generate the difference, then Nifi can get the data from there and convert to XML if needed. As generating the difference seems to be the main challenge here, perhaps start by finding a way to do this, and from here it should be trivial to design the Nifi flow around this. ---- If this answers the question, consider marking this as the answer

DennisJaheruddi · ‎09-23-2019

Sending an email every time an error occurs may be problatic. The only thing that seems to come close is the site to site bulletin report Processor. As mentioned you can find the messages in the logs. If your main concern is accessibility, look into LogSearch on HDP.

DennisJaheruddi · ‎09-23-2019

For this usecase the most natural solution is probably as follows: 1. Send message to a script for processing 2. Let the script turn your 1 row into an output of N rows 3. Optional: Split the rows in Nifi

Shelton · ‎08-30-2019

@irfangk1 If its an HDP cluster then I assume you are using Ambari for managing the HDF cluster, you will need to first prepare the 2 new hosts see the Prepare the EnvironmentCloudera document Then add the hosts to the cluster see Add host to cluster Thereafter add HDF to these 2 new nodes it follows the same procedure as adding HDF Services on an existing HDP Cluster hTH

DennisJaheruddi · ‎08-29-2019

A quick search suggests that libhdfs can do it, but I have not tried it myself.

DennisJaheruddi · ‎08-28-2019

As explained elsewhere by Andy: You can accomplish this with a ConvertRecord processor. Register an Avro schema describing the expected format in a Schema Registry (controller service), and create a CSVReader implementation to convert this incoming data to the generic Apache NiFi internal record format. Similarly, use a CSVRecordSetWriter with your output schema to write the data back to CSV in whatever columnar order you like. For more information on the record processing philosophy and some examples, see Record-oriented data with NiFi and Apache NiFi Records and Schema Registries.

Online	Offline
Last Visited	‎12-15-2021 03:18 AM

Member Since	‎01-07-2019 03:54 AM
Last Visited	‎12-15-2021 03:18 AM
Posts	220
Kudos received	31

Cloudera Community

Re: 在启用kerberos的集群flink程序如何连接集群外未启用认证的kafka

Re: Attribute validation against MSSQL database

Re: Put array with Dates on nifi flowfile

Re: NiFi templates don't include all controller se...

Re: Concatenations of Multiple Attributes in Nifi

Re: How to zip file from ExecuteStreamCommand

Re: Timestamp Format Issue (NiFi's PutDatabaseReco...

Re: Cannot Update or Overwrite variables

Re: File wait in NiFi

Re: NiFi Compare 2 xml files and create xml file w...

Re: How capture the Nifi processor error messages.

Re: Pivot one column to rows from CSV file along ...

Re: i want to add 2 host or instance in NiFi

Re: Python script to process files on HDFS

Re: Mapping different header names to hive column ...