@MWM @cotopaul   If you get the record reader/writer using the schema(s) you want,  you do not have to do any magic to convert values, it should just work.     Only use, inferSchema long enough to get the structure when you have none.  Then copy/paste it and use it as @cotopaul has described in place of InferSchema.  You can also use Schema Registry.  Make the edits you need to satisfy reader (upstream), writer (downstream) as they are sometimes needing minor adjustments like in this case. ... View more

@vaishaakb  I noticed this same activity after deploying to the latest version of CM and after deploying parcels in my Lab cluster.  I started getting P2P violations from my IDS and IPS. Is there any way to control the external p2p process?  I've gone ahead and attached screen captures from my firewall.  CDP - 7.1.9-1.cdh7.1.9.p0.44702451 - CM - 7.11.3 Example of the detection: All 5 of my nodes repeatedly trying to talk across the globe.  ... View more

@kuhbrille Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks. ... View more

if the partition data exists like below: <s3:bucket>/<some_location>/<part_column>=<part_value>/<filename> you can create a external table by specifiying above location and run 'msck repair table <table_name> sync partitions' to sync partitions. validate the data by running some sample select statements. Once it's done you can create new external table with another bucket and run insert statement with dynamic partition. Ref - https://cwiki.apache.org/confluence/display/hive/dynamicpartitions ... View more

@tqiu Circling back to see if you got a chance to review our update? - V ... View more

Hi @rupeshh I am also facing a similar situation and wanted to clarify from you a few things. I see in the Configuration for the ExecuteStreamCommand you have referred to the venv inside the repository for the command path, does that mean you have the venv within the docker container? or are you referring to the one outside the docker container (in the host machine)?         ... View more

So I copied only those nars which we use, and container could launch now. Though I have to remove few nars which were causing issues, like nifi-ssl-context-service-nar-1.10.0.nar. And now existing flows dont have issues with properties which are obsolete in 1.22.0 as 1.10.0 nars are used for those components. Thanks for all the inputs. ... View more

@JohnnyRocks, as @steven-matison said, you should avoid linking so many ReplaceText. I am not quite sure I understood your flow exactly, but something tells me that before reaching ReplaceText, something is not properly configured in your NiFi Flow. First of all, when using the classic Java Data Format, MM will always transpose in a two digit month, meaning that month from 1 to 9 will be automatically appended with a leading zero. "dd" will do the same trick but for days. As I see in your post, you said that your CSV reader is configured to read the data as MM/dd/yy, which should be fine, but somehow something is missing here ---> How do you reach the format of dd/MM/yyyy? What I would personally try to do is to convert all those date values in the same format. So instead of all those ReplaceText, I would try to insert an UpdateRecord Processor, where I would define my RecordReader and my RecordWritter with the desired schemas (make sure that your column is type int with logicaly type date). Next, in that processor, I would change the Replacement Value Strategy into "Record Path Value" and I would press on + and add a new property. I would call it "/Launch_Date" (pay attention to the leading slash) and I would assign it the value " format( /Launch_Date, "dd/MM/yyyy", "Europe/Bucharest") " (or any other timezone you require -- if you require your data in UTC, just remove the coma and the timezone). ... View more

@kothari  It is not Ranger's job to inform the client applications using Ranger what users belong to what group.  Each client application is responsible for determining which groups the user authenticated into that service belong to. The policies generated by Ranger are downloaded by the client applications.  Within that downloaded policy json will be a resource identifier(s), list if user identities authorized (read, write, and/or delete) , and list of group identities authorized (read, write, or delete) against each resource identifier.  So when client checks the downloaded policies from Ranger it is looking for the user identity being authorized and if client is aware of the group(s) that user belongs to, will also check authorization for that group identity.   so in your case, it i s most likely that your client service/application has not been configured with the same user and group association setup in your Ranger service.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

I'm facing the same issue with the ADF Hive connector. It would be great if you could provide your configuration details. ... View more