How do I create the folder name with current times...

Jasthi · ‎06-12-2019

I am trying to create the directory with current timestamp as the name of the folder in hdfs what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?

Jasthi · ‎06-12-2019

Hi @Shu I am trying to create the directory with current time stamp in hdfs and as above I tried but did not work since it doesnt have extract_date what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?

Jasthi · ‎10-17-2018

@Arun A K Hi Arun, did anyone resolve this issue ?

Jasthi · ‎10-12-2018

hey @aashish soni did you fix this ?

Jasthi · ‎10-12-2018

@ Jonathan Sneep Thank you so much for help it is working now.....

Jasthi · ‎10-11-2018

@Jonathan Sneep https://community.hortonworks.com/articles/87217/change-nifi-flow-using-rest-api-part-1.html as per step 7 I used the below command : curl -X PUT -H 'Content-Type: application/json' -d '{"component":{"id":"1ffd1b86-0165-1000-ffff-ffffe8f36d71","name":"GetFile","config":{"Input Directory":"/tmp/hortontest","Keep Source File":"false"},"revision":{"clientId":"5f10fbca-0166-1000-ae68-13829a991131","version":2}}' http://nifinode:9090/nifi-api/processors/1ffcfa18-0165-1000-ffff-ffffa779f243 Output: Unrecognized field "Input Directory" (class org.apache.nifi.web.api.dto.ProcessorConfigDTO), not marked as ignorable (17 known properties: "yieldDuration", "descriptors", "runDurationMillis", "concurrentlySchedulableTaskCount", "schedulingStrategy", "autoTerminatedRelationships", "customUiUrl", "lossTolerant", "properties", "comments", "schedulingPeriod", "defaultConcurrentTasks", "annotationData", "executionNode", "penaltyDuration", "defaultSchedulingPeriod", "bulletinLevel"]) at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 105] (through reference chain: org.apache.nifi.web.api.entity.ProcessorEntity["component"]->org.apache.nifi.web.api.dto.ProcessorDTO["config"]->org.apache.nifi.web.api.dto.ProcessorConfigDTO["Input Directory"])

Jasthi · ‎10-11-2018

Thanks for quick response Jon. Is the only way I could do that or are any other ways I can modify the processor properties with rest api calls ?

Jasthi · ‎10-11-2018

I have been trying to change the GetFile processor properties (Input Directory,keep resource file...etc) using rest api call but getting below response : Command: curl -i -X PUT -H 'Content-Type: application/json' -d '{"revision":{"version":3,"clientId":"5f10fbca-0166-1000-ae68-13829a991131"},"processor":{"id":"1ffcfa18-0165-1000-ffff-ffffa779f243","parentGroupId":"1fe8ebbf-0165-1000-0000-00003595dafd","config":{"properties":{"Input Directory":"/tmp/hortontest"}}}}' http://node1:9090/nifi-api/controller/process-groups/1fe8ebbf-0165-1000-0000-00003595dafd/processors/1ffcfa18-0165-1000-ffff-ffffa779f243 Output: HTTP/1.1 404 Not Found Date: Thu, 11 Oct 2018 13:48:40 GMT X-Frame-Options: SAMEORIGIN Content-Type: text/plain Vary: Accept-Encoding Content-Length: 42 Server: Jetty(9.4.3.v20170317) The specified resource could not be found.

Jasthi · ‎09-06-2018

@Sharath V Yeah sure sharath. Here I have attached the screen shot of my current workflow which is running in the production to get email whenever any processor fails it will alert you with email and the reason.nifi-email-alert.png In attached flow I was tailing my nifi-app.log file with current time and then routing the content if it contains with ERROR. Then I am executing the script for getting the process group name wherever particular processor failed below is the script : def flowFile = session.get() if(!flowFile) return processGroupName = context.procNode?.getProcessGroup().getName() processGroupId = context.procNode?.processGroupIdentifier ?: 'unknown' flowFile = session.putAttribute(flowFile, 'processGroupName', processGroupName) flowFile = session.putAttribute(flowFile, 'processGroupId', processGroupId) session.transfer(flowFile, REL_SUCCESS) And finally I am sending both error message and process group name in the email with attached file.

Jasthi · ‎08-31-2018

As you said I followed the steps but when I do ambari-server upgrade its showing below : Using python /usr/bin/python Upgrading ambari-server INFO: Upgrade Ambari Server INFO: Updating Ambari Server properties in ambari.properties ... WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings INFO: Updating Ambari Server properties in ambari-env.sh ... INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization. ERROR: No management packs found that can be upgraded! ERROR: Exiting with exit code -1. REASON: No management packs found that can be upgraded! I tried to start ambari server but its showing below: ERROR [main] AmbariServer:1078 - Failed to run the Ambari Server org.apache.ambari.server.AmbariException: Current database store version is not compatible with current server version, serverVersion=2.6.2.0, schemaVersion=2.5.1 at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkDBVersionCompatible(DatabaseConsistencyCheckHelper.java:245) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1063)

Jasthi · ‎08-31-2018

@amarnath reddy pappu Hi amar, I did not do in the way you mentioned If I want to proceed in the correct way what would be steps should I follow ? Should I uninstall everything or any other workaround ?

Jasthi · ‎08-31-2018

Hi @amarnath reddy pappu thanks for quick response. I could see this error after I was trying to install new version from ambari. Here is full trace from one of ambari agent (all agents are reporting same). Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 487, in <module> InstallPackages().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 93, in actionexecute self.stack_root_folder = Script.get_stack_root() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 667, in get_stack_root stack_root = json.loads(stack_root_json) File "/usr/lib/ambari-agent/lib/ambari_simplejson/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 335, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 353, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

Jasthi · ‎08-31-2018

I am upgrading HDF 3.0.1 to HDF 3.1.2 with ambari 2.6.2.0 But it fails with the error : raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Right now I have upgraded ambari sucessfully from 2.5.1 to 2.6.2 and ran the mpack to register the version into ambari I could do that and when I trying install it its throwing error like above. Could someone give any suggestions?

Jasthi · ‎08-21-2018

@Steven Matison Thanks for your quick response and It is working fine now but I have two concerns here: 1.I am getting bunch of emails with same error message rather I would only need one email with all error message. 2.If I include all error message in message body it looks so weird so I am planning to message body contains only "Process Group Name" & "Process Group ID". For process group name I found in HWX blog that ExcuteScript processor will do with below code: def flowFile = session.get() if(!flowFile)return processGroupName = context.procNode?.getProcessGroup().getName() flowFile = session.putAttribute(flowFile,'processGroupName', processGroupName) session.transfer(flowFile, REL_SUCCESS) Where should I have ExcuteScript processor in my current flow ? (my current flow is :TailFile--->RouteOnContent-->ExtractText-->PutEmail.) How would I configure that my message body contains only "Process Group Name" & "Process Group ID" ? Thanks, Jasthi.

Jasthi · ‎08-17-2018

@Steven Matison Could you please elaborate more ? Right now I am using TailFile--->RouteOnContent-->PutEmail. So in this flow i am able to get the only errors from log file in attached file in an email. My question is how can I capture those error message and put as the message of PutEmail processor ?

Jasthi · ‎08-16-2018

I have a Process Group with 6 individual processor are running.So if one or two of Processors throws the error message in bulletins that needs to captured and sent an email with error message ? Example workflow: getfile---->Puts3object-->Updateattribute-->AttributeTOJSON-->ReplaceText-->PublishKafka if workflow throws a error at Puts3object & AttributeTOJSON i need capture both bulletins message and then send the content to PutEmail processor . Could someone give the idea how to achieve this ?

Jasthi · ‎07-31-2018

No I have been using the plain getFile which is having input as a simple folder and we are not doing any regex on those files. FYI.... In my entire workflow all processors are behaving the same.

Jasthi · ‎07-31-2018

I am using GetFile processor..

Jasthi · ‎07-31-2018

Yeah as you guys said I did restart the instance it worked but If I stop the processor and start again then same issue. Should I restart nifi Instance every time whenever processor stops and starts since we are dev phase we have to stop and start the processor ?

Jasthi · ‎07-30-2018

Thanks for quick response @Matt Burgess Its not disappearing since long time. Is there anyway I could kill the threads forcefully and if not Could you let me know what will be the solution to overcome this ? Thanks Jasti.

Jasthi · ‎07-30-2018

We have processors in process group but some of them are not showing up the start button. Here I have attached the screen shot..... Could someone help on this ? nifi-error.png

Jasthi · ‎07-10-2018

somedata = COGROUP clientdata BY clienid, enrolldata BY clientid, claims BY clientid FOREACH somedata GENERATE flatten(clientdata , enrolldata , claims) as ( all meembers inside clientdata, enrolldata , claims) // at once so that we can send to hbase to one rowkey with multiple column familes.

Jasthi · ‎06-18-2018

I have data center A and B. Whenever clients send the data to FTP that needs to write to Both A & B data centers respectively.So that If my Data center A is down I can run my jobs on Data center B. Could someone please drop suggestions how to do implement this approach ? Thanks...

Jasthi · ‎06-13-2018

Thanks for response. Small correction.. From custom code the message is gonna publish to kafka queue from there I am picking the JSON message to pass to EvaluateJsonPath processor. the EvaluateJsonPath has now two values one is source path and one is destination path. As you said you can use FetchS3Object to get the file from S3, how should I pass the source path to FetchS3Object processor and then how should I pass the destination path to PutFile processor? Could you explain me briefly ? Right now my flow is like attached screen shot. PFA...

Jasthi · ‎06-12-2018

S3bucket path of file (source) and destination path(local file share) is sent from a custom java code. Nifi has to convert this into JSON format and extract the file from S3 bucket and place it in local file share. Can somebody share the thoughts on this and give me a idea how to implement this ?

Jasthi · ‎05-10-2018

yup got it thanks.By mistake I have unchecked Enable SSL checkbox in NIFI config after checking back I could see them . Thanks @Matt Clarke

Jasthi · ‎05-10-2018

I couldn't see any user it was empty that is the problem.The earlier image is reference and the one now I have attached is my screen shot . I hope this clears your doubt & its my bad It wasn't clear in earlier posts . you can refer above image ( nifi-ssl-my-screen.png) that is my current screen shot & the one I mentioned earlier was reference that is nifi-ssl.png . @Umair Khan

Jasthi · ‎05-10-2018

nifi-ssl-my-screen.pngI got you. the image I have attached is from different blog so like the image its not showing up for me when I am logged into any of nifi nodes. The earlier image is reference and the one now I have attached is my screen shot . I hope this clears your doubt & its my bad It wasn't clear in earlier posts .

Jasthi · ‎05-10-2018

I am expecting to see user name (CN=niifiadmin, OU=HORTONWORKS) at top right corner that I mentioned in the image of the first post. I am using the User/client certificate that are generated by NiFi TLS-toolkit

Jasthi · ‎05-10-2018

@Arti Wadhwani

Online	Offline
Last Visited	‎07-03-2018 10:57 AM

Date Registered	‎07-03-2018 07:10 AM
Last Visited	‎07-03-2018 10:57 AM
Total Messages Posted	63
Total Kudos Received	1

How do I create the folder name with current times...

Re: NiFi - Creating the output directory from the ...

Re: Issue Configuring ListS3

Re: Error Starting Streaming Analytics Manager

Re: How to change the nifi processor properties wi...

Re: How to change the nifi processor properties wi...

Re: How to change the nifi processor properties wi...

How to change the nifi processor properties with n...

Re: NiFi email alerting with failure message which...

Re: HDF 3.0.1 to HDF 3.1.2 express upgrade fails?

Re: HDF 3.0.1 to HDF 3.1.2 express upgrade fails?

Re: HDF 3.0.1 to HDF 3.1.2 express upgrade fails?

HDF 3.0.1 to HDF 3.1.2 express upgrade fails?

Re: NiFi email alerting with failure message which...

Re: NiFi email alerting with failure message which...

NiFi email alerting with failure message which we ...

Re: In a Nifi-workflow some of the processors are ...

Re: In a Nifi-workflow some of the processors are ...

Re: In a Nifi-workflow some of the processors are ...

Re: In a Nifi-workflow some of the processors are ...

In a Nifi-workflow some of the processors are not ...

How to flatten pig bags ?

How to write the same data to two different data c...

Re: Nifi for reading json data and extract source ...

Nifi for reading json data and extract source and ...

Re: After Enabling the ssl in NIFI cluster its now...

Re: After Enabling the ssl in NIFI cluster its now...

Re: After Enabling the ssl in NIFI cluster its now...

Re: After Enabling the ssl in NIFI cluster its now...

Re: After Enabling the ssl in NIFI cluster its now...