Member since
07-03-2018
63
Posts
1
Kudos Received
0
Solutions
06-12-2019
08:07 PM
I am trying to create the directory with current timestamp as the name of the folder in hdfs what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
06-12-2019
08:02 PM
Hi @Shu I am trying to create the directory with current time stamp in hdfs and as above I tried but did not work since it doesnt have extract_date what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?
... View more
10-12-2018
01:57 PM
@ Jonathan Sneep Thank you so much for help it is working now.....
... View more
10-11-2018
04:47 PM
@Jonathan Sneep https://community.hortonworks.com/articles/87217/change-nifi-flow-using-rest-api-part-1.html as per step 7 I used the below command : curl -X PUT -H 'Content-Type: application/json' -d '{"component":{"id":"1ffd1b86-0165-1000-ffff-ffffe8f36d71","name":"GetFile","config":{"Input Directory":"/tmp/hortontest","Keep Source File":"false"},"revision":{"clientId":"5f10fbca-0166-1000-ae68-13829a991131","version":2}}' http://nifinode:9090/nifi-api/processors/1ffcfa18-0165-1000-ffff-ffffa779f243 Output: Unrecognized field "Input Directory" (class org.apache.nifi.web.api.dto.ProcessorConfigDTO), not marked as ignorable (17 known properties: "yieldDuration", "descriptors", "runDurationMillis", "concurrentlySchedulableTaskCount", "schedulingStrategy", "autoTerminatedRelationships", "customUiUrl", "lossTolerant", "properties", "comments", "schedulingPeriod", "defaultConcurrentTasks", "annotationData", "executionNode", "penaltyDuration", "defaultSchedulingPeriod", "bulletinLevel"]) at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 105] (through reference chain: org.apache.nifi.web.api.entity.ProcessorEntity["component"]->org.apache.nifi.web.api.dto.ProcessorDTO["config"]->org.apache.nifi.web.api.dto.ProcessorConfigDTO["Input Directory"])
... View more
10-11-2018
03:44 PM
Thanks for quick response Jon. Is the only way I could do that or are any other ways I can modify the processor properties with rest api calls ?
... View more
10-11-2018
02:02 PM
I have been trying to change the GetFile processor properties (Input Directory,keep resource file...etc) using rest api call but getting below response : Command: curl -i -X PUT -H 'Content-Type: application/json' -d '{"revision":{"version":3,"clientId":"5f10fbca-0166-1000-ae68-13829a991131"},"processor":{"id":"1ffcfa18-0165-1000-ffff-ffffa779f243","parentGroupId":"1fe8ebbf-0165-1000-0000-00003595dafd","config":{"properties":{"Input Directory":"/tmp/hortontest"}}}}' http://node1:9090/nifi-api/controller/process-groups/1fe8ebbf-0165-1000-0000-00003595dafd/processors/1ffcfa18-0165-1000-ffff-ffffa779f243 Output: HTTP/1.1 404 Not Found
Date: Thu, 11 Oct 2018 13:48:40 GMT
X-Frame-Options: SAMEORIGIN
Content-Type: text/plain
Vary: Accept-Encoding
Content-Length: 42
Server: Jetty(9.4.3.v20170317)
The specified resource could not be found.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
09-06-2018
07:37 PM
@Sharath V Yeah sure sharath. Here I have attached the screen shot of my current workflow which is running in the production to get email whenever any processor fails it will alert you with email and the reason.nifi-email-alert.png In attached flow I was tailing my nifi-app.log file with current time and then routing the content if it contains with ERROR. Then I am executing the script for getting the process group name wherever particular processor failed below is the script : def flowFile = session.get()
if(!flowFile) return
processGroupName = context.procNode?.getProcessGroup().getName()
processGroupId = context.procNode?.processGroupIdentifier ?: 'unknown'
flowFile = session.putAttribute(flowFile, 'processGroupName', processGroupName)
flowFile = session.putAttribute(flowFile, 'processGroupId', processGroupId)
session.transfer(flowFile, REL_SUCCESS) And finally I am sending both error message and process group name in the email with attached file.
... View more
08-31-2018
08:52 PM
As you said I followed the steps but when I do ambari-server upgrade its showing below : Using python /usr/bin/python
Upgrading ambari-server
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
ERROR: No management packs found that can be upgraded!
ERROR: Exiting with exit code -1.
REASON: No management packs found that can be upgraded! I tried to start ambari server but its showing below: ERROR [main] AmbariServer:1078 - Failed to run the Ambari Server
org.apache.ambari.server.AmbariException: Current database store version is not compatible with current server version, serverVersion=2.6.2.0, schemaVersion=2.5.1
at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkDBVersionCompatible(DatabaseConsistencyCheckHelper.java:245)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1063)
... View more
08-31-2018
05:37 PM
@amarnath reddy pappu Hi amar, I did not do in the way you mentioned If I want to proceed in the correct way what would be steps should I follow ? Should I uninstall everything or any other workaround ?
... View more
08-31-2018
05:19 PM
Hi @amarnath reddy pappu thanks for quick response. I could see this error after I was trying to install new version from ambari. Here is full trace from one of ambari agent (all agents are reporting same). Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 487, in <module>
InstallPackages().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 93, in actionexecute
self.stack_root_folder = Script.get_stack_root()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 667, in get_stack_root
stack_root = json.loads(stack_root_json)
File "/usr/lib/ambari-agent/lib/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/ambari-agent/lib/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
... View more
08-31-2018
03:45 PM
I am upgrading HDF 3.0.1 to HDF 3.1.2 with ambari 2.6.2.0 But it fails with the error : raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded Right now I have upgraded ambari sucessfully from 2.5.1 to 2.6.2 and ran the mpack to register the version into ambari I could do that and when I trying install it its throwing error like above. Could someone give any suggestions?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Cloudera DataFlow (CDF)
08-21-2018
02:15 PM
@Steven Matison Thanks for your quick response and It is working fine now but I have two concerns here: 1.I am getting bunch of emails with same error message rather I would only need one email with all error message. 2.If I include all error message in message body it looks so weird so I am planning to message body contains only "Process Group Name" & "Process Group ID". For process group name I found in HWX blog that ExcuteScript processor will do with below code:
def flowFile = session.get() if(!flowFile)return processGroupName = context.procNode?.getProcessGroup().getName() flowFile = session.putAttribute(flowFile,'processGroupName', processGroupName) session.transfer(flowFile, REL_SUCCESS) Where should I have ExcuteScript processor in my current flow ? (my current flow is :TailFile--->RouteOnContent-->ExtractText-->PutEmail.) How would I configure that my message body contains only "Process Group Name" & "Process Group ID" ? Thanks, Jasthi.
... View more
08-17-2018
09:27 PM
@Steven Matison Could you please elaborate more ? Right now I am using TailFile--->RouteOnContent-->PutEmail. So in this flow i am able to get the only errors from log file in attached file in an email. My question is how can I capture those error message and put as the message of PutEmail processor ?
... View more
08-16-2018
03:12 PM
I have a Process Group with 6 individual processor are running.So if one or two of Processors throws the error message in bulletins that needs to captured and sent an email with error message ? Example workflow: getfile---->Puts3object-->Updateattribute-->AttributeTOJSON-->ReplaceText-->PublishKafka if workflow throws a error at Puts3object & AttributeTOJSON i need capture both bulletins message and then send the content to PutEmail processor . Could someone give the idea how to achieve this ?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
07-31-2018
02:59 PM
No I have been using the plain getFile which is having input as a simple folder and we are not doing any regex on those files. FYI.... In my entire workflow all processors are behaving the same.
... View more
07-31-2018
02:22 PM
I am using GetFile processor..
... View more
07-31-2018
01:32 PM
Yeah as you guys said I did restart the instance it worked but If I stop the processor and start again then same issue. Should I restart nifi Instance every time whenever processor stops and starts since we are dev phase we have to stop and start the processor ?
... View more
07-30-2018
07:23 PM
Thanks for quick response @Matt Burgess Its not disappearing since long time. Is there anyway I could kill the threads forcefully and if not Could you let me know what will be the solution to overcome this ? Thanks Jasti.
... View more
07-30-2018
05:19 PM
We have processors in process group but some of them are not showing up the start button. Here I have attached the screen shot..... Could someone help on this ? nifi-error.png
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
07-10-2018
08:30 PM
somedata
= COGROUP clientdata BY clienid, enrolldata BY clientid, claims BY
clientid
FOREACH
somedata GENERATE flatten(clientdata , enrolldata , claims) as ( all
meembers inside clientdata, enrolldata , claims) // at once so that we can
send to hbase to one rowkey with multiple column
familes.
... View more
Labels:
- Labels:
-
Apache Pig
06-18-2018
07:37 PM
I have data center A and B. Whenever clients send the data to FTP that needs to write to Both A & B data centers respectively.So that If my Data center A is down I can run my jobs on Data center B. Could someone please drop suggestions how to do implement this approach ? Thanks...
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
06-13-2018
02:41 PM
Thanks for response. Small correction.. From custom code the message is gonna publish to kafka queue from there I am picking the JSON message to pass to EvaluateJsonPath processor. the EvaluateJsonPath has now two values one is source path and one is destination path. As you said you can use FetchS3Object to get the file from S3, how should I pass the source path to FetchS3Object processor and then how should I pass the destination path to PutFile processor? Could you explain me briefly ? Right now my flow is like attached screen shot. PFA...
... View more
06-12-2018
07:41 PM
S3bucket path of file (source) and destination
path(local file share) is sent from a custom java code. Nifi has to convert
this into JSON format and extract the file from S3 bucket and place it in local
file share. Can somebody share the thoughts on this and give me a idea how to implement this ?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
05-10-2018
05:26 PM
yup got it thanks.By mistake I have unchecked Enable SSL checkbox in NIFI config after checking back I could see them . Thanks @Matt Clarke
... View more
05-10-2018
05:10 PM
I couldn't see any user it was empty that is the problem.The earlier image is reference and the one now I have attached is my screen shot . I hope this clears your doubt & its my bad It wasn't clear in earlier posts . you can refer above image ( nifi-ssl-my-screen.png) that is my current screen shot & the one I mentioned earlier was reference that is nifi-ssl.png . @Umair Khan
... View more
05-10-2018
05:06 PM
nifi-ssl-my-screen.pngI got you. the image I have attached is from different blog so like the image its not showing up for me when I am logged into any of nifi nodes. The earlier image is reference and the one now I have attached is my screen shot . I hope this clears your doubt & its my bad It wasn't clear in earlier posts .
... View more
05-10-2018
02:40 PM
I am expecting to see user name (CN=niifiadmin, OU=HORTONWORKS) at top right corner that I mentioned in the image of the first post. I am using the User/client certificate that are generated by NiFi TLS-toolkit
... View more
05-10-2018
02:26 PM
@Arti Wadhwani
... View more