Member since
07-03-2018
64
Posts
2
Kudos Received
0
Solutions
07-28-2022
10:27 AM
Hi @Jasthi , These are alerts coming from the CM. For more details on those, you can click on specific alerts which can give more clarity as to why the alert is seen. For more details, you can refer the below doc : https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ht_regionserver.html
... View more
09-30-2019
07:53 AM
@Jasthi How can you get the processgroupname and ID ? this script get the current processgroup and not the one where the error occur . I am facing the same problem and i trying to create a log error flow that logs everything on the server , right now i was able to get the error and send by email , but there is nothing that indentifies in what flow the error happened
... View more
06-13-2019
01:44 AM
@Veerendra Nath Jasthi In UpdateAttribute add new attribute as ts value as ${now():format("yyyy_MM_dd_HH_mm_ss_SSS")} Example: ts attribute will have value as 2019_06_12_20_42_26_762 Then in PutHDFS processor configure directory as /<path>/${ts} (or) You can skip UpdateAttribute processor and directly use directory name as /<path>/${now():format("yyyy_MM_dd_HH_mm_ss_SSS")} In PutHDFS processor. This will create a directory in HDFS with current timestamp value. You can change the format of the timestamp using NiFi expression language.
... View more
10-12-2018
01:57 PM
@ Jonathan Sneep Thank you so much for help it is working now.....
... View more
08-11-2018
03:35 PM
I've seen the problem you describe with getfile processor when I used it with nifi deployed on Windows OS and the directory I listed had very large number of recursive subdirectories. In source files of nifi 1.3 or 1.4 I noticed there was a code replacing listfiles with DirectoryInputStream methods, but later it was removed
... View more
06-13-2018
02:41 PM
Thanks for response. Small correction.. From custom code the message is gonna publish to kafka queue from there I am picking the JSON message to pass to EvaluateJsonPath processor. the EvaluateJsonPath has now two values one is source path and one is destination path. As you said you can use FetchS3Object to get the file from S3, how should I pass the source path to FetchS3Object processor and then how should I pass the destination path to PutFile processor? Could you explain me briefly ? Right now my flow is like attached screen shot. PFA...
... View more
05-10-2018
05:29 PM
@Veerendra Nath Jasthi Glad to hear we got this all worked out... 🙂 Please take a moment to login and click the "accept" link for this answer to close out the thread. - Thank you, Matt
... View more
05-04-2018
06:50 PM
@Veerendra Nath Jasthi The DN there is coming from the keystore being used by your NiFi nodes. I have no idea why the certs created for your servers all have nifiadmin in them.... ... But just like your user DN, the node identities must match exactly with what is in those server certs in the keystore.. - <property name="Node Identity 1">CN=nifiadmin, OU=NIFIrsdevhdf1.medassurant.local, OU=NIFI</property>
<property name="Node Identity 2">CN=nifiadmin, OU=NIFIrsdevhdf2.medassurant.local, OU=NIFI/</property>
<property name="Node Identity 3">CN=nifiadmin, OU=NIFIrsdevhdf3.medassurant.local, OU=NIFI</property> - so you will need to edit your node identities so they match the above and once again stop NiFi, remove your users.xml and authorizations.xml files, and then start NiFi again via Ambari. - Thank you, Matt
... View more
03-01-2018
07:14 PM
1 Kudo
You need to use a new tag, not one of the existing ones. Typically it is the "version" keyword followed by current timestamp. If you don't absolutely need to use the REST API or you don't want to deal with the version tag, you should use configs.sh/configs.py (the sh is not supported in some older Ambari versions). Sample get call is - /var/lib/ambari-server/resources/scripts/configs.py -a get -l <ambari_server_host> -n <cluster_name> -c capacity-scheduler -f /tmp/cs.json Sample output in /tmp/cs.json is - {
"properties": {
"yarn.scheduler.capacity.maximum-am-resource-percent": "0.4",
"yarn.scheduler.capacity.maximum-applications": "10000",
"yarn.scheduler.capacity.node-locality-delay": "40",
"yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator",
"yarn.scheduler.capacity.queue-mappings-override.enable": "false",
"yarn.scheduler.capacity.root.acl_administer_queue": "*",
"yarn.scheduler.capacity.root.capacity": "100",
"yarn.scheduler.capacity.root.queues": "Hive",
"yarn.scheduler.capacity.root.accessible-node-labels": "*",
"yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.Hive.maximum-capacity": "100",
"yarn.scheduler.capacity.root.Hive.user-limit-factor": "4",
"yarn.scheduler.capacity.root.Hive.state": "RUNNING",
"yarn.scheduler.capacity.root.Hive.capacity": "100"
}
} Help is - /var/lib/ambari-server/resources/scripts/configs.py -h To perform the change you want to make, edit the file /tmp/cs.json with your desired change (update value of yarn.scheduler.capacity.root.Hive.user-limit-factor in your case) then use the "-a set" option with the same file. Sample cmd provided below - /var/lib/ambari-server/resources/scripts/configs.py -a set -l <ambari_server_host> -n <cluster_name>-c capacity-scheduler -f /tmp/cs1.json Note, you need to refresh queues to make this change take effect. You can do it by running rmadmin via command line - yarn rmadmin -refreshQueues Or, use the Ambari REST API - curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' http://<ambari-server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/<view_instance_name>/resources/scheduler/configuration/saveAndRefresh You do not need to restart RM for capacity scheduler changes. However, if you make changes to other configs like yarn-site via configs.py you need to restart RM. You can do so by using Ambari REST APIs as shown below. Stop RM - curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"INSTALLED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER Start RM - curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"STARTED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER
... View more
06-12-2019
08:02 PM
Hi @Shu I am trying to create the directory with current time stamp in hdfs and as above I tried but did not work since it doesnt have extract_date what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?
... View more