Member since
09-29-2015
58
Posts
34
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
388 | 01-12-2017 02:49 PM | |
460 | 09-29-2016 01:39 PM | |
1107 | 06-21-2016 03:10 PM | |
3702 | 05-16-2016 07:12 PM | |
5843 | 04-08-2016 02:06 PM |
01-12-2017
02:49 PM
4 Kudos
Yes you can, here is an example
... View more
01-12-2017
02:49 PM
2 Kudos
Are you able to answer and accept your own answer to a question you post?
... View more
- Tags:
- hcc
01-12-2017
02:40 PM
yarn-client is not supported as per new guidelines, only yarn-cluster mode is supported within livy interpreter from Zeppelin
... View more
01-12-2017
02:39 PM
1 Kudo
yarn-client is not supported, only yarn-cluster is supported
... View more
01-11-2017
02:33 PM
@elkan li which user are you logged in as when launching? Can you try an su hive then launch the hive cli? It seems the issue is with the launching user
... View more
01-11-2017
02:30 PM
1 Kudo
@sagar pavan the Diagnostic message indicates the Users AM resource limit is exceeded. Please review the capacity scheduler's AM resource limit and raise it from the default 20%, this should allow the AM container to be launched
... View more
12-21-2016
03:24 PM
3 Kudos
livy documentation states to add jars with the interpreter key value livy.spark.jars. This is applicable when running livy interpreter in yarn-cluster mode only (livy.spark.master). This can be done in local or yarn-client mode as well however by adding the jars to the /usr/hdp/<version>/livy/repl-jars directory. This will add the jars to the spark submit command run by livy when executing within a notebook in modes other than yarn-cluster.
... View more
- Find more articles tagged with:
- Data Science & Advanced Analytics
- FAQ
- issue-resolution
- livy
- Spark
- zeppelin
- zeppelin-notebook
Labels:
12-13-2016
05:27 PM
This does not look like an issue with the jar being included but rather an issue with the import statement. I breifly looked on google and see similar descriptions stating to try org.mongodb. I would focus on the import statement more than the inclusion of the jar for livy.
... View more
12-13-2016
04:25 PM
@Mickaël GERVAIS check to make sure livy interpreter is listed in the interpreter bindings for the notebook. Also, set DEBUG on the livy server and check in the livy out file produced on the server. Finally, make sure you have restarted livy and zeppelin to pick up the changes. I tested and it did work for me.
... View more
12-13-2016
02:22 PM
The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server: /usr/hdp/2.5.3.0-37/livy/repl-jars
... View more
09-29-2016
01:39 PM
@ARUN Yes, you can use node labels and queues together. Here is some documentation regarding that: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_yarn_resource_mgt/content/configuring_node_labels.html
... View more
09-27-2016
11:53 AM
2 Kudos
That is not a good solution as you are saying yarn does not need any healthy disks to function. You essentially disable the health check with this. There is an underlying problem with the disks and that is the reason for it being marked unhealthy.
... View more
09-27-2016
11:49 AM
1 Kudo
@Mourad Chahri You can go to the ResourceManager UI. From there you should see a nodes link on the left side of the screen. If you click on that, you should see all of your NodeManagers and the reason for it being listed as unhealthy may be shown here. It is most likely due to yarn local dirs or log dirs. You may be hitting the disk threshold for this. There are a couple of parameters you can check for this. yarn.nodemanager.disk-health-checker.min-healthy-disks yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb Finally, if that does not reveal the issue, you should look in /var/log/hadoop-yarn/yarn. Your previous comment shows you were looking in /var/log/hadoop/yarn which is not where the NodeManager log is located. I hope this helps.
... View more
09-26-2016
06:26 PM
@Ahmed ELJAMI this looks like just an INFO message. I would look at the logs for each attempt to see why it is failing. You should be able to see this in the RM UI.
... View more
09-23-2016
02:40 PM
@rama These directories are used by yarn for job logs. There are similar directories used for localization, the yarn-local dirs. They are not distributed so much as used when containers are allocated on that node. They should get cleaned up when jobs complete but can leave orphaned files in the event of a Resource Manager restart or a Node Manager restart. The directories are configured via yarn and it is a comma separated list of locations so you can add additional mounts/directories but they will apply to all node managers managed by Yarn. Hope this helps.
... View more
09-12-2016
03:14 PM
oops, your input is slashes, not dashes. Can you try with the needed format?
... View more
09-12-2016
02:42 PM
@Mayank Pandey what does the following produce? select log_source_time, from_unixtime(unix_timestamp(substr(log_source_time,0,11),'dd-MMM-yyyy')) as todateformat from table1 limit 2;
... View more
09-12-2016
01:15 PM
2 Kudos
@Mayank Pandey there are some ways of converting the date. For example something like select inp_dt, from_unixtime(unix_timestamp(substr(inp_dt,0,11),'dd-MMM-yyyy')) as todateformat from table; There are several ways to attempt this if you do a google search on your needs.
... View more
09-12-2016
01:01 PM
@mike harding to add to this, Tez by default first initializes an AM whereas MapReduce does so at submission only. This is the reason you see the behavior you describe. The tez container has a timeout setting as you stated and that will determine how long lived that initial AM is
... View more
06-29-2016
11:46 AM
@nejm hadjmbarek, in the information you provided, it seems your oozie max concurrency has been reached for the coordinator. You therefore have a number of applications waiting AM resources. Check you max AM resource percentage in capacity scheduler and consider raising it to either .5 or .6 which states that of the total resources, RM can assign our 50 or 60 percent to AM containers.
... View more
06-21-2016
03:10 PM
1 Kudo
@Arthur GREVIN it appears you only have one nodemanager deployed on those nodes. It is allocating 7 GB from that single node and that is why it is showing only that. You would need to deploy nodemanagers on the other 3 existing nodes.
... View more
05-16-2016
07:12 PM
2 Kudos
@Tim Veil It is possible to use the Ambari REST API to change that config. Below is an example: curl -v -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT http://<AMBARI-SERVER>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration --data '{
"Clusters": {
"desired_config": [
{
"type": "capacity-scheduler",
"tag": "version14534007568115",
"service_config_version_note": "To test",
"properties": {
"yarn.scheduler.capacity.maximum-am-resource-percent": 0.2,
"yarn.scheduler.capacity.maximum-applications": 10000,
"yarn.scheduler.capacity.node-locality-delay": 40,
"yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
"yarn.scheduler.capacity.queue-mappings-override.enable": false,
"yarn.scheduler.capacity.root.acl_administer_queue": "*",
"yarn.scheduler.capacity.root.capacity": 100,
"yarn.scheduler.capacity.root.queues": "default",
"yarn.scheduler.capacity.root.accessible-node-labels": "*",
"yarn.scheduler.capacity.root.default.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.default.maximum-capacity": 100,
"yarn.scheduler.capacity.root.default.user-limit-factor": 0.5,
"yarn.scheduler.capacity.root.default.state": "RUNNING",
"yarn.scheduler.capacity.root.default.capacity": 100
}
}
]
}
}'
... View more
05-16-2016
07:01 PM
@Tim Veil I believe the REST API for the scheduler only supports GET operations as per https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
... View more
05-03-2016
01:01 PM
Hi @Vinay MP , It seems you are not able to contact the ResourceManager. What port is RM listening on? You should be able to do a ps -ef | grep resourcemanager and then do a netstat -tulpn | grep <PID> to find out.
... View more
04-13-2016
12:07 PM
@Ryan Cicak The sandbox provides many of the defaults used during normal installation. You can change the 3x replication in the configs but the sandbox is mainly to allow usage of the tutorials.
... View more
04-08-2016
06:42 PM
@Arvind Kandaswamy it is complaining about the ping port listener. This is not port 8080 but is defined in your /etc/ambari-agent/conf/ambari-agent.ini. It is normally port 8670. Try running netstat -tulpn | grep 8670 to see what is on that port.
... View more
04-08-2016
03:05 PM
1 Kudo
@Saurabh Kumar To add to this, you could investigate third party dev projects such as https://github.com/nmilford/clean-hadoop-tmp
... View more
04-08-2016
02:06 PM
2 Kudos
Hi @Alena Melnikova , there is a capacity and max capacity. The max is what determines elasticity but setting that alone is not enough. That is at queue level. You have to adjust the user-limit-factor so that a user can leverage more than just the capacity. You essentially are saying that a user can use X times capacity. For example, if you say 2 for user-limit-capacity in this example, the user will be able to leverage 80 percent (40x2) Capacity is the queue ceiling, you cannot set to 100 as the queues have to add up to 100 across root. I hope this helps. Ian
... View more
04-08-2016
01:56 PM
Please try to execute the command manually on the node and see what it returns: conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0 This also looks specific to the environment: sudo:no tty present andno askpass program specified. You could try manually starting on the node which may give more details.
... View more
02-23-2016
03:13 PM
1 Kudo
The release of HDP 2.3.4 offers a newer version of the ATS which is version 1.5. Part of configuring this new ATS version includes providing the following property if Tez is used: <property>
<name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
<value>org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl</value>
<description>Plugins that can translate a timeline entity read request into a list of timeline cache ids, separated by commas. </description> </property> If this property is included but the tez-client is not present, a message will be seen in the ATS log stating ClassNotFound and the ATS server will fail shortly after issuing a startup. The full documentation for configuring ATS 1.5 is located here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_yarn_resource_mgt/content/ch08s02.html A documentation bug has been filed as the only reference to this in the configuration section is this: "The Tez cache plugin class must be in the timeline server’s classpath."
... View more
- Find more articles tagged with:
- ats
- Hadoop Core
- Issue Resolution
- tez
- YARN
Labels: