Member since
02-18-2016
72
Posts
19
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
504 | 07-10-2017 04:10 PM | |
1149 | 07-10-2017 04:01 PM | |
2980 | 04-25-2017 05:01 PM | |
1722 | 03-02-2017 06:35 PM | |
3683 | 12-20-2016 02:13 PM |
07-10-2017
04:10 PM
2 Kudos
I believe this is related to the setting on your capacity scheduler. If you didn't setup the scheduler, then all resources will be allocated to the first user, and other user have to wait. Please refer the documentation for detail: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_hive-performance-tuning/content/section_create_configure_yarn_capacity_scheduler_queues.html
... View more
07-10-2017
04:01 PM
so far not many mechanisms can be used to do what you asked for. 1. You can check LLAP UI to see which queries is running, and how long it has been running. You can then dig into each node to see how many executors are used to run. 2. You can tail the LLAP log to see the mappers/reducers are used running for queries. --- This is most close one to what you ask for. 3. In the Tez view who are the owner of the query, you can check running status as well, and you can see the swimming line to further understand the cost for each steps within the query. But this will be available after the query is finished. The intention of LLAP is for queries with short turn around time, long running/larger query should not be part of LLAP process.
... View more
05-05-2017
01:58 PM
It looks like your cluster is Kerberizied. To have this UI work, you need to do: 1. Configure your laptop/desktop OS Kerberos client 2. Use command line or tool to kinit a kerberos ticket 3. Configure your browser for SPNEGO, you can search it online for your specific browser. It is also recommended that you use Storm Monitoring View for the purpose: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.0/bk_ambari-views/content/ch_using_storm_view.html
... View more
04-27-2017
02:10 PM
you may try to use the same string to connect through sqlline: cd /usr/hdp/current/phoenix-client/bin/
./psql.py jdbc:phoenix:fr0-datalab-p21.bdata.corp,fr0-datalab-p22.bdata.corp,fr0-datalab-p23.bdata.corp:2181:/hbase-secure:my_user@MY_DOMAIN:/path_to_my_keytab.keytab /usr/hdp/current/phoenix-client/doc/examples/WEB_STAT.sql /usr/hdp/current/phoenix-client/doc/examples/WEB_STAT.csv /usr/hdp/current/phoenix-client/doc/examples/WEB_STAT_QUERIES.sql If it still has issue, then you can check the phoenix log to see the problem. If it is passed, then the issue is with squirrel setting.
... View more
04-25-2017
09:22 PM
If just for embeded use, simply tell where your AMS data located by setting “hbase.rootdir" and "hbase.tmp.dir” directory configurations in Ambari Metrics > Configs > Advanced ams-hbase-site are using a large partition, such as: file:///grid/0/var/lib/ambari-metrics-collector/hbase . If you choose to setup a distributed mode, then you need to set Ambari Metrics > Configs > General "Metrics Service operation mode" to distributed. In the Advanced ams-hbase-site, set up hbase.cluster.distributed to true, and hbase.rootdir to a hdfs location. It really depends on your usage of cluster, the size of AMS may vary greatly. If you have a lot of activities on the cluster, you may choose to give more space to that folder. There is no definite formula to decide how much storage is required for different usage. But here is guideline for disk and memory setting I copied from the http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_reference_guide/content/_ams_general_guidelines.html: Cluster Environment Host Count Disk Space Collector Mode TTL Memory Settings Single-Node Sandbox 1 2GB embedded Reduce TTLs to 7 Days metrics_collector_heap_size=1024 hbase_regionserver_heapsize=512 hbase_master_heapsize=512 hbase_master_xmn_size=128 PoC 1-5 5GB embedded Reduce TTLs to 30 Days metrics_collector_heap_size=1024 hbase_regionserver_heapsize=512 hbase_master_heapsize=512 hbase_master_xmn_size=128 Pre-Production 5-20 20GB embedded Reduce TTLs to 3 Months metrics_collector_heap_size=1024 hbase_regionserver_heapsize=1024 hbase_master_heapsize=512 hbase_master_xmn_size=128 Production 20-50 50GB embedded n.a. metrics_collector_heap_size=1024 hbase_regionserver_heapsize=1024 hbase_master_heapsize=512 hbase_master_xmn_size=128 Production 50-200 100GB embedded n.a. metrics_collector_heap_size=2048 hbase_regionserver_heapsize=2048 hbase_master_heapsize=2048 hbase_master_xmn_size=256 Production 200-400 200GB embedded n.a. metrics_collector_heap_size=2048 hbase_regionserver_heapsize=2048 hbase_master_heapsize=2048 hbase_master_xmn_size=512 Production 400-800 200GB distributed n.a. metrics_collector_heap_size=8192 hbase_regionserver_heapsize=122288 hbase_master_heapsize=1024 hbase_master_xmn_size=1024 regionserver_xmn_size=1024 Production 800+ 500GB distributed n.a. metrics_collector_heap_size=12288 hbase_regionserver_heapsize=16384 hbase_master_heapsize=16384 hbase_master_xmn_size=2048 regionserver_xmn_size=1024
... View more
04-25-2017
06:11 PM
Essentially behind the scene, AMS is using Hbase for storing the information, so it can be embeded(default) and distributed deployed. To launch AMS, you need add AMS service through ambari. Checking on https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-user-guide/content/ch_using_ambari_metrics.html for more details.
... View more
04-25-2017
06:06 PM
since it is already kerberized, then you need to include principle in your jdbc url, also it will create /hbase-secure entry in zookeeper by default. the url should look like: jdbc:phoenix:server1,server2,server3:2181:/hbase-secure:/keytabs/myuser.headless.keytab:somebody@HDP.LOCAL
... View more
04-25-2017
05:05 PM
looks like you need to setup jdbc connection correctly for your mysql from local host.
... View more
04-25-2017
05:01 PM
1 Kudo
I am not sure about your use case. If you want just include file1 into hive table, you have to copy those files into separate folders. The alternative way might be you can including all data into the hive table, and let hive to control what data can be selected/seen etc.
... View more
04-25-2017
04:52 PM
it looks like you still use the public repo. You need to select the local repo by click the radio button in your previous version/repository selection page.
... View more
04-18-2017
09:55 PM
One more thing, have you import the certificate into the keystore on the node where you beeline is running?
... View more
04-18-2017
03:55 AM
I assume you have already done hdfs group mapping. Could you please share your setting on the setting for ranger user sync ?
... View more
04-18-2017
03:43 AM
Could you please share your setting on the ranger, including the hive plugin and knox plugin? In the meantime, please check the ranger audit to see any denied action on this connection.
... View more
03-07-2017
02:58 PM
Could you open a new thread, so that somebody else will chime in if I am not available?
... View more
03-03-2017
06:18 PM
The interfaces are required component. I think when you enter into that page, it will give you sample value, i.e. hftp://sandbox.hortonworks.com:50070, you then need to change the url to you own setting if not on the sandbox. Detail information on these setting, you can find at https://falcon.apache.org/EntitySpecification.html. The properties are optional, you can define your own properties. You can check in the link I listed above as well.
... View more
03-03-2017
02:59 PM
have you seen any other views by setting correct permission? If just only the files view, you may follow the steps below: You need to set up an HDFS proxy user for the Ambari daemon account. For example, if ambari-server daemon is runnng as root , you set up a proxy user for root in core-site by adding and changing properties in HDFS > Configs > Custom core-site: hadoop.proxyuser.root.groups=* hadoop.proxyuser.root.hosts=* Restart the required components as indicated by Ambari.
... View more
03-02-2017
09:22 PM
the views is the group that was pre-defined in a tutorial before this one. So, you need to create a group called views before you can assign the falcon user to this group. In order to add user/group to access views, you can just follow this path: admin->manage ambari->views->click one of the views->click that view link-> go to permission->add user or group that can access to that view.
... View more
03-02-2017
07:02 PM
I think you meant to incremental update on the hive table? As of now, I think that is the way to go. As you may aware HDFS itself is immutable, so everything you written is not changeable. So, the incremental update on the hive table is trying to mimic the update on the DBMS like mysql which it can update on the records. In recent version of Hive, the transaction function capability was added, which will allow the update on the records on the Hive table. But I personally have not test it on the incremental update using this capability.
... View more
03-02-2017
06:54 PM
I am copying the part of tutorial below for your reference. You can see the newly added falcon user. Click on it to assign it a group so that it can access Ambari views.
Write "views" and select it in LocalGroupMembership box and then click on tick mark to add a falcon user in the "views" group.
... View more
03-02-2017
06:35 PM
First, I think you meant HDFS/Falcon replication. Hbase has its own replication method. Regarding you see screen empty, that's the permission issue. I think you missed one step in the tutorial that to setup permission (assign to views group in the tutorial) for the newly created falcon user.
... View more
02-24-2017
06:05 PM
You can kill it from linux command from the server that run Ambari metrics and grafana. But also you may need to check the log why it is not able to shutdown. pkill -KILL -u ams
... View more
02-24-2017
05:20 PM
can you share your command for your consumer? bin/kafka-console-consumer.sh --zookeeper yourZookeeperHost:2181 --topic yourTopicName --from-beginning
... View more
02-23-2017
09:39 PM
from your picture, it looks like you have a zookeeper on cmtoldhwdpadm01.dev.bmocm.com, how about cmtoldhwdpadm02, and cmtoldhwdpadm03? Are there zookeeper running on those two? Now, please follow the steps below: /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server cmtoldhwdpadm01.dev.bmocm.com ls /brokers/ids/ <-- here you should see two ids as there are two broker nodes get /brokers/ids/<id1> and <id2> <--- Please share the output
... View more
02-23-2017
09:26 PM
I think you have extra white spaces/line breaks at the end of file.
... View more
02-23-2017
09:14 PM
not sure what are the url for your zookeeper sever. you should run as /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server cmtoldhwdpadm03.dev.bmocm.com:2181 if cmtoldhwdpadm03.dev.bmocm.com is your zookeeper server.
... View more
02-23-2017
08:55 PM
understand that your zookeeper is running. Want you to run zkCli at the command line, so that we can check whether it can connect from nodes to the zookeeper server.
... View more
02-23-2017
08:40 PM
try to check the zkCli, to see whether you are able to connect to zookeeper. As mentioned above post.
... View more
02-23-2017
08:31 PM
It looks like your brokers are not running correctly, or at least not registered on the zookeeper. Please check the kafka server log, and see what's the reason for that. The server log is located at /var/log/kafka.
... View more
02-10-2017
04:58 AM
you may consider Atlas and Ranger for that functionality. See this tutorial http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/ for reference.
... View more