Member since
04-17-2018
14
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
21280 | 07-25-2018 04:43 PM | |
538 | 04-30-2018 02:27 AM |
07-25-2018
04:43 PM
1 Kudo
Your response is a bit overcomplicated. I resolved my own question. Below is a type of a query from HIVE Metastore I was looking for: SELECT DBS.NAME AS OWNER, TBLS.TBL_NAME as OBJECT_NAME, TBL_COMMENTS.TBL_COMMENT as OBJECT_DESCRIPTION, TBLS.TBL_ID as OBJECT_ID, TBLS.TBL_TYPE as OBJECT_TYPE, 'VALID' as OBJECT_STATUS, COLUMNS_V2.COLUMN_NAME, COLUMNS_V2.COMMENT as COLUMN_DESCRIPTION, COLUMNS_V2.TYPE_NAME AS DATA_TYPE COLUMNS_V2.* FROM DBS JOIN TBLS ON DBS.DB_ID = TBLS.DB_ID JOIN SDS ON TBLS.SD_ID = SDS.SD_ID JOIN COLUMNS_V2 ON COLUMNS_V2.CD_ID = SDS.CD_ID JOIN (SELECT DISTINCT TBL_ID, TBL_COMMENT FROM (SELECT TBLS.TBL_ID TBL_ID, TABLE_PARAMS.PARAM_KEY, TABLE_PARAMS.PARAM_VALUE, CASE WHEN TABLE_PARAMS.PARAM_KEY = 'comment' THEN TABLE_PARAMS.PARAM_VALUE ELSE '' END TBL_COMMENT FROM TBLS JOIN TABLE_PARAMS ON TBLS.TBL_ID = TABLE_PARAMS.TBL_ID) TBL_COMMENTS_INTERNAL) TBL_COMMENTS ON TBLS.TBL_ID = TBL_COMMENTS.TBL_ID;
... View more
07-25-2018
02:51 PM
@dsun thanks for the response. Are the table/column comments stored somewhere in HIVE Metastore? I already know about the describe command and Atlas. I need to extract the table/column comments into a table/file, not simply view them on screen. I hope this makes sense. Does anyone else know how to query table/column comments using HIVE Metastore? Thanks.
... View more
07-24-2018
09:33 PM
I am trying to figure out how to query table and column comments (descriptions) in HIVE Metastore. If there is a way to add the comments to HIVE tables/columns, shouldn't be a way to query/report on the comments from HIVE Metastore? Please help 🙂 Best Regards, Michael
... View more
Labels:
- Labels:
-
Apache Hive
04-30-2018
03:26 AM
@Jay Kumar SenSharma I've already done this: sudo systemctl disable firewalld sudo service firewalld stop Redirecting to /bin/systemctl stopfirewalld.service Using xx.xxx.xxx.xxx:8080 comes back with the same error. I can ping xx.xxx.xxx.xxx or FQDN with no problems.
... View more
04-30-2018
03:16 AM
@Jay Kumar SenSharma I am using elastic IP, accessible from the Internet. Security group assigned to instance currently allows all traffic on all ports, 0.0.0.0/0 ::/0 in and out
... View more
04-30-2018
03:05 AM
Background: AWS EC2 instance. OS: CentOS Linux 7 x86_64 HVM EBS 1704_01 (ami-0cfcdb69) Successfully installed and setup Ambari server. Followed all the steps in the guide. Ambari server is up and running: [centos@ec2-xx-xxx-xxx-xxx ~]$ sudo ambari-server status Using python/usr/bin/python Ambari-server status Ambari Server running Found Ambari Server PID: 2179 at: /var/run/ambari-server/ambari-server.pid Problem: Can't connect to Ambari Web UI to continue the configuration. Here is the error in Chrome browser: This site can’t be reached ec2-xx-xxx-xxx-xxx.us-east-2.compute.amazonaws.com refused to connect. ERR_CONNECTION_REFUSED Did not have this problem with Ubuntu ... I can ping the server successfully What could be wrong?
... View more
Labels:
04-30-2018
02:27 AM
@Geoffrey Shelton Okot found the solution. Firewalld was not installed on the image I am using. After running the command from this link, everything worked. It was not the permissions issue, firewalld was not on the image. Centos has an unclear way of explaining this. https://docs-old.fedoraproject.org/en-US/Fedora/19/html/Security_Guide/sec-Check_if_firewalld_is_installed.html Thanks for taking a look at this. Best
... View more
04-28-2018
07:12 PM
Hi Everyone, I am installing HDP2.6.0.0 onto a 3 node AWS cluster running CentOS Linux 7 x86_64 HVM EBS 1704_01. Got stuck on Configuring iptables step in the manual https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-installation/content/configuring_iptables.html Here is the error: $ sudo systemctl disable firewalld Failed to execute operation: Access denied Also tried this (same w/out sudo): $ systemctl disable firewalld ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-unit-files === Authentication is required to manage system service or unit files. Authenticating as: Cloud User (centos) Password: ==== AUTHENTICATION COMPLETE === Failed to execute operation: Access denied Any idea why is it access denied? What is the right way to do this step?
... View more
Labels:
04-20-2018
03:37 AM
Changed ambari-agent.ini on all the nodes, replaced the private IP of the master node with elastic IP of the master node. Stopped and started the ambari server. This did not help, same issue continues. Issue is not resolved. Any other checks/trouble-shooting options?
... View more
04-19-2018
12:23 AM
@Geoffrey Shelton Okot I've installed everything (assuming agents as well) using ambari. I have not done any manual setup, to my knowledge. Below are the contents the /etc/hosts and /etc/ambari-agent/conf/ambari-agent.ini Contents of /etc/hosts (I've masked last two numbers here with x for security): 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 18.217.xxx.xx ec2-18-217-xxx-xx.us-east-2.compute.amazonaws.com 18.218.xxx.xx ec2-18-218-xxx-xx.us-east-2.compute.amazonaws.com 52.15.xxx.xx ec2-52-15-xxx-xx.us-east-2.compute.amazonaws.com ======================================== Contents of /etc/ambari-agent/conf/ambari-agent.ini (host name below is set to the aws private IP that the 1st elastic IP in /etc/hosts above is pointing to, 18.217.xxx.xx, I have not set it anywhere myself, so this must have been determined by ambari setup process automatically. What do you recommend? Should I manually change it to the elastic IP from /etc/hosts, on each node? Any other files to check?): [server] hostname=ip-172-31-10-118.us-east-2.compute.internal url_port=8440 secured_url_port=8441 connect_retry_delay=10 max_reconnect_retry_delay=30 [agent] logdir=/var/log/ambari-agent piddir=/var/run/ambari-agent prefix=/var/lib/ambari-agent/data ;loglevel=(DEBUG/INFO) loglevel=INFO data_cleanup_interval=86400 data_cleanup_max_age=2592000 data_cleanup_max_size_MB =100 ping_port=8670 cache_dir=/var/lib/ambari-agent/cache tolerate_download_failures=true run_as_user=root parallel_execution=0 alert_grace_period=5 status_command_timeout=5 alert_kinit_timeout=14400000 system_resource_overrides=/etc/resource_overrides ; memory_threshold_soft_mb=400 ; memory_threshold_hard_mb=1000 ; ignore_mount_points=/mnt/custom1,/mnt/custom2 [security] keysdir=/var/lib/ambari-agent/keys server_crt=ca.crt passphrase_env_var_name=AMBARI_PASSPHRASE ssl_verify_cert=0 credential_lib_dir=/var/lib/ambari-agent/cred/lib credential_conf_dir=/var/lib/ambari-agent/cred/conf credential_shell_cmd=org.apache.hadoop.security.alias.CredentialShell [network] ; this option apply only for Agent communication use_system_proxy_settings=true [services] pidLookupPath=/var/run/ [heartbeat] state_interval_seconds=60 dirs=/etc/hadoop,/etc/hadoop/conf,/etc/hbase,/etc/hcatalog,/etc/hive,/etc/oozie, /etc/sqoop, /var/run/hadoop,/var/run/zookeeper,/var/run/hbase,/var/run/templeton,/var/run/oozie, /var/log/hadoop,/var/log/zookeeper,/var/log/hbase,/var/run/templeton,/var/log/hive ; 0 - unlimited log_lines_count=300 idle_interval_min=1 idle_interval_max=10 [logging] syslog_enabled=0
... View more
04-18-2018
04:59 PM
@Geoffrey Shelton Okot thanks for responding and giving the pointers. I did configure and test the passwordless connect between the hosts in the cluster. I can ssh to any host, from any host on the cluster without entering the password. I did folllow the procedure from the installation guide for setting up /etc/hosts, but I will post the contents of /etc/hosts later today.
... View more
04-18-2018
03:26 AM
@Geoffrey Shelton Okot Reviewed the link. Thank you for taking a look at this. Best answer in the link just provides a link to general AWS documentation on Elastic IP and VPC DNS. I did not find reasons nor solutions for the issue I detailed here. I've assigned elastic IPs to my cluster prior to the installation of the Ambari. Installation completed with no issues. Services are not starting due to failed connections, some of which are detailed in my original post. Same elastic IPs have been assigned to the cluster since its launch. Is there a way to trouble-shoot these connections? Why the setup did not configure these connections automatically? Is there a way/steps to verify that everything is properly configured on the cluster for the above connections to work?
... View more
04-17-2018
04:54 PM
@Geoffrey Shelton Okot Thanks, taking a look.
... View more
04-17-2018
06:57 AM
Background: first time installing hadoop on a cluster single-handedly. AWS 3 instances, Ubuntu 16, 1 XL node 2 L nodes, all with 40 Gb storage each. Latest version of HDP 2.6.4. Installed successfully, started ambari successfully, logged into web admin portal ... Followed all the steps in the documentation. Problem: most services failed to start ("heartbeat lost"), 39 alerts. Tried starting services manually via the web portal, tried restarting the ambari server, did not help. Below are some of the alerts: HDFS NameNode Web UI: Connection failed to http://ec2-18-217-xxxx.us-east-2.compute.amazonaws.com:50070 (<urlopen error [Errno 111] Connection refused>) Yarn App Timeline Web UI: Connection failed to http://ec2-18-218-xxxx.us-east-2.compute.amazonaws.com:8188/ws/v1/timeline (<urlopen error [Errno 111] Connection refused>)
CRIT MapReduce2 History Server Process Connection failed: [Errno 111] Connection refused to ec2-18-218-xxxx.us-east-2.compute.amazonaws.com:19888 ... and many more
... View more
Labels: