Additional information that a new partition *could* be added per day. If a day was missed how would it be detected? You could write some very esoteric SQL: do some date math, create a table, add the values for {start-date} to {end-date}, Write a UDF that created an array for that start-date to end date Use Explode with Lateral View to expand that into a table of values Join this table to SHOW PARTITIONS table_name; -- never tried this but it should work, or like I said before joining on "select distinct(partition_column) from table" should work. But even if you had a SQL script that did this... so what... it won't alert you that something is wrong. It's sql, it doesn't have scripting functions like alerts. You'd still be manually running it....so why not write a script that does alert you? This really is a programming issue not a SQL issue. I think writing a script is likely the best idea. You already probably have a script that runs daily to insert this data... why not add a condition that if there isn't data to add, that it writes to a "No data for this day table"? Just a thought. Hope this helps. ... View more

I reread, you response... you can compare SHOW PARTITIONS table_name; with select distinct(partion_column) from table_name; That should work out... but I'm not really sure how you inserted the data in the first place without specifying the partition. ... View more

@Saikrishna Tarapareddy Got it. You want to use the following hive command: MSCK REPAIR TABLE table_name; from the documentation: ...which will add metadata about partitions to the Hive metastore for partitions for which such metadata doesn't already exist. In other words, it will add any partitions that exist on HDFS but not in metastore to the metastore. If you are hardcore and don't like that command you could figure out the existing paritions with: SHOW PARTITIONS table_name; and then explore the data to find what isn't added and add the partitions manually: ALTER TABLE table_name ADD PARTITION partition_spec ... View more

@Saikrishna Tarapareddy You are correct Hive Partitions are literally directories under the table name in the hive warehouse. (HDFS) You can simply inspect HDFS for that table and you will see the partition folders, and see if there is data inside them. @rtrivedi pointed you to correct command to see if there are missing partitions that have not been added to the Hive Metastore. See msck for more detail. If you feel this answer was helpful please mark it as the best answer. If you feel @rtrivedi was more helpful please mark it as the best answer. ... View more

Worked for me once I installed the correct version of epel If that doesn't work for you and you get: (check that epel install centos7 and not centos6) Error: Package: R-core-3.4.1-1.el6.x86_64 (epel) Requires: libicui18n.so.42()(64bit) Error: Package: R-core-3.4.1-1.el6.x86_64 (epel) Requires: libicuuc.so.42()(64bit) Error: Package: qpdf-libs-5.1.1-5.el6.x86_64 (epel) Requires: libpcre.so.0()(64bit) FYI to fix this you can just edit the repo file to point to 7 instead of 6 (and fix the gpg check) ... View more

nm I eventually got it: cluster create --version 2.6 --description "Haoop Pilot" --password D@T@L@K3_2017 --enableKnoxGateway true --stackRepoId HDP-2.6 --stackBaseURL "http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3" --stack "HDP" --utilsBaseURL "http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7" --utilsRepoId "HDP-UTILS-1.1.0.21" --os RedHat --ambariRepoBaseURL http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3 --ambariVersion 2.5.0.3-7 --ambariRepoGpgKey http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins --verify true Cluster creation started So I guess just a better error mesasge would be helpful saying that "--version" required. ... View more

I did with no real help... but I found what I think it's not happy about: 223501 [Spring Shell] DEBUG org.apache.http.wire - http-outgoing-3 << "me":"HDFS_CLIENT"},{"name":"SQOOP"},{"name":"METRICS_MONITOR"},{"name":"INFRA_SOLR_CLIENT"},{"name":"HBASE_MASTER"},{"name":"TEZ_CLIENT"},{"name":"ATLAS_SERVER"},{"name":"ZOOKEEPER_CLIENT"},{"name":"HCAT"},{"name":"PIG"},{"name":"KAFKA_BROKER"},{"name":"MAPREDUCE2_CLIENT"},{"name":"ATLAS_CLIENT"},{"name":"OOZIE_CLIENT"},{"name":"HBASE_CLIENT"},{"name":"INFRA_SOLR"},{"name":"HIVE_CLIENT"}],"name":"host_group_5","cardinality":"1"}],"Blueprints":{"blueprint_name":"edc-ranger-template","stack_name":"HDP","stack_version":"2.6"}},"description":"","inputs":[],"id":2036,"blueprintName":"edc-ranger-template","hostGroupCount":6,"status":"USER_MANAGED","public":false}[\r][\n]" 223532 [Spring Shell] DEBUG org.apache.http.wire - http-outgoing-3 >> "{"name":"hadoop-pilot-ranger-test3-rg","blueprintId":2036,"blueprint":null,"description":"Haoop Pilot","hostGroups":[{"name":"host_group_3","constraint":{"instanceGroupName":"host_group_3","constraintTemplateName":null,"hostCount":1},"recipeIds":[],"recoveryMode":"MANUAL","recipes":null},{"name":"host_group_6","constraint":{"instanceGroupName":"host_group_6","constraintTemplateName":null,"hostCount":1},"recipeIds":[5],"recoveryMode":"MANUAL","recipes":null},{"name":"host_group_2","constraint":{"instanceGroupName":"host_group_2","constraintTemplateName":null,"hostCount":1},"recipeIds":[],"recoveryMode":"MANUAL","recipes":null},{"name":"host_group_1","constraint":{"instanceGroupName":"host_group_1","constraintTemplateName":null,"hostCount":1},"recipeIds":[],"recoveryMode":"MANUAL","recipes":null},{"name":"host_group_4","constraint":{"instanceGroupName":"host_group_4","constraintTemplateName":null,"hostCount":4},"recipeIds":[],"recoveryMode":"MANUAL","recipes":null},{"name":"host_group_5","constraint":{"instanceGroupName":"host_group_5","constraintTemplateName":null,"hostCount":1},"recipeIds":[],"recoveryMode":"MANUAL","recipes":null}],"emailNeeded":false,"emailTo":null,"gateway":{"enableGateway":true,"path":null,"topologyName":null,"exposedServices":["ALL"],"ssoProvider":null,"signCert":null,"signPub":null,"gatewayType":null,"ssoType":null},"enableSecurity":false,"userName":"admin","password":"D@T@L@K3_2017","kerberos":{"masterKey":null,"admin":null,"password":null,"url":null,"realm":null,"principal":null,"ldapUrl":null,"containerDn":null,"tcpAllowed":false},"ldapRequired":false,"sssdConfigId":null,"sssdConfig":null,"ldapConfigId":null,"ldapConfig":null,"validateBlueprint":false,"ambariStackDetails":{"stack":"HDP","version":null,"os":"RedHat","stackRepoId":"HDP-2.6","stackBaseURL":"http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3","utilsRepoId":"HDP-UTILS-1.1.0.21","utilsBaseURL":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7","verify":true},"ambariRepoDetailsJson":{"version":"2.5.0.3-7","baseUrl":"http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3","gpgKeyUrl":"http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins"},"rdsConfigIds":[],"ambariDatabaseDetails":null,"rdsConfigJsons":[],"fileSystem":null,"configStrategy":"ALWAYS_APPLY_DONT_OVERRIDE_CUSTOM_VALUES","enableShipyard":false,"blueprintInputs":[],"blueprintCustomProperties":null,"customContainer":null,"customQueue":"default","connectedCluster":null}[\r][\n]" 223568 [Spring Shell] DEBUG org.apache.http.wire - http-outgoing-3 << "{"validationErrors":{"post.arg1.ambariStackDetails.version":"may not be null"}}" If you read carefully I think it's complaining about "ambariStackDetails":{"stack":"HDP","version":null," so how do I set that? ... View more