@Ayush Garg Can you just restart the hbase by selecting hbase-->Service Actions---> Restart all you could be having dead process,can you check if you have a PID cat /var/run/hbase/hbase-hbase-master.pid HTH ... View more

@Shobhna Dhami WAL is the write-ahead log for all puts/deletes executes to a table. The WAL ensures Hbase has durable writes if it's not a production environment you can actually delete the file mentioned in the log. This could be an issue with stale WAL splits, try removing WAL's from below hdfs location and restart the HBase master and region services. $ hdfs dfs -rm -skipTrash /apps/hbase/data/WALs/* Then restart the Master and region servers. ... View more

Part 3 of the previous kerberization document ... View more

@Tom Burke Setup the Server: Install Kerberos KDC and Admin Server $ apt update && apt upgrade -y $ apt install krb5-kdc krb5-admin-server krb5-config -y $ krb5_newrealm Locate and edit the krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] default_realm = TEST.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true [realms] TEST.COM = { kdc = server.test.com admin_server = server.test.com } [domain_realm] .test.com = TEST.COM test.com = TEST.COM KDC configuration Locate and edit the kdc.conf /etc/krb5kdc/kdc.conf. [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] TEST.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal } Create the Kerberos database This should pick your REALM for the krb5.conf and kdc.conf you will be prompted for a master password keep it preciously it will be useful for the Ambari Kerberos wizard # /usr/sbin/kdb5_util create -s output Loading random data Initializing database '/var/kerberos/krb5kdc/principal' for realm 'TEST.COM', master key name 'K/M@TEST.COM' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: Re-enter KDC database master key to verify: Locate and edit the kadm5.acl Assign Administrator Privilege by editing the kadm5.acl in /var/kerberos/krb5kdc/kadm5.acl replace the EXAMPLE.COM with your realm */admin@TEST.COM * Restart the KDC and kadmin Set the 2 daemons to auto start at boot else your cluster won't start # /etc/rc.d/init.d/krb5kdc start Starting Kerberos 5 KDC: [ OK ] # /etc/rc.d/init.d/kadmin start Starting Kerberos 5 Admin Server: Create a Kerberos Admin Use the same master password # kadmin.local -q "addprinc admin/admin" Output Authenticating as principal root/admin@TEST.COM with password. WARNING: no policy specified for admin/admin@TEST.COM; defaulting to no policy Enter password for principal "admin/admin@TEST.COM": Re-enter password for principal "admin/admin@TEST.COM": Principal "admin/admin@TEST.COM" created. Check if the root principal was created Go to Ambari and enable Kerberos See attached Kerberos setup for HDP 3.1 they are quite similar save for the new UI ... View more

@Marcel-Jan Krijgsman So frustrating indeed have you tried running the hive import from /usr/hdp/2.6.5.0-292/atlas/hook-bin ? The output should look like below # ./import-hive.sh Using Hive configuration directory [/etc/hive/conf] Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout. Enter username for atlas :- admin Enter password for atlas :- Hive Meta Data imported successfully!!! After running successfully you should be able to see your tables in Atlas ... View more

@Nethaji R Here is an example of loading a CSV file. I generated public data available # Generated sample data There is data that is readiky available from http://www.generatedata.com/ # Sample context of names.txt basically format name and email separated by comma Maxwell,risus@Quisque.com Alden,blandit.Nam.nulla@laciniamattisInteger.ca Ignatius,non.bibendum@Cumsociisnatoque.com Keaton,mollis.vitae.posuere@incursus.co.uk Charles,tempor@idenimCurabitur.net Jared,a@congueelit.net Jonas,Suspendisse.ac@Nulla.ca # Precreate the namespace Invoke hbase shell as the hbase user $ hbase shell This should match the csv file hbase(main):004:0> create 'jina','cf' 0 row(s) in 2.3580 seconds => Hbase::Table - jina # Created a directory in the hbase user home in hdfs $ hdfs dfs -mkdir /user/hbase/test # Copied the name.txt to hdfs $ hdfs dfs -put name.txt /user/hbase/test Evoke the hbase load utility Load the csv to hbase using -Dimporttsv Track the evolution in the YARN UI $ cd /usr/hdp/current/hbase-client/ $ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=","-Dimporttsv.columns=HBASE_ROW_KEY,cf jina /user/hbase/test/name.txt ..... ..... 2019-01-28 10:49:09,708 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x69e1dd28 connecting to ZooKeeper ensemble=nanyuki.dunnya.com:2181 2019-01-28 10:49:09,719 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-292--1, built on 05/11/2018 07:15 GMT 2019-01-28 10:49:09,719 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=nanyuki.dunnya.com 2019-01-28 10:49:09,719 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112 2019-01-28 10:49:09,719 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2019-01-28 10:49:09,719 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre ....... 019-01-28 12:06:14,837 INFO [main] mapreduce.Job: map 0% reduce 0% 2019-01-28 12:06:33,197 INFO [main] mapreduce.Job: map 100% reduce 0% 2019-01-28 12:06:40,926 INFO [main] mapreduce.Job: Job job_1548672281316_0003 completed successfully 2019-01-28 12:06:41,640 INFO [main] mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=186665 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=3727 HDFS: Number of bytes written=0 HDFS: Number of read operations=2 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=31502 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=15751 Total vcore-milliseconds taken by all map tasks=15751 Total megabyte-milliseconds taken by all map tasks=24193536 Map-Reduce Framework Map input records=100 Map output records=100 Input split bytes=118 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=125 CPU time spent (ms)=2590 Physical memory (bytes) snapshot=279126016 Virtual memory (bytes) snapshot=3279044608 Total committed heap usage (bytes)=176160768 ImportTsv Bad Lines=0 File Input Format Counters Bytes Read=3609 File Output Format Counters Bytes Written=0 # Now scan the hbase table hbase(main):005:0> scan 'jina' ROW COLUMN+CELL Alden column=cf:, timestamp=1548673532506, value=imperdiet.non@euarcu.edu Alfonso column=cf:, timestamp=1548673532506, value=sed.leo.Cras@elit.net Amal column=cf:, timestamp=1548673532506, value=scelerisque.scelerisque@nisisem.net Aquila column=cf:, timestamp=1548673532506, value=orci@arcu.com Armando column=cf:, timestamp=1548673532506, value=egestas@vel.ca Avram column=cf:, timestamp=1548673532506, value=Morbi.quis@ornare.edu Basil column=cf:, timestamp=1548673532506, value=ligula.Aenean.euismod@arcuvel.org Brandon column=cf:, timestamp=1548673532506, value=Quisque@malesuada.co.uk Brendan column=cf:, timestamp=1548673532506, value=ut.dolor.dapibus@senectus.net Brock column=cf:, timestamp=1548673532506, value=libero.Donec@vehiculaet.com Burton column=cf:, timestamp=1548673532506, value=In.tincidunt.congue@turpis.org Cade column=cf:, timestamp=1548673532506, value=quis.lectus@Curae.com Cairo column=cf:, timestamp=1548673532506, value=est.ac.facilisis@ligula.net Calvin column=cf:, timestamp=1548673532506, value=ante.Maecenas.mi@magnaSuspendisue.org Castor column=cf:, timestamp=1548673532506, value=orci.Ut.semper@enim.net Cedric column=cf:, timestamp=1548673532506, value=Maecenas.iaculis@bibendum.edu Charles column=cf:, timestamp=1548673532506, value=in@nibh.co.uk Clark column=cf:, timestamp=1548673532506, value=amet.risus@maurisMorbi.co.uk Cyrus column=cf:, timestamp=1548673532506, value=odio@ipsumCurabitur.org Daquan column=cf:, timestamp=1548673532506, value=dolor.sit@nequenonquam.net Deacon column=cf:, timestamp=1548673532506, value=bibendum.sed@egetvenenatis.ca Dieter column=cf:, timestamp=1548673532506, value=ac@interdumfeugiatSed.com Eagan column=cf:, timestamp=1548673532506, value=molestie.Sed.id@pellentesddictum.com Elliott column=cf:, timestamp=1548673532506, value=gravida.sagittis.Duis@miDuisrisus.com Erich column=cf:, timestamp=1548673532506, value=mauris.Suspendisse@Sedid.co.uk Francis column=cf:, timestamp=1548673532506, value=eu.odio.Phasellus@eu.org Garrison column=cf:, timestamp=1548673532506, value=malesuada.vel@nuncullamcorpereu.org Geoffrey column=cf:, timestamp=1548673532506, value=amet@est.com Gray column=cf:, timestamp=1548673532506, value=condimentum@ligulaconsuerrhoncus.org Hamilton column=cf:, timestamp=1548673532506, value=tortor@lacusCrasinterdum.ca Henry column=cf:, timestamp=1548673532506, value=velit.in@augueeutempor.ca Hoyt column=cf:, timestamp=1548673532506, value=tristique.senectus@Inornasagittis.net .......... Sylvester column=cf:, timestamp=1548673532506, value=Morbi.quis@dis.co.uk Tate column=cf:, timestamp=1548673532506, value=purus.ac.tellus@Nullanissiaecenas.com Theodore column=cf:, timestamp=1548673532506, value=Mauris.nulla.Integer@vestibuluris.net Thomas column=cf:, timestamp=1548673532506, value=fringilla.est@adipiscing.org Victor column=cf:, timestamp=1548673532506, value=eleifend.vitae.erat@velarcbitur.co.uk Wayne column=cf:, timestamp=1548673532506, value=sed.turpis.nec@vel.ca Zane column=cf:, timestamp=1548673532506, value=vel.pede@Integertinciduntaliquam.net Zeus column=cf:, timestamp=1548673532506, value=ac.risus.Morbi@Duisvolutpat.ca 89 row(s) in 0.5300 seconds Voila you csv file is now in hbase !! ... View more

@Michael Bronson If you have exhausted all other avenues YES, Step 1 Check and compare the /usr/hdp/current/kafka-broker symlinks Step 2 Download both env'es as backup from the problematic and functioning cluster Upload the functioning cluster env to the problematic one, since you have a backup Start kafka through ambari Step 3 sed -i 's/verify=platform_default/verify=disable/'/etc/python/cert-verification.cfg Step 4 Lastly, if the above steps don't remedy the issue, then remove and -re-install the ambari-agent and remember to manually point to the correct ambari server in the ambari-agent.ini ... View more

@Michael Bronson If you can start your brokers from the CLI then that means your env is not set properly as Ambari depends on that env to successfully start or stop a component. What you could do is export the env from the problematic cluster and compare it meticulously against the env from the working cluster using the procedures I sent above. You should be able to see the difference Can you also validate that the symlinks are okay ... View more

@Bhushan Kandalkar Good it worked out but you shouldn't have omitted the information about the architecture ie Load balancer such info is critical in the analysis ....:-) Happy hadooping ... View more