Resources

Consider increasing network capacity to overcome the challenge caused due to non locality of blocks.Create configuration groups of datanodes exclusively for HBASE, disabling HDFS balancer on this group and allow only hbase balancer. Follow this url Host_Config_Groups to create host config groups.Few temporary workarounds can also be applied if problem is severe and need immediate attention :-Disable HDFS balancer permanently on the cluster and run it manually on need basis. (Please spin a support case and have the situation discussed before implementing this workaround.)In case the performance issue needs to be fixed post running of HDFS Balancer, a major compaction could be manually initiated. For performance gains, major compaction is run on off peak hours such as weekends. This article Compaction_Best_Practices is a recommended read here.Scheduling major compaction after scheduled balancer rather than vice versa.HDFS although has introduced concept of "favored nodes" feature but HBa

PROBLEM Running sqoop import command in direct mode for accessing a Netezza data warehouse appliance hangs at 100% mapsqoop import --options-file sqoop_opts_file.opt...INFO mapreduce.Job: Running job: job_1465914632244_0005INFO mapreduce.Job: Job job_1465914632244_0005 running in uber mode : falseINFO mapreduce.Job: map 0% reduce 0%INFO mapreduce.Job: map 25% reduce 0%INFO mapreduce.Job: map 50% reduce 0%INFO mapreduce.Job: map 100% reduce 0% The sqoop_opts_file.opt had the following options :-connectjdbc:netezza://xxxxxxxxxxxxxxxxxxxxxx:5480/--usernameXXXX --password***************--direct--direct-split-size1000000--compress--tabletable_name--target-dir/user/root/table_name--verbose Yarn logs show the below errorsERROR [Thread-14] org.apache.sqoop.mapreduce.db.netezza.NetezzaJDBCStatementRunner: Unable to execute external table exportorg.netezza.error.NzSQLException: ERROR: found delim ',' in a data field, specify escapeChar '\' option in the external table definition RESOLUTION

PROBLEM :The example workflow submitted by users was failing with the below ClassNotFound exceptionsjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.example.SampleMapper not found ROOT CAUSE : These classes are found in oozie-examples-<version>.jar file, which is expected to be present in the lib folder of the job. RESOLUTION :Add the oozie-examples-<version>.jar file to the lib folder of the job and then submit the job.

PROBLEMWhen we query hbase tables through hive, it always creates a fetch task instead of running a MR task. The parameterhive.fetch.task.conversion.threshold controls whether a fetch task would run or a Map Reduce. If hive.fetch.task.conversion.thresholdis less than the table size, it will use MapReduce Job. The default value of the above parameter is 1GB.Create a 'hbase_hive' external table in hive, make sure the hbase table is more than 1GB. [root@node1 ~]# hadoop fs -du -s -h /apps/hbase/data/data/default/hbase-hive 3.4 G /apps/hbase/data/data/default/hbase-hiveFrom beeline analyze the explain plan, which launches a fetch task instead of Map Reduce job, even when the size of the table is more than 1GB0: jdbc:hive2://node1.hwxblr.com:10000/> explain select * from hbase_hive where key = '111111A111111' ; +----------------------------------------------------------------------------------------------------------+--+ | Explain | +------------------------------------------------------

SYMPTOM: User has latest HDP integrated with kerberos. While starting the datanode user gets the message: Login failure for dn/host1@EXAMPLE.NET from keytab /etc/security/keytabs/dn.service.keytab. But the principal is dn/host1.bc@EXAMPLE.NETWhere host1 is the hostname of the datanode host andEXAMPLE.NET is the REALM name.ERROR: The output of klist command is as below -$klist -kt /etc/security/keytabs/dn.service.keytab Keytab name: FILE:/etc/security/keytabs/dn.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 0 12/21/2016 10:38:13 dn/host1.bc@EXAMPLE.NETIn logs it shows - dn/host1@EXAMPLE.NETWhere as it should show - dn/host1.bc@EXAMPLE.NET ROOT CAUSE: This is issue with entries in /etc/host file.RESOLUTION: User has below entry in /etc/hosts file -<ipaddress> <hostname> <FQDN> <FQDN> Now the order is changed to <ipaddress> <FQDN> <hostname> <FQDN> Which resol

SYMPTOM: there are lots of heap dump file created under the directory controlled by java flag -XX:HeapDumpPath, user complains that HiveServer2 has generated these dump since this flag was setup by him for HiveServer2.looking at the HiveServer2 logs, there was no trace for startup and stop services. even there was no sign of any type of failure in the logs.ROOT CAUSE:At very preliminary analysis of heap dump I examine the process arguments which looks like this hive     12345 1234  9.8 27937992 26020536 ?   Sl   Dec01 19887:12  \_ /etc/alternatives/java_sdk_1.8.0/bin/java -Xmx24576m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhdp.          version=2.3.4.20-3 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.2.-258/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.   path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.-258/h

Question: We added the following settings to our knox topology so that knox may search in ldap for the groups that a user is in, so that we can efficiently control knox permissions with ranger.<!-- changes needed for group sync--> <param> <name>main.ldapGroupContextFactory</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> </param> <param> <name>main.ldapRealm.authorizationEnabled</name> <value>true</value> </param> <param> <name>main.ldapRealm.groupSearchBase</name> <value>OU=Infrastructure,OU=EXAMPLE,DC=test,DC=local</value> </param> <param> <name>main.ldapRealm.groupObjectClass</name> <value>group</value> </param> <param> <name>main.ldapRealm.groupIdAttribute</name> <value>cn</value> </param> Our search base and attribute settings are the same as those we have in our ha

You FSImage can be exported to XML for diagnostics using the followinghadoop oiv -i fsimage -p XML -o output.xml

Ever come across below error while starting HDFS datanode in unsecure cluster ?016-12-22 09:00:41,045 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:02016-12-22 09:00:41,152 INFO datanode.DataNode (DataNode.java:shutdown(1915)) - Shutdown complete.2016-12-22 09:00:41,153 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) - Exception in secureMainjava.net.SocketException: Permission denied at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:475) at io.netty.channel.DefaultChannelPipeline$He

Create the self signed certificate and add it to a keystore file using: keytool -genkey -alias example.com -keyalg RSA -keystore keystore.jks -keysize 20482. List the keystore entries to verify that the certificate was added. Note that a keystore can contain multiple such certificates:keytool -list -keystore keystore.jks3. Export this certificate from keystore.jks to a certificate file:keytool -export -alias example.com -file example.com.crt -keystore keystore.jks4. Add this certificate to the client's truststore to establish trust:keytool -import -trustcacerts -alias example.com -file example.com.crt -keystore truststore.jks5. Verify that the certificate exists in truststore.jks:keytool -list -keystore truststore.jks6. Set hive.server2.thrift.sasl.qop=auth in HS2 configsThen start HiveServer2, login with user->kinit->beeline and try to connect with beeline using:!connect jdbc:hive2://<hs2_hostname>:10001/default;principal=<hive_principal>;transportMode=http;httpPat

SYMPTOM: Created a user in Ranger. User is visible in ranger DB but not visible in Ranger UIERROR: Logged into mysql DB and executed below command -SELECT * FROM ranger.x_user where user_name in ('userA');==> This shows the user exist in x_user table.SELECT * FROM ranger.x_portal_user where user_name in ('userA');==> The user is also present in x_portal_user ROOT CAUSE: Suspected corruption on Ranger DB sometimes.RESOLUTION: Executing below command resolved the issue>INSERT INTO x_portal_user_role VALUES(NULL,'2016-09-09 00:00:00','2016-09-09 00:00:00',1,1,(SELECT id FROM x_portal_user WHERE login_id='XXXX'),'ROLE_USER',1); Replace XXXX with the login_id of the user ('userA') You can replace 'ROLE_USER' with 'ROLE_SYS_ADMIN' if you want it to be an admin

SYMPTOM: HDP upgrade was failed on HDFS startup. Namenode was not able to start and below were log messages - ERROR: From the detailed logs we see below error -ROOT CAUSE: The above log clearly indicates"ClassNotFound" error. Customer has integrated customer jar in hadoop which was causing the issue.RESOLUTION: There was custom jar which was already in place with Previous HDP version [located in path - /usr/hdp/2.4.3.0-37/hadoop/sas*.jar ]. Adding the jar from earlier version to the upgraded version path [ie. /usr/hdp/2.5.3.0-37/hadoop/] resolved the issue.[Note: There was custom implementation of SAS with hadoop for the setup and hence the custom jars were present in path mentioned above ie. /usr/hdp/2.4.3.0-37/hadoop/sas*.jar. Default setup never includes any custom app/jar implementation with hadoop. It usually refers or org.apache.hadoop class.]

SYMPTOM:hive metastore process is going down frequently with following exceptions:2016-12-16 01:30:45,016 ERROR [Thread[Thread-5,5,main]]: thrift.TokenStoreDelegationTokenSecretManager (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover thread received unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:131) at org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:76) at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256) at org.apache.hadoop

SYMPTOM: HiveServer2 is hung and is not able to execute simple query like show tables, during the investigation, we took some jstacks and realize that the following thread is processing very slow."HiveServer2-Handler-Pool: Thread-86129" #86129 prio=5 os_prio=0 tid=0x00007f3ad9e1a800 nid=0x1003b runnable [0x00007f3a73b0a000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$TreeNode.find(HashMap.java:1851) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:1979) at java.util.HashMap.putVal(HashMap.java:637) at java.util.HashMap.put(HashMap.java:611) at org.apache.hadoop.hive.ql.ppd.ExprWalkerProcFactory.ext

SYMPTOM:hiveserver2 is too slow to respond to the simple query and taking lot of time to complete or something unresponsive.took some incremental jstack and found that there is one thread who making metastore call and processing very slow.Thread 24233: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame) - oracle.net.ns.Packet.receive() @bci=157, line=311 (Compiled frame) - oracle.net.ns.DataPacket.receive() @bci=1, line=105 (Compiled frame) - oracle.net.ns.NetInputStream.getNextPacket() @bci=48, line=305 (Compiled frame) - oracle.net.ns.NetInputStream.read(byte[], int, int) @

SYMPTOM We get below error while installing new HDP version packages before upgrading to latest HDP version on SUSE linux.2016-12-21 13:46:47,919 - Package Manager failed to install packages. Error: Execution of '/usr/bin/zypper --quiet install --auto-agree-with-licenses --no-confirm livy_2_3_2_0_2950' returned 104. File 'repomd.xml' from repository 'AMBARI-2.4.1.0.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'AMBARI-2.4.1.0.repo' because of the above error. File 'repomd.xml' from repository 'HDP.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'HDP.repo' because of the above error. No provider of 'livy_2_3_2_0_2950' found. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packages retry_count=ag

SYMPTOM:beeline connection to hiveserver2 is failing with the following exception intermittently."The initCause method cannot be used. To set the cause of this exception, use a constructor with a Throwable[] argument."further looking at the HiveServer2 logs we saw stack trace like thisava.sql.SQLException: Could not retrieve transation read-only status server at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2259) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getProductName(MetaStoreDirectSql.java:171) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.determineDbType(MetaStoreDirectSql.java:152) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:122) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:300) a

SYMPTOM:HiveServer2 remains in hung state, jstack reveals the following trace."HiveServer2-Handler-Pool: Thread-139105" prio=10 tid=0x00007ff34e080800 nid=0x3d43e in Object.wait() [0x00007ff30974e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.ipc.Client.call(Client.java:1417) - locked <0x00000003e1c5f298> (a org.apache.hadoop.ipc.Client$Call) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy23.checkAccess(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.checkAccess(ClientNamenodeProtocolTranslatorPB.java:1469) at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at o

Business NeedI needed to extract links from web pages using JSoup. I originally wrote a microservice for my NiFi MP3 Jukebox. I built a custom NiFi Processor instead. It's a pretty simple process and there's a ton of great articles on how to do it referenced below. Custom NiFi Process Development Process One thing I found useful was having at least one JUnit to test running your processor. Deploying takes a while especially if you need to deploy to a cluster of servers. I found a lot of great NIFI Custom Processor Unit and Integration tests online (see reference area). It's really easy to develop custom processors in Java. In your tests you can input files and get out real files. In your tests you should have a saved copy of what the valid file should be and then you can compare the output. Your JUnits can be triggered from Jenkins or other build tools. NiFi Custom Processors can be developed in your standard development process (TDD, CD, CI, Autodeploy). For the N

- Many times while working on some issues we want to take a quick look at the "/etc/ambari-server/conf/ambari.properties" without actually doing a SSH to the ambari host. - In order to access the Ambari Service using the following URL: (here "erie1.example.com" is ambari hostname)http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER.- We will find that "ambari.properties" values are being returned as part of the JSON response something like following:{ "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER", "RootServiceComponents" : { "component_name" : "AMBARI_SERVER", "component_version" : "2.4.1.0", "server_clock" : 1482322132, "service_name" : "AMBARI", "properties" : { "AAA" : "BBB", "agent.package.install.task.timeout" : "1800", "agent.stack.retry.on_repo_unavailability" : "false", "agent.stack.retry.tries" : "5", "agent.task.timeout" : "900", "agent.threadpool.size.max" : "25

I hope people might find this useful when diagnosing performance related issues on their HDP cluster. I have been a linux admin for over 15 years and still I refer to this almost daily.

End Of Stream means that the client closed the connection (from the point of view of the server). The server then tries to cleanup by closing the socket explicitly, in some cases that results in debug messages you see. EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket Notice that the session id is 0 - so either this is a zk client that failed before establishing a session, or more likely it's a monitoring command (which never establishes a session (working as designed)). There is nothing worry about here and this is normal behaviour.

ENV: HDP-2.5Java : openjdk version "1.8.0_111"the following storm topology consist of a KafkaSpout and a SinkTypeBoltStep 1: Create pom.xml with following dependencies<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>hadoop</groupId> <artifactId>KafkaSpoutStorm</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>stormkafka</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <repositories> <repository> <id>HDPReleases</id> <name>HDP Releases</name> <url>http://repo.hortonw

In Ambari it's possible to supply custom properties for both eclipselink and the database driver through "/etc/ambari-server/conf/ambari.properties". Following are the eclipselink properties that can be configured using a prefix of "server.persistence.properties.". The following eclipselink properties should be defined with the prefix prepended in front of them.server.persistence.properties.eclipselink.jdbc.batch-writing.size=25server.persistence.properties.eclipselink.profiler=QueryMonitor- The "QueryMonitor" is used to measure query executions and cache hits. This can be useful for performance analysis in a complex system. The batch writing, this value is the number of statements to batch (default: 100).With the "QueryMonitor" we can get the following kind of logging inside the "ambari-server.out"org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByHostAndName 0 1336org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByNa

Security and DNS Scanning With Apache NiFiWhen I was working in CyberSecurity, I often needed to run a number of command line Linux tools, especially those from Kali. I recommend you take a look at Kali, it is a nice Linux distribution with all the security tools you would like. Please read up on them first as some are dangerous, destructive or in other ways not tools you want to run on an open network or on other people's machines.My Flow calls a few items and builds up a JSON file to save in Hadoop. I could easily convert this to store in Hive or Phoenix or even send this JSON through Kafka to Storm or Spark for machine learning analysis.Step 1: GetFile Obtain domains to check (could be JMS, files, file, hard code or fed from an application).Step 2: ExecuteStreamCommand Call the NS Lookup script to check the TXT record for a domain. TXT records could contain random text or they could have security information from SPF or DKIM. You could also call dig.Step 3: ExecuteStreamCo