Community Articles

You FSImage can be exported to XML for diagnostics using the followinghadoop oiv -i fsimage -p XML -o output.xml

Ever come across below error while starting HDFS datanode in unsecure cluster ?016-12-22 09:00:41,045 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:02016-12-22 09:00:41,152 INFO datanode.DataNode (DataNode.java:shutdown(1915)) - Shutdown complete.2016-12-22 09:00:41,153 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) - Exception in secureMainjava.net.SocketException: Permission denied at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:475) at io.netty.channel.DefaultChannelPipeline$He

Create the self signed certificate and add it to a keystore file using: keytool -genkey -alias example.com -keyalg RSA -keystore keystore.jks -keysize 20482. List the keystore entries to verify that the certificate was added. Note that a keystore can contain multiple such certificates:keytool -list -keystore keystore.jks3. Export this certificate from keystore.jks to a certificate file:keytool -export -alias example.com -file example.com.crt -keystore keystore.jks4. Add this certificate to the client's truststore to establish trust:keytool -import -trustcacerts -alias example.com -file example.com.crt -keystore truststore.jks5. Verify that the certificate exists in truststore.jks:keytool -list -keystore truststore.jks6. Set hive.server2.thrift.sasl.qop=auth in HS2 configsThen start HiveServer2, login with user->kinit->beeline and try to connect with beeline using:!connect jdbc:hive2://<hs2_hostname>:10001/default;principal=<hive_principal>;transportMode=http;httpPat

SYMPTOM: Created a user in Ranger. User is visible in ranger DB but not visible in Ranger UIERROR: Logged into mysql DB and executed below command -SELECT * FROM ranger.x_user where user_name in ('userA');==> This shows the user exist in x_user table.SELECT * FROM ranger.x_portal_user where user_name in ('userA');==> The user is also present in x_portal_user ROOT CAUSE: Suspected corruption on Ranger DB sometimes.RESOLUTION: Executing below command resolved the issue>INSERT INTO x_portal_user_role VALUES(NULL,'2016-09-09 00:00:00','2016-09-09 00:00:00',1,1,(SELECT id FROM x_portal_user WHERE login_id='XXXX'),'ROLE_USER',1); Replace XXXX with the login_id of the user ('userA') You can replace 'ROLE_USER' with 'ROLE_SYS_ADMIN' if you want it to be an admin

SYMPTOM: HDP upgrade was failed on HDFS startup. Namenode was not able to start and below were log messages - ERROR: From the detailed logs we see below error -ROOT CAUSE: The above log clearly indicates"ClassNotFound" error. Customer has integrated customer jar in hadoop which was causing the issue.RESOLUTION: There was custom jar which was already in place with Previous HDP version [located in path - /usr/hdp/2.4.3.0-37/hadoop/sas*.jar ]. Adding the jar from earlier version to the upgraded version path [ie. /usr/hdp/2.5.3.0-37/hadoop/] resolved the issue.[Note: There was custom implementation of SAS with hadoop for the setup and hence the custom jars were present in path mentioned above ie. /usr/hdp/2.4.3.0-37/hadoop/sas*.jar. Default setup never includes any custom app/jar implementation with hadoop. It usually refers or org.apache.hadoop class.]

SYMPTOM:hive metastore process is going down frequently with following exceptions:2016-12-16 01:30:45,016 ERROR [Thread[Thread-5,5,main]]: thrift.TokenStoreDelegationTokenSecretManager (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover thread received unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:131) at org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:76) at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256) at org.apache.hadoop

SYMPTOM: HiveServer2 is hung and is not able to execute simple query like show tables, during the investigation, we took some jstacks and realize that the following thread is processing very slow."HiveServer2-Handler-Pool: Thread-86129" #86129 prio=5 os_prio=0 tid=0x00007f3ad9e1a800 nid=0x1003b runnable [0x00007f3a73b0a000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$TreeNode.find(HashMap.java:1851) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.find(HashMap.java:1861) at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:1979) at java.util.HashMap.putVal(HashMap.java:637) at java.util.HashMap.put(HashMap.java:611) at org.apache.hadoop.hive.ql.ppd.ExprWalkerProcFactory.ext

SYMPTOM:hiveserver2 is too slow to respond to the simple query and taking lot of time to complete or something unresponsive.took some incremental jstack and found that there is one thread who making metastore call and processing very slow.Thread 24233: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise) - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame) - oracle.net.ns.Packet.receive() @bci=157, line=311 (Compiled frame) - oracle.net.ns.DataPacket.receive() @bci=1, line=105 (Compiled frame) - oracle.net.ns.NetInputStream.getNextPacket() @bci=48, line=305 (Compiled frame) - oracle.net.ns.NetInputStream.read(byte[], int, int) @

SYMPTOM We get below error while installing new HDP version packages before upgrading to latest HDP version on SUSE linux.2016-12-21 13:46:47,919 - Package Manager failed to install packages. Error: Execution of '/usr/bin/zypper --quiet install --auto-agree-with-licenses --no-confirm livy_2_3_2_0_2950' returned 104. File 'repomd.xml' from repository 'AMBARI-2.4.1.0.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'AMBARI-2.4.1.0.repo' because of the above error. File 'repomd.xml' from repository 'HDP.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'HDP.repo' because of the above error. No provider of 'livy_2_3_2_0_2950' found. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packages retry_count=ag

SYMPTOM:beeline connection to hiveserver2 is failing with the following exception intermittently."The initCause method cannot be used. To set the cause of this exception, use a constructor with a Throwable[] argument."further looking at the HiveServer2 logs we saw stack trace like thisava.sql.SQLException: Could not retrieve transation read-only status server at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOPersistenceManager.getDataStoreConnection(JDOPersistenceManager.java:2259) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getProductName(MetaStoreDirectSql.java:171) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.determineDbType(MetaStoreDirectSql.java:152) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:122) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:300) a

SYMPTOM:HiveServer2 remains in hung state, jstack reveals the following trace."HiveServer2-Handler-Pool: Thread-139105" prio=10 tid=0x00007ff34e080800 nid=0x3d43e in Object.wait() [0x00007ff30974e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.ipc.Client.call(Client.java:1417) - locked <0x00000003e1c5f298> (a org.apache.hadoop.ipc.Client$Call) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy23.checkAccess(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.checkAccess(ClientNamenodeProtocolTranslatorPB.java:1469) at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at o

Business NeedI needed to extract links from web pages using JSoup. I originally wrote a microservice for my NiFi MP3 Jukebox. I built a custom NiFi Processor instead. It's a pretty simple process and there's a ton of great articles on how to do it referenced below. Custom NiFi Process Development Process One thing I found useful was having at least one JUnit to test running your processor. Deploying takes a while especially if you need to deploy to a cluster of servers. I found a lot of great NIFI Custom Processor Unit and Integration tests online (see reference area). It's really easy to develop custom processors in Java. In your tests you can input files and get out real files. In your tests you should have a saved copy of what the valid file should be and then you can compare the output. Your JUnits can be triggered from Jenkins or other build tools. NiFi Custom Processors can be developed in your standard development process (TDD, CD, CI, Autodeploy). For the N

- Many times while working on some issues we want to take a quick look at the "/etc/ambari-server/conf/ambari.properties" without actually doing a SSH to the ambari host. - In order to access the Ambari Service using the following URL: (here "erie1.example.com" is ambari hostname)http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER.- We will find that "ambari.properties" values are being returned as part of the JSON response something like following:{ "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER", "RootServiceComponents" : { "component_name" : "AMBARI_SERVER", "component_version" : "2.4.1.0", "server_clock" : 1482322132, "service_name" : "AMBARI", "properties" : { "AAA" : "BBB", "agent.package.install.task.timeout" : "1800", "agent.stack.retry.on_repo_unavailability" : "false", "agent.stack.retry.tries" : "5", "agent.task.timeout" : "900", "agent.threadpool.size.max" : "25

I hope people might find this useful when diagnosing performance related issues on their HDP cluster. I have been a linux admin for over 15 years and still I refer to this almost daily.

End Of Stream means that the client closed the connection (from the point of view of the server). The server then tries to cleanup by closing the socket explicitly, in some cases that results in debug messages you see. EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket Notice that the session id is 0 - so either this is a zk client that failed before establishing a session, or more likely it's a monitoring command (which never establishes a session (working as designed)). There is nothing worry about here and this is normal behaviour.

ENV: HDP-2.5Java : openjdk version "1.8.0_111"the following storm topology consist of a KafkaSpout and a SinkTypeBoltStep 1: Create pom.xml with following dependencies<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>hadoop</groupId> <artifactId>KafkaSpoutStorm</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>stormkafka</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <repositories> <repository> <id>HDPReleases</id> <name>HDP Releases</name> <url>http://repo.hortonw

In Ambari it's possible to supply custom properties for both eclipselink and the database driver through "/etc/ambari-server/conf/ambari.properties". Following are the eclipselink properties that can be configured using a prefix of "server.persistence.properties.". The following eclipselink properties should be defined with the prefix prepended in front of them.server.persistence.properties.eclipselink.jdbc.batch-writing.size=25server.persistence.properties.eclipselink.profiler=QueryMonitor- The "QueryMonitor" is used to measure query executions and cache hits. This can be useful for performance analysis in a complex system. The batch writing, this value is the number of statements to batch (default: 100).With the "QueryMonitor" we can get the following kind of logging inside the "ambari-server.out"org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByHostAndName 0 1336org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByNa

Security and DNS Scanning With Apache NiFiWhen I was working in CyberSecurity, I often needed to run a number of command line Linux tools, especially those from Kali. I recommend you take a look at Kali, it is a nice Linux distribution with all the security tools you would like. Please read up on them first as some are dangerous, destructive or in other ways not tools you want to run on an open network or on other people's machines.My Flow calls a few items and builds up a JSON file to save in Hadoop. I could easily convert this to store in Hive or Phoenix or even send this JSON through Kafka to Storm or Spark for machine learning analysis.Step 1: GetFile Obtain domains to check (could be JMS, files, file, hard code or fed from an application).Step 2: ExecuteStreamCommand Call the NS Lookup script to check the TXT record for a domain. TXT records could contain random text or they could have security information from SPF or DKIM. You could also call dig.Step 3: ExecuteStreamCo

This article will go over the concepts of security in an Hbase cluster. More specifically we will concentrate on ACL based security and how to apply at the different levels of granularity on an Hbase model.From an overall security perspective and acces control list , or ACL, is a list of permissions associated with an objectACLs focus on the access rules pattern.ACL logicHbase access contol lists are granted on different levels of data abstractions and cover types of operations.Hbase data layoutBefore we go further let us clear out the hierarchical elements that compose the datastorage HbaseCELL : All values written to Hbase are stored in a what is know as a CELL. (Cell can also be refered to as KeyValue).Cells are identified by a multidimensionnal key {row, column, qualifier, timestamp}. In the example above : CELL => Rowkey1,CF1,Q11,TS1COLUMN FAMILY : A column Family groups together arbitrary cells.TABLE : All Cells belong to a Column family and are organized into a table.NAMESPA

while running the sqoop command from the java program with -verbose option can result into race condition during obtaining lock on the console appender.we can workaround this with the help of SSHXCUTE framework which will create java program and sqoop command context separately.ENV: HDP 2.4Java Version : JDK-8Step 1: download sshxcute jar from https://sourceforge.net/projects/sshxcute/Step 2: Create RunSqoopCommand.javaimport net.neoremind.sshxcute.core.SSHExec;import net.neoremind.sshxcute.core.ConnBean;import net.neoremind.sshxcute.task.CustomTask;import net.neoremind.sshxcute.task.impl.ExecCommand;public class RunSqoopCommand {public static void main(String args[]) throws Exception{ ConnBean cb = new ConnBean("localhost", "root","hadoop"); SSHExec ssh = SSHExec.getInstance(cb); ssh.connect(); CustomTask sqoopCommand = new ExecCommand("sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true -Dmapred.job.na

SYMPTOMRunning java action via Oozie workflow fails with below error: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, Could not find Yarn tags property (mapreduce.job.tags) java.lang.RuntimeException: Could not find Yarn tags property (mapreduce.job.tags) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.getChildYarnJobs(LauncherMainHadoopUtils.java:52) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.killChildYarnJobs(LauncherMainHadoopUtils.java:87) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:44) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36) .ROOT CAUSE Missing or Yarn related jar file conflict in Oozie sharelib..RESOLUTION Complete the following steps as oozie user in the Oozie node:1. Recreate Oozie sharelib using below command/usr/hdp/<hdp-version>/oozie/bin/oozie-setup.sh sharelib create -localli

SYMPTOM Beeline fails with below error:$ beeline --verbose Beeline version 0.14.0.2.2.6.0-2800 by Apache Hive beeline> !connect jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM scan complete in 8ms Connecting to jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM Enter username for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: kuldeepk Enter password for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hive/lib/hive-jdbc-0.14.0.2.2.6.0-2800-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: S

SYMPTOM: Access Audit logs show 6 hours behind from Central Timezone. This is related to https://issues.apache.org/jira/browse/RANGER-336ERROR: Ranger Access Audit logs show 6 hours behind from Central Timezone. ROOT CAUSE: This is bug - https://issues.apache.org/jira/browse/RANGER-336RESOLUTION: Create a file name "ranger-admin-env-javaopts.sh" with below entry in path "/usr/hdp/current/ranger-admin/conf/" - export JAVA_OPTS=" ${JAVA_OPTS} -Duser.timezone=UTC"Save the file and restart Ranger admin service.

SYMPTOM: Ambari agent not able to register with Ambari server. ERROR: Below is the error logs - ERROR 2016-12-19 10:17:54,387 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.comTraceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0)UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byteERROR 2016-12-19 10:17:54,388 Controller.py:195 - Error:'utf8' codec can't decode byte 0x

These are the steps to build and run spark streaming application, it was built and tested on HDP-2.5setup:ENV: HDP2.5scala : 2.10.4sbt: 0.13.11mkdir spark-streaming-example cd spark-streaming-example/ mkdir -p src/main/scala cd src/main/scalasample code:import org.apache.spark.rdd.RDDimport org.apache.spark.sql.{SQLContext, SaveMode}import org.apache.spark.streaming.StreamingContextimport org.apache.spark.streaming.Secondsimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.streaming.Time;import org.apache.spark.SparkConfimport org.apache.spark.SparkContextimport org.apache.spark.rdd.RDDimport org.apache.spark.streaming.{Time, Seconds, StreamingContext}import org.apache.spark.util.IntParamimport org.apache.spark.sql.SQLContextimport org.apache.spark.storage.StorageLevelobject SqlNetworkWordCount { def main(args: Array[String]) { val sparkConf = new SparkConf().setAppName("SqlNetworkWordCount") val ssc = new StreamingContext(spark