We are on 2.5.3 and moving to 2.6.1 - this cluster also hosts lucidworks-hdpsearch Solr cloud - rather than installing the packages manually we added them to Ambari for easy start/stop, Should we remove them before the upgrade or leave them as it is - These nodes are also running ambari-metrics monitor - ambari and HST agents too. If I put the host itself on maintenance mode the above components will be left out. Any suggestions? Thanks in advance. ... View more

Hello gurus, We were facing issue with Kafka and decided to re-installed service (removed via Ambari) Now the kafka brokers refuse to start with following errors. FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.ClassNotFoundException: org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer I'm out of ideas on where to look and what jars. my /etc/kafka and /usr/hdp/current paths are good. Thanks in advance. Mayank ... View more

Hello experts, For no reason after a restart all the users can't see each others job view info from the RM UI. Each user can ONLY see the application info on RM UI running under their ID. RM HA Kerberized below is the error - 2016-10-14 14:27:42,109 ERROR webapp.AppBlock (AppBlock.java:render(233)) - Failed to read the attempts of the application application_1476471035662_0002. java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727) at org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:221) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMAppBlock.render(RMAppBlock.java:65) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) at org.apache.hadoop.yarn.webapp.View.render(View.java:235) at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: User mayank does not have privilage to see this aplication application_1473331035333_09 Thanks in advance Mayank ... View more

We tried 3 scenarios and all failed. 1) Creating Hive Table on HBase. (proper hive/hbase polices for user) hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz", "hbase.mapred.output.outputtable" = "xyz"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@ABC.COM' (action=create) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:254) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:664) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:769) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:496) at org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:216) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1140) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:212) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1533) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2) Create Table directly on Hbase hbase(main):001:0> create 'emp', 'personal', 'professional' ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@ABC.COM' (action=create) 3) Using Grant from HBase Shell hbase(main):001:0> grant 'abc', 'RWCA' ERROR: org.apache.hadoop.hbase.coprocessor.CoprocessorException: SSLContext must not be null at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1171) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7553) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1878) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1860) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) The easy solution we tried is to disable SSL and it worked. We also tried creating different policies and policy on namespace 'default:*' worked and users were ONLY able to create the TABLE. They were NOT able to scan the table and issue (3) still had same problem. " ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user abc@ABC.COM',action: scannerOpen, tableName:hello1, family:col1. " Regional Servers were playing a important role here on Ranger Policies. While HBase Master was able to wirte to Policy Cache the cache for Regional Servers was 0 bytes.(communication blocked) We distributed the keystore and truststore from HBase Master to all the worker nodes running Regional Server and restarted Hbase and this solved the issue. How to do? Ambari-Hbase-Config- Filter "SSL" 1) xasecure.policymgr.clientssl.keystore - Find the /path/keystore.file.name and distribute it to all the machines (keep the path/file name same) 2) xasecure.policymgr.clientssl.truststore - - Find the /path/truststore.file.name and distribute it to all the machines (keep the path/file name same) Hope this will help. Thanks Mayank ... View more

Strangely, we have a client script which is working fine on "pyspark/default" mode however fails on "pyspark --master yarn" mode. This client has 3 environments and the script is failing on one of the environment only, all the configs are same across the environments and they were upgraded recently. The log from RM shows below.(see bold for 2 versions) CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/*<CPS>/usr/hdp/current/hadoop-client/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.3.4.31-2/hadoop/lib/hadoop-lzo-0.6.0.2.3.4.31-2.jar:/etc/hadoop/conf/secure SPARK_LOG_URL_STDERR -> http://bvlhdppdn05.conocophillips.net:8042/node/containerlogs/container_e1394_1466953988053_3651_01_000003/katarm/stderr?start=-4096 SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1466953988053_3651 SPARK_YARN_CACHE_FILES_FILE_SIZES -> 188131242,335155,37562 SPARK_USER -> mayank SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,PRIVATE SPARK_YARN_MODE -> true SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1467231616831,1467231616933,1467231616967 SPARK_HOME -> /usr/hdp/2.3.2.0-2950/spark/ PYTHONPATH -> /usr/hdp/2.3.4.31-2/spark/python/lib/py4j-0.8.2.1-src.zip:/usr/hdp/2.3.4.31-2/spark/python/:<CPS>{{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.8.2.1-src.zip SPARK_LOG_URL_STDOUT -> http://client.net:8042/node/containerlogs/container_e1394_1466953988053_3651_01_000003/mayank/stdout?start=-4096 SPARK_YARN_CACHE_FILES -> hdfs://clustername/user/mayank/.sparkStaging/application_1466953988053_3651/spark-assembly-1.5.2.2.3.4.31-2-hadoop2.7.1.2.3.4.31-2.jar#__spark__.jar,hdfs://clustername/user/mayank/.sparkStaging/application_1466953988053_3651/pyspark.zip#pyspark.zip,hdfs://clutername/user/mayank/.sparkStaging/application_1466953988053_3651/py4j-0.8.2.1-src.zip#py4j-0.8.2.1-src.zip Also the CLI throws below error, SPARK_HOME is pointing to old version for serializer however using right python libs from new version. File "/usr/hdp/2.3.4.31-2/spark/python/pyspark/rdd.py", line 1295, in takeUpToNumLeft yield next(iterator) File "/usr/hdp/2.3.2.0-2950/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 139, in load_stream yield self._read_with_length(stream) File "/usr/hdp/2.3.2.0-2950/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 164, in _read_with_length I have set the right path for $SPARK_USER under my .bashrc. Thanks Mayank ... View more

Hello experts, One of our client was facing below error while doing a left outer join (scope was to flatten the data) error "Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:193, Vertex vertex_1461710542055_2118_9_00 [Map 3] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2," We suggested a work around to set below to false since in certain cases where a partitioned table has a DDL change. Vectorization has a problem. "set hive.vectorized.execution.enabled=false" And the above solved the issue. My Question : What is the benefit of having the parameter? My understanding is vectorization minimizes high CPU use, so a 'false' value mean we lose performance? How can we add a new attributes on a Large Partition table without performance degradation or getting queries to fail as it did earlier? Suggestions for best practices. Thanks Mayank ... View more