Hi, I am trying to read a xlsx file and create a dataframe out of it. i have attached the xlsx file. Can anybody provide some clue on it. there are columns based on year and sub columns in it. the end goal is to create a dataframe out of it and store in a hive table for further analysis. if the year related columns are not there, then it is a straight forward one. Any thoughts or help would be great. here we have year 2004 as example and 5 columns under it. similarly there will be many years and corresponding sub columns. ... View more

How to move a file from edge node to hdfs using an oozie action. Are there any examples that i can refer to? ... View more

I am getting the following error while executing the below commands. Phoenix version used is : 4.8.1 But when i change the select query as this, I dont see error. Just multiplying it by 1.0 . Though it is stored as decimal, when i try to retrieve , i get the error SELECT (9.71221055496e-05*REPORT_NUM)*(1-DEDUCT_NUM/100)*1.0*MULTIPLER FROM MY_TEST_TABLE; CREATE TABLE MY_TEST_TABLE( UUID1 VARCHAR NOT NULL, UUID2 VARCHAR NOT NULL, ACTION_DATE DATE NOT NULL, ACTION_ID BIGINT NOT NULL, REPORT_NUM DECIMAL(23,6), MULTIPLER DECIMAL(23,6), DEDUCT_NUM DECIMAL(23,6)CONSTRAINT PK PRIMARY KEY (UUID1,UUID2,ACTION_DATE,ACTION_ID)); upsert into MY_TEST_TABLE(UUID1,UUID2,ACTION_DATE,ACTION_ID,REPORT_NUM,MULTIPLER,DEDUCT_NUM) values ('k1','k2','2018-01-18',10105736387,135228000,1.0,0.0); SELECT (9.71221055496e-05*REPORT_NUM)*(1-DEDUCT_NUM/100)*MULTIPLER FROM MY_TEST_TABLE; Error: ERROR 206 (22003): The data exceeds the max capacity for the data type. DECIMAL(38,34) (state=22003,code=206) java.sql.SQLException: ERROR 206 (22003): The data exceeds the max capacity for the data type. DECIMAL(38,34) at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:455) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.exception.DataExceedsCapacityException.<init>(DataExceedsCapacityException.java:34) at org.apache.phoenix.expression.DecimalMultiplyExpression.evaluate(DecimalMultiplyExpression.java:67) at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69) at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524) at sqlline.Rows$Row.<init>(Rows.java:157) at sqlline.BufferedRows.<init>(BufferedRows.java:38) at sqlline.SqlLine.print(SqlLine.java:1649) at sqlline.Commands.execute(Commands.java:833) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:807) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) ... View more

Can apache atlas be installed without having apache ranger installed on the cluster? ... View more

Can we just upgrade nifi alone (to the latest version) and still control it using ambari for HDF. Will there be any dependency issue? ... View more

We are running a Mapreduce program which makes use of apache pheonix to get and process some data. The job fails with a kerberos error. The same program when run standalone runs fine. We are making use of keytab to authenticate and the keytab is used in the connection object to create to a phoenix connection. The MR job starts and launches containers,but the phoenix connection couldn't be established and the job fails with the error 2017-11-08 18:38:36,121 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Opening socket connection to server 172.16.3.197/172.16.3.197:2181. Will not attempt to authenticate using SASL (unknown error) 2017-11-08 18:38:36,122 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Socket connection established to 172.16.3.197/172.16.3.197:2181, initiating session 2017-11-08 18:38:36,138 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 172.16.3.197/172.16.3.197:2181, sessionid = 0x35e92d03f140314, negotiated timeout = 40000 2017-11-08 18:38:36,693 INFO [main] org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix 2017-11-08 18:38:36,693 WARN [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system already initialized! 2017-11-08 18:38:38,948 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2017-11-08 18:38:38,949 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:612) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:157) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:738) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:735) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:897) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:866) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1209) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:373) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:200) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:350) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:324) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 26 more 2017-11-08 18:38:42,802 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2017-11-08 18:38:42,803 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ... View more

I am trying to use the backup utility of hbase to backup hbase tables to S3. But i am facing the following error despite giving the correct access key and secret key. hbase backup create full s3a://$ACCESS_KEY:$SECRET_KEY@hbase-bucket/tables -set systems Error is ERROR [main] util.AbstractHBaseTool: Error running command-line tool java.nio.file.AccessDeniedException: s3a://xxxxxxxxxxxxxxxxxxxxxxxxxxxxx+g@hbase-bucket/tables: getFileStatus on s3a://xxxxxxxxxxxxxxxxxxxxxxxxxxx+g@hbase-bucket/tables: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 99EC757749F1D141), S3 Extended Request ID: gTGozzH9nuMdbfqfmMwrkPh17iuacp0CXQZ3jzcaYNnvdzxgExXQjzxZOrDG+RT0y/ArKI2QOfU But i am able to use the command without any error hadoop fs -ls s3a://hbase-bucket/ i have set fs.s3a.access.key and fs.s3a.secret.key in core-site.xml any thoughts, i have given full permissions to the bucket ... View more

I have co hosted both HDP and HDF using the same ambari. the cluster is kerberised too. After installing schema registry of HDF, i am not able to access the UI. I am prompted for a user name/password. I tried some kerberos principals and passwords, but none of them worked. i do see an error like this. IS there a way to avoid this or even skip kerberos for the registry UI. Ambari used is 2.5.2 HDf - 3.0 HTTP ERROR 403 Problem accessing /. Reason: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) ... View more

hdf-hdp.pngI recently came to know that both HDP and HDF can be installed via the same ambari. But when i tried installing HDF , i am unable to do so. I added the management pack for HDF 3.0 and registered in ambari, but i am unable to install it. I have attached the screen shot of this.Can you please suggest what to do. The ambari version is 2.5.0.3 ... View more

I am getting this error when i try to create a basic shell action in the WFM view. this error occurs when i tried to save the work flow Error occurred while saving workflow org.apache.ambari.view.PersistenceException: Caught exception trying to find org.apache.oozie.ambari.view.workflowmanager.model.Workflow where workflowDefinitionPath='/user/arun/oozie/shellworkflow.xml' Hide Logjava.lang.RuntimeException: org.apache.ambari.view.PersistenceException: Caught exception trying to find org.apache.oozie.ambari.view.workflowmanager.model.Workflow where workflowDefinitionPath='/user/arun/oozie/shellworkflow.xml' at org.apache.oozie.ambari.view.workflowmanager.WorkflowsRepo.getWorkflowByPath(WorkflowsRepo.java:66) at org.apache.oozie.ambari.view.workflowmanager.WorkflowManagerService.saveWorkflow(WorkflowManagerService.java:56) at org.apache.oozie.ambari.view.OozieProxyImpersonator.saveWorkflow(OozieProxyImpersonator.java:249) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.acce ... View more

I executed the following rest api command to delete old versions of HDP. Seems there is a bug in the script. the version is not getting deleted.A '/' is missing in the command that is to be executed for the cleaning. Env used: Ambari - 2.5.0.3 , HDP 2.6 curl 'http://ambarihost:8080/api/v1/clusters/cluster01/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.4.0.0-169"}}, "Requests/resource_filters": [{"hosts":"host1,host2,host3"}]}' Results are like this 2017-08-31 17:21:52,102 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {} 2017-08-31 17:21:52,156 - call returned (0, '2.3.2.0-2800\n2.4.0.0-169\n2.6.0.3-8') 2017-08-31 17:21:52,157 - available versions: ['2.3.2.0-2800', '2.4.0.0-169', '2.6.0.3-8'] 2017-08-31 17:21:52,158 - version 2.3.2.0-2800 added to remove 2017-08-31 17:21:52,421 - Removing /usr/hdp/2.3.2.0-2800 2017-08-31 17:21:52,421 - Execute[('rm', '-f', '/usr/hdp2.3.2.0-2800')] {'sudo': True} Command completed successfully! ... View more

From here can i download the docker image for HDP 2.6 using wget on a linux terminal.? ... View more

We recently upgraded from ambari 2.2 to 2.5 and hdp from 2.4 to 2.6. We have hbase bucket cache enabled. But after the upgrade hbase master is continuously failing. the exception is shown below. But once we disabled the bucket cache, the hbase master process is coming up. How to fix this? java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2756) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2770) Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65) at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:311) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:221) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:576) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:425) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ... View more

We recently upgraded to ambari 2.5 from 2.2. For every 2 hours we are getting ambari alert mails stating that there are 223 stale alerts. And in another 2 minutes, this alert gets cleared. I have attached the stale alert message. Anything can i do to get rid of this stalealerts.txt ... View more