Member since
01-21-2016
290
Posts
76
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1027 | 01-09-2017 11:00 AM | |
326 | 12-15-2016 09:48 AM | |
1421 | 02-03-2016 07:00 AM |
03-14-2019
05:48 AM
Hi, I am trying to read a xlsx file and create a dataframe out of it. i have attached the xlsx file. Can anybody provide some clue on it. there are columns based on year and sub columns in it. the end goal is to create a dataframe out of it and store in a hive table for further analysis. if the year related columns are not there, then it is a straight forward one. Any thoughts or help would be great. here we have year 2004 as example and 5 columns under it. similarly there will be many years and corresponding sub columns.
... View more
Labels:
01-29-2018
07:42 AM
How to move a file from edge node to hdfs using an oozie action. Are there any examples that i can refer to?
... View more
Labels:
01-18-2018
07:51 AM
I am getting the following error while executing the below commands. Phoenix version used is : 4.8.1 But when i change the select query as this, I dont see error. Just multiplying it by 1.0 . Though it is stored as decimal, when i try to retrieve , i get the error SELECT (9.71221055496e-05*REPORT_NUM)*(1-DEDUCT_NUM/100)*1.0*MULTIPLER FROM MY_TEST_TABLE; CREATE TABLE MY_TEST_TABLE( UUID1 VARCHAR NOT NULL, UUID2 VARCHAR NOT NULL, ACTION_DATE DATE NOT NULL, ACTION_ID BIGINT NOT NULL, REPORT_NUM DECIMAL(23,6), MULTIPLER DECIMAL(23,6), DEDUCT_NUM DECIMAL(23,6)CONSTRAINT PK PRIMARY KEY (UUID1,UUID2,ACTION_DATE,ACTION_ID)); upsert into MY_TEST_TABLE(UUID1,UUID2,ACTION_DATE,ACTION_ID,REPORT_NUM,MULTIPLER,DEDUCT_NUM) values ('k1','k2','2018-01-18',10105736387,135228000,1.0,0.0); SELECT (9.71221055496e-05*REPORT_NUM)*(1-DEDUCT_NUM/100)*MULTIPLER FROM MY_TEST_TABLE; Error: ERROR 206 (22003): The data exceeds the max capacity for the data type. DECIMAL(38,34) (state=22003,code=206) java.sql.SQLException: ERROR 206 (22003): The data exceeds the max capacity for the data type. DECIMAL(38,34) at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:455) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.exception.DataExceedsCapacityException.<init>(DataExceedsCapacityException.java:34) at org.apache.phoenix.expression.DecimalMultiplyExpression.evaluate(DecimalMultiplyExpression.java:67) at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69) at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:524) at sqlline.Rows$Row.<init>(Rows.java:157) at sqlline.BufferedRows.<init>(BufferedRows.java:38) at sqlline.SqlLine.print(SqlLine.java:1649) at sqlline.Commands.execute(Commands.java:833) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:807) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292)
... View more
Labels:
12-20-2017
05:29 AM
I am using the zookeeper provided by HDF for nifi and i do see znodes created for nifi as shown below. ls /nifi/leaders
[Primary Node, Cluster Coordinator] How can i access the znodes Primary Node and Cluster Coordinator Any help would be great
... View more
Labels:
12-19-2017
03:59 PM
Can apache atlas be installed without having apache ranger installed on the cluster?
... View more
Labels:
11-22-2017
10:45 AM
Does anyone have a standard directory structure for hdfs that can be used as a template for creating a data lake or a basic hadoop cluster. Like how we have a standard directory structure for a web application is there one for a hadoop cluster. IF any one have a sample or a thoughts, please share.
... View more
Labels:
11-13-2017
01:32 AM
Can we just upgrade nifi alone (to the latest version) and still control it using ambari for HDF. Will there be any dependency issue?
... View more
Labels:
11-09-2017
06:43 AM
Hi @Jay Kumar SenSharma, We have done a kinit at the start itself. Thats how the yarn containers are gtting launched. But from there the phoenix connection is not able to obtained
... View more
11-09-2017
06:14 AM
We are running a Mapreduce program which makes use of apache pheonix to get and process some data. The job fails with a kerberos error. The same program when run standalone runs fine. We are making use of keytab to authenticate and the keytab is used in the connection object to create to a phoenix connection. The MR job starts and launches containers,but the phoenix connection couldn't be established and the job fails with the error 2017-11-08 18:38:36,121 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Opening socket connection to server 172.16.3.197/172.16.3.197:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-08 18:38:36,122 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Socket connection established to 172.16.3.197/172.16.3.197:2181, initiating session
2017-11-08 18:38:36,138 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 172.16.3.197/172.16.3.197:2181, sessionid = 0x35e92d03f140314, negotiated timeout = 40000
2017-11-08 18:38:36,693 INFO [main] org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2017-11-08 18:38:36,693 WARN [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system already initialized!
2017-11-08 18:38:38,948 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-11-08 18:38:38,949 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:612)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:157)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:738)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:735)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:897)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:866)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1209)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:373)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:200)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:350)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:324)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 26 more
2017-11-08 18:38:42,802 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-11-08 18:38:42,803 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
... View more
Labels:
10-27-2017
09:36 AM
Hi @Erik Naslund, I tried the same. IT didnt'work for me. Do we need to reboot the EC2 instance after adding the role
... View more
10-27-2017
09:28 AM
I am trying to use the backup utility of hbase to backup hbase tables to S3. But i am facing the following error despite giving the correct access key and secret key. hbase backup create full s3a://$ACCESS_KEY:$SECRET_KEY@hbase-bucket/tables -set systems Error is ERROR [main] util.AbstractHBaseTool: Error running command-line tool
java.nio.file.AccessDeniedException: s3a://xxxxxxxxxxxxxxxxxxxxxxxxxxxxx+g@hbase-bucket/tables: getFileStatus on s3a://xxxxxxxxxxxxxxxxxxxxxxxxxxx+g@hbase-bucket/tables: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 99EC757749F1D141), S3 Extended Request ID: gTGozzH9nuMdbfqfmMwrkPh17iuacp0CXQZ3jzcaYNnvdzxgExXQjzxZOrDG+RT0y/ArKI2QOfU But i am able to use the command without any error hadoop fs -ls s3a://hbase-bucket/ i have set fs.s3a.access.key and fs.s3a.secret.key in core-site.xml any thoughts, i have given full permissions to the bucket
... View more
Labels:
10-05-2017
05:26 AM
Hi Kuldeep, thanks for this hack. IS there a similar hack that can be applied to schema registry too?
... View more
10-04-2017
07:55 AM
1 Kudo
@Jay SenSharma, i didnt enable spnego for any of the services. other service ui's i am able to access without any issues. Is there a way to disable spnego for registry alone, i couldn't find that option in ambari
... View more
10-04-2017
07:32 AM
I have co hosted both HDP and HDF using the same ambari. the cluster is kerberised too. After installing schema registry of HDF, i am not able to access the UI. I am prompted for a user name/password. I tried some kerberos principals and passwords, but none of them worked. i do see an error like this. IS there a way to avoid this or even skip kerberos for the registry UI. Ambari used is 2.5.2 HDf - 3.0 HTTP ERROR 403 Problem accessing /. Reason: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
... View more
09-05-2017
07:00 AM
@vsharma, Vivek, I upgraded ambari to 2.5.2.0 and tried installing the management pack but after registering the HDF version, i click on Install on the "cluster name" and ambari is redirected to the versions page where i dont see the HDF version.at all. I have attached the screen shots.hdf-3.pngonlyhdp.png
... View more
09-04-2017
12:40 PM
Hi Jasper, I have attached the screen shot. I am not able to install HDP. you could see it from the attached screenshot.
... View more
09-04-2017
05:21 AM
hdf-hdp.pngI recently came to know that both HDP and HDF can be installed via the same ambari. But when i tried installing HDF , i am unable to do so. I added the management pack for HDF 3.0 and registered in ambari, but i am unable to install it. I have attached the screen shot of this.Can you please suggest what to do. The ambari version is 2.5.0.3
... View more
09-01-2017
06:07 AM
I am getting this error when i try to create a basic shell action in the WFM view. this error occurs when i tried to save the work flow Error occurred while saving workflow
org.apache.ambari.view.PersistenceException: Caught exception trying to find org.apache.oozie.ambari.view.workflowmanager.model.Workflow where workflowDefinitionPath='/user/arun/oozie/shellworkflow.xml'
Hide Logjava.lang.RuntimeException: org.apache.ambari.view.PersistenceException: Caught exception trying to find org.apache.oozie.ambari.view.workflowmanager.model.Workflow where workflowDefinitionPath='/user/arun/oozie/shellworkflow.xml'
at org.apache.oozie.ambari.view.workflowmanager.WorkflowsRepo.getWorkflowByPath(WorkflowsRepo.java:66) at org.apache.oozie.ambari.view.workflowmanager.WorkflowManagerService.saveWorkflow(WorkflowManagerService.java:56) at org.apache.oozie.ambari.view.OozieProxyImpersonator.saveWorkflow(OozieProxyImpersonator.java:249) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.acce
... View more
Labels:
08-31-2017
12:00 PM
I executed the following rest api command to delete old versions of HDP. Seems there is a bug in the script. the version is not getting deleted.A '/' is missing in the command that is to be executed for the cleaning. Env used: Ambari - 2.5.0.3 , HDP 2.6 curl 'http://ambarihost:8080/api/v1/clusters/cluster01/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.4.0.0-169"}}, "Requests/resource_filters": [{"hosts":"host1,host2,host3"}]}'
Results are like this
2017-08-31 17:21:52,102 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
2017-08-31 17:21:52,156 - call returned (0, '2.3.2.0-2800\n2.4.0.0-169\n2.6.0.3-8')
2017-08-31 17:21:52,157 - available versions: ['2.3.2.0-2800', '2.4.0.0-169', '2.6.0.3-8']
2017-08-31 17:21:52,158 - version 2.3.2.0-2800 added to remove
2017-08-31 17:21:52,421 - Removing /usr/hdp/2.3.2.0-2800
2017-08-31 17:21:52,421 - Execute[('rm', '-f', '/usr/hdp2.3.2.0-2800')] {'sudo': True}
Command completed successfully!
... View more
08-31-2017
10:40 AM
Starting from HDP 3.0 , is kafka getting moved out. Will it be part of HDF?
... View more
Labels:
08-25-2017
08:22 AM
@Michael Young, Will ambari 2.5.0.3 support both HDP and HDF (as management pack) or ambari 2.5.1.0 is required?
... View more
08-23-2017
07:24 AM
From here can i download the docker image for HDP 2.6 using wget on a linux terminal.?
... View more
08-21-2017
10:57 AM
@Ankit Singhal, we have set this as shown below export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize=94208m " and our bucket cache size is 92160 mb This was working perfectly before the upgrade. 92 GB(max direct memory) > 90 GB (bucket cache) we will try setting on the master one also and get back
... View more
08-21-2017
09:20 AM
@Akhil S Naik, Thanks for your answer. but it wont help our case. we already had the bucket cache enabled and we are making use of it. SO removing the bucket cache is not a good option for us as we need to retain that. These were our values hbase.bucketcache.size = 92160 MB MaxDirectMemorySize=94208 MB
... View more
08-21-2017
07:39 AM
We recently upgraded from ambari 2.2 to 2.5 and hdp from 2.4 to 2.6. We have hbase bucket cache enabled. But after the upgrade hbase master is continuously failing. the exception is shown below. But once we disabled the bucket cache, the hbase master process is coming up. How to fix this? java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2756)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2770)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:311)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:221)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:576)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:425)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
... View more
Labels:
08-20-2017
01:58 AM
We recently upgraded to ambari 2.5 from 2.2. For every 2 hours we are getting ambari alert mails stating that there are 223 stale alerts. And in another 2 minutes, this alert gets cleared. I have attached the stale alert message. Anything can i do to get rid of this stalealerts.txt
... View more
Labels:
08-07-2017
11:16 AM
Thanks @Sindhu, just wanted to come up with the db instance sizing.
... View more
08-07-2017
05:18 AM
Is there any standard or template available for hive metastore database sizing. We need to provide to the database platform team our requirements. we have a 30 node cluster with 3 masters and 27 slaves.
... View more
Labels:
08-01-2017
07:51 AM
@Kartik Ramalingam, yes it is listening netstat -na | grep 6667
tcp 0 0 172.16.3.196:6667 0.0.0.0:* LISTEN
tcp 0 0 172.16.3.196:57076 172.16.3.196:6667 TIME_WAIT
tcp 0 0 172.16.3.196:6667 172.16.3.197:58774 ESTABLISHED
tcp 0 0 172.16.3.196:6667 172.16.3.197:39880 ESTABLISHED
... View more