Member since
04-01-2019
20
Posts
0
Kudos Received
0
Solutions
02-24-2020
07:21 AM
I am trying to load data into hbase table but it is throwing error :
Caused by: <line 2, column 57> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.pig.backend.hadoop.hbase.HbaseStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1339) at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1324) at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5184) at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7782) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
commands executed:
register /usr/hdp/2.6.4.63-2/hbase/lib/hbase-*.jar
rawd = LOAD '/user/bdauser/hbasedata.txt' USING PigStorage(',') AS (product:charArray,type:charArray);
: grunt> STORE rawd into 'hbase://odsvivonext:pighbasetest' USING apache.pig.backend.hadoop.hbase.HbaseStorage('hbasedata:product,hbasedata:type');
... View more
Labels:
02-19-2020
12:42 AM
I am trying to schedule hbase job with oozie but when i run the job I am getting unknown host exception on nameservice . We are using nameservice as namenode is HA enabled .
... View more
Labels:
02-18-2020
01:51 AM
I have created oozie job to write/read data from hbase but my job is getting killed with error variable [hbase] cannot be resolved .
... View more
Labels:
02-14-2020
12:29 AM
I have copied hbase-commom.jar in oozie share lib path under /user/oozie/share/lib/java and /user/oozie/sahre/lib/pig but this jar is not getting picked up by oozie and throwing an error hbase class not found but when i copy jars to /usr/hdp/current/oozie/libext/ and /usr/hdp/current/oozie/lib and /usr/hdp/current/oozie/oozie-server/webapps/oozie/WEB-INF/lib/ . oozie is able to pick jar and job is getting completed successfully .
I am confused why oozie is not able to pick up jar though jars are present in hdfs oozie share lib but when i copy in local oozie server in web inf and libext job is getting completed successfully .
... View more
Labels:
01-30-2020
02:08 AM
I have streaming application which is ingesting data into kafka in HDP cluster but I am not understanding how to manage schema , I do not want to send schema with every record . How can I manage with schema registry in hdp cluster without installing HDF .
... View more
01-07-2020
05:13 AM
I want to read habse tables and perform transformation over data and store final result into kafka topic with the help of spark streaming job . I am following below procedure to achieve the above requirement .
I am using newAPIHadoopRDD to read hbase table from spark stream and run transformation over data , In this step I am loading data into RDD but to I want to register schema also with Kafka as my final destination is insertion of records into hive .
Basically i am following below steps :
1. read hbase tables and load data into spark rdd.
2. perform transformation .
3. load transformed data into kafka topic .
all above steps I am running via spark stream job . for step first newAPIHadoopRDD will help to read data , for transformation I have functions in spark also custom functions and this final data load into kafka acting spark job as producer for kafka and utilizing kafka apis to achieve the same .But I am not sure how to register hbase schema with kafka ?
... View more
01-07-2020
02:07 AM
I have staging area and data loaded in staging is in avro format . can I create ORC file from avro file format directly without creating avro table in hive ? as data in already in the binary format in avro , is it possible to directly create ORC table rather than first creating DDL in hive in avro format and later insert data in ORC table from avro ?
... View more
Labels:
12-13-2019
01:29 AM
@Shelton thanks for your reply ! this cross realm works when we want to setup cross realm between AD and MIT KDC server , but when your edge node is in different realm and datanodes are in different realm how will you setup corss realm ? my problem is when AD users will login edge node and get kerberos ticket which will be communicated to namenode and namenode will gice delegate token to authenticate further , but as this case datanodes are in not in the same realm as edge node hence same delegate token won't work . how do i setup cross realm between edge node and datanodes . do i need to setup like following : 1) cross realm MIT KDC (edge node ) to AD . 2) cross realm MIT KDC (datanodes ) to AD. I have got confused with cross realm ideas though link provided you MIT KDC to AD trust I do understand but I am not able to get solution when edge node and data nodes both are in different real . thanks
... View more
12-12-2019
01:39 AM
I have edge node in different realm and data node is in different real , kerberos is integrated with Active Directory (AD) .
As per my understanding if realm is same for all the nodes in the machine in that case user logs in edge node gets kerberos ticket as kerberos is integrated with AD . When user will submit its job ticket is transerred to namenode and authetication will take place once authenticated namenode will use grant delegate token . with the help of delegate token in all other nodes job will continue to work .
But as in this case datanodes are in different realm though user gets delegate token when user logs in edge node but authentication will fail in data node as its realm is not correct . What is solution in this case ?
... View more
Labels:
10-11-2019
02:57 AM
I am confused with hbase snapshot cloning mechanism . snapshot cloning in hbase allows creation of new table from existing snapshot taken for some other table , but any modification to newly created table does not have any impact on either snapshot or original table for what snaphsot was created . below is example What I am trying to justify on above statement : suppose we have hbase table tab , and it has two records cf1:fname ='Anurag' cf2:Iname ='Mishra' . .Now with this structure I have created snapshot of this table with name tabsnapshot . As snapshot clone operation allows creating new table let's assume I have created new table with name newtab . Hence now newtab table with same structure and data as tab table has been created , when I update record in the newtab table it does not make any difference in the actual table neither on snapshot . How is it possible when there is no data movement still any update on the new table does not update data for original table it does update only for itself but no disturbance in the original table . As per the functionality wise this should be the only case new table created from the snapshot should not disturb other table , but snapshot operation does not involve data copy or movement and does not affect region server how come update on data does not make any difference on original hbase table >
... View more
Labels:
04-21-2019
08:58 AM
I am new to cloudera , earlier I was working on Hortonworks hadoop distribution . I am installing cloudera manager following its documentation page , but while setting database(MYSQL ) with cloudera manager I am facing below error : root@instance-1 etc]# /usr/share/cmf/schema/scm_prepare_database.sh postgresql scm scm scm JAVA_HOME=/usr/java/jdk1.6.0_31 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.6.0_31/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/util/PSQLException : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: com.cloudera.enterprise.dbutil.DbCommandExecutor. Program will exit. --> Error 1, giving up (use --force if you wish to ignore the error)
... View more
Labels:
04-01-2019
05:56 AM
there are three kafka brokers , in the cluster but one of the broker is not available in the zookeeper . In the cluster there are two role groups one is default role group where there are two kafka brokers and other one with other name with one broker available . What could be issue that lead to unavailibilty of one kafka broker in zookeeper.
... View more
09-07-2018
01:13 PM
There are very old files under /spark2-history and because of this spark2 web ui is getting stuck on loading history , I have gone thorough some of the documentations where it is mentioned setting below property will solve the issue .Please find below property : spark.history.fs.cleaner.enabled spark.history.fs.cleaner.maxAge
spark.history.fs.cleaner.interval would these property be same for the spark2 as well ? also I do not see these property already existing , and in spark docs default property is mentioned there . https://spark.apache.org/docs/latest/monitoring.html
... View more
Labels:
09-06-2018
12:27 PM
I am getting error when i try to access oozie workflow manager view : java.lang.RuntimeException: java.lang.NullPointerException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3036) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218) at org.apache.ambari.server.view.ViewURLStreamProvider.getHttpURLConnection(ViewURLStreamProvider.java:239) at org.apache.ambari.server.view.ViewURLStreamProvider.getInputStream(ViewURLStreamProvider.java:216) at org.apache.ambari.server.view.ViewURLStreamProvider.readFrom(ViewURLStreamProvider.java:103) at org.apache.ambari.server.view.ViewURLStreamProvider.readAs(ViewURLStreamProvider.java:117) at org.apache.ambari.server.view.ViewURLStreamProvider.readAsCurrent(ViewURLStreamProvider.java:131) at org.apache.oozie.ambari.view.AmbariIOUtil.readFromUrl(AmbariIOUtil.java:45) at org.apache.oozie.ambari.view.OozieDelegate.readFromOozie(OozieDelegate.java:152) at org.apache.oozie.ambari.view.OozieDelegate.consumeService(OozieDelegate.java:118) at org.apache.oozie.ambari.view.OozieDelegate.consumeService(OozieDelegate.java:111) at org.apache.oozie.ambari.view.OozieProxyImpersonator.handleGet(OozieProxyImpersonator.java:453) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at I have set below property in custum-oozie : oozie.service.ProxyUserService.proxyuser.ambari-server-AADSPRD.hosts=*
oozie.service.ProxyUserService.proxyuser.ambari-server-AADSPRD.groups=* oozie.authentication.type=kerberos I followed below link for the resolving the same but still I am facing the issue : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-views/content/wfm-kerberos-setup.html
... View more
Labels:
03-16-2018
01:26 PM
@Aditya Sirna Thanks Aditya , but is there a way instead of rest api , clients can be installed with the help of ambari .
... View more
03-16-2018
09:45 AM
I want to install clients for the spark and spark2 in all the nodes services are already installed . can some one help me to the procedure how to achieve the above purpose . when I click on the service tab and then I see services option but I can not proceed further ., since all services are pre-selected . I want to install clients in all nodes . please help me on this Thanks in advacne
... View more
01-05-2018
05:30 AM
@Jay Kumar SenSharma Qorg/apache/ambari/server/controller/AmbariServer.class class in file:/usr/lib/ambari-server/ambari-server-2.6.0.0.267.jar!/org/apache/ambari/server/controller/AmbariServer.class
05 Jan 2018 05:06:29,311 INFO [main] Configuration:3048 - Reading password from existing file
05 Jan 2018 05:06:29,320 INFO [main] Configuration:3542 - Hosts Mapping File null
05 Jan 2018 05:06:29,321 INFO [main] HostsMap:60 - Using hostsmap file null
05 Jan 2018 05:06:29,793 INFO [main] ControllerModule:224 - Detected POSTGRES as the database type from the JDBC URL
05 Jan 2018 05:06:29,842 INFO [main] ControllerModule:545 - Searching package org.apache.ambari.server for annotations matching [interface org.apache.ambari.server.EagerSingleton, interface org.apache.ambari.server.StaticallyInject, interface org.apache.ambari.server.AmbariService]
05 Jan 2018 05:06:30,128 INFO [main] ClasspathScannerUtils:54 - Checking package [org.apache.ambari.server] for binding candidates.
05 Jan 2018 05:06:30,664 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService
05 Jan 2018 05:06:30,671 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.AlertNoticeDispatchService
05 Jan 2018 05:06:30,673 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.AmbariServerAlertService
05 Jan 2018 05:06:30,674 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.CachedAlertFlushService
05 Jan 2018 05:06:30,827 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.MetricsRetrievalService
05 Jan 2018 05:06:30,829 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.RetryUpgradeActionService
05 Jan 2018 05:06:30,849 INFO [main] ControllerModule:615 - Searching package org.apache.ambari.server.notifications.dispatchers for dispatchers matching interface org.apache.ambari.server.notifications.NotificationDispatcher
05 Jan 2018 05:06:30,930 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.SNMPDispatcher
05 Jan 2018 05:06:30,937 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.EmailDispatcher
05 Jan 2018 05:06:30,944 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AlertScriptDispatcher
05 Jan 2018 05:06:30,945 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AmbariSNMPDispatcher
05 Jan 2018 05:06:30,946 INFO [main] ControllerModule:679 - Searching package org.apache.ambari.server.checks for classes matching class org.apache.ambari.server.checks.AbstractCheckDescriptor
05 Jan 2018 05:06:31,111 INFO [main] ControllerModule:715 - Registered pre-upgrade check class org.apache.ambari.server.checks.HostsMasterMaintenanceCheck
05 Jan 2018 05:06:31,111 INFO [main] ControllerModule:715 - Registered pre-upgrade check class org.apache.ambari.server.checks.ServicesMaintenanceModeCheck tcp6 0 0 :::8080 :::* LISTEN 2710/java service iptables status : ● iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
Active: inactive (dead) [root@namenode ~]# telnet 104.197.146.171 8080
Trying 104.197.146.171...
telnet: connect to address 104.197.146.171: Connection timed out [root@namenode ~]# telnet namenode.c.famous-augury-181309.internal 8080
Trying 10.128.0.2...
Connected to namenode.c.famous-augury-181309.internal.
Escape character is '^]'. nOte : I am using google cloud platfrom vm instance to install cluster. [root@namenode ~]# telnet namenode.c.famous-augury-181309.internal 8080
Trying 10.128.0.2...
Connected to namenode.c.famous-augury-181309.internal.
Escape character is '^]'. if i open ambari web ui replacing ip address with hostname it shows : This site can’t be reached namenode.c.famous-augury-181309.internal’s server IP address could not be found.
Search Google for namenode famous augury 181309 internal 8080 ERR_NAME_NOT_RESOLVED
... View more
01-04-2018
08:14 PM
I have started ambari server but when i open ambari web ui it shows me connection time out . I ran telnet command : telnet ambari_ipaddress 8080 trying .... telnet: connect to address Connection timed out I am beginner to hadoop please help me out Thanks in advance
... View more
Labels:
11-20-2017
07:42 PM
hi , amabri -server status i have cehcked already it is in running state. when I try to access amabri web ui it shows me some DNS problem and if i repalce with ip it gives error as time out or more time taken, I have installed ambari server on machine that ip address i am taking as amabri server access.
... View more
11-20-2017
07:15 PM
Hi, i am new to all of this,, i have created my instances on google cloud but i after ambari server setup , i tried to access amabri through UI but i am not able to access
... View more
Labels: