About warrior

warrior · ‎02-24-2020

I am trying to load data into hbase table but it is throwing error : Caused by: <line 2, column 57> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.pig.backend.hadoop.hbase.HbaseStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1339) at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1324) at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5184) at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7782) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) commands executed: register /usr/hdp/2.6.4.63-2/hbase/lib/hbase-*.jar rawd = LOAD '/user/bdauser/hbasedata.txt' USING PigStorage(',') AS (product:charArray,type:charArray); : grunt> STORE rawd into 'hbase://odsvivonext:pighbasetest' USING apache.pig.backend.hadoop.hbase.HbaseStorage('hbasedata:product,hbasedata:type');

warrior · ‎02-19-2020

I am trying to schedule hbase job with oozie but when i run the job I am getting unknown host exception on nameservice . We are using nameservice as namenode is HA enabled .

warrior · ‎02-18-2020

I have created oozie job to write/read data from hbase but my job is getting killed with error variable [hbase] cannot be resolved .

warrior · ‎02-14-2020

I have copied hbase-commom.jar in oozie share lib path under /user/oozie/share/lib/java and /user/oozie/sahre/lib/pig but this jar is not getting picked up by oozie and throwing an error hbase class not found but when i copy jars to /usr/hdp/current/oozie/libext/ and /usr/hdp/current/oozie/lib and /usr/hdp/current/oozie/oozie-server/webapps/oozie/WEB-INF/lib/ . oozie is able to pick jar and job is getting completed successfully . I am confused why oozie is not able to pick up jar though jars are present in hdfs oozie share lib but when i copy in local oozie server in web inf and libext job is getting completed successfully .

warrior · ‎01-30-2020

I have streaming application which is ingesting data into kafka in HDP cluster but I am not understanding how to manage schema , I do not want to send schema with every record . How can I manage with schema registry in hdp cluster without installing HDF .

warrior · ‎01-07-2020

I want to read habse tables and perform transformation over data and store final result into kafka topic with the help of spark streaming job . I am following below procedure to achieve the above requirement . I am using newAPIHadoopRDD to read hbase table from spark stream and run transformation over data , In this step I am loading data into RDD but to I want to register schema also with Kafka as my final destination is insertion of records into hive . Basically i am following below steps : 1. read hbase tables and load data into spark rdd. 2. perform transformation . 3. load transformed data into kafka topic . all above steps I am running via spark stream job . for step first newAPIHadoopRDD will help to read data , for transformation I have functions in spark also custom functions and this final data load into kafka acting spark job as producer for kafka and utilizing kafka apis to achieve the same .But I am not sure how to register hbase schema with kafka ?

warrior · ‎01-07-2020

I have staging area and data loaded in staging is in avro format . can I create ORC file from avro file format directly without creating avro table in hive ? as data in already in the binary format in avro , is it possible to directly create ORC table rather than first creating DDL in hive in avro format and later insert data in ORC table from avro ?

warrior · ‎12-13-2019

@Shelton thanks for your reply ! this cross realm works when we want to setup cross realm between AD and MIT KDC server , but when your edge node is in different realm and datanodes are in different realm how will you setup corss realm ? my problem is when AD users will login edge node and get kerberos ticket which will be communicated to namenode and namenode will gice delegate token to authenticate further , but as this case datanodes are in not in the same realm as edge node hence same delegate token won't work . how do i setup cross realm between edge node and datanodes . do i need to setup like following : 1) cross realm MIT KDC (edge node ) to AD . 2) cross realm MIT KDC (datanodes ) to AD. I have got confused with cross realm ideas though link provided you MIT KDC to AD trust I do understand but I am not able to get solution when edge node and data nodes both are in different real . thanks

warrior · ‎12-12-2019

I have edge node in different realm and data node is in different real , kerberos is integrated with Active Directory (AD) . As per my understanding if realm is same for all the nodes in the machine in that case user logs in edge node gets kerberos ticket as kerberos is integrated with AD . When user will submit its job ticket is transerred to namenode and authetication will take place once authenticated namenode will use grant delegate token . with the help of delegate token in all other nodes job will continue to work . But as in this case datanodes are in different realm though user gets delegate token when user logs in edge node but authentication will fail in data node as its realm is not correct . What is solution in this case ?

warrior · ‎10-11-2019

I am confused with hbase snapshot cloning mechanism . snapshot cloning in hbase allows creation of new table from existing snapshot taken for some other table , but any modification to newly created table does not have any impact on either snapshot or original table for what snaphsot was created . below is example What I am trying to justify on above statement : suppose we have hbase table tab , and it has two records cf1:fname ='Anurag' cf2:Iname ='Mishra' . .Now with this structure I have created snapshot of this table with name tabsnapshot . As snapshot clone operation allows creating new table let's assume I have created new table with name newtab . Hence now newtab table with same structure and data as tab table has been created , when I update record in the newtab table it does not make any difference in the actual table neither on snapshot . How is it possible when there is no data movement still any update on the new table does not update data for original table it does update only for itself but no disturbance in the original table . As per the functionality wise this should be the only case new table created from the snapshot should not disturb other table , but snapshot operation does not involve data copy or movement and does not affect region server how come update on data does not make any difference on original hbase table >

warrior · ‎04-21-2019

I am new to cloudera , earlier I was working on Hortonworks hadoop distribution . I am installing cloudera manager following its documentation page , but while setting database(MYSQL ) with cloudera manager I am facing below error : root@instance-1 etc]# /usr/share/cmf/schema/scm_prepare_database.sh postgresql scm scm scm JAVA_HOME=/usr/java/jdk1.6.0_31 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.6.0_31/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/util/PSQLException : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: com.cloudera.enterprise.dbutil.DbCommandExecutor. Program will exit. --> Error 1, giving up (use --force if you wish to ignore the error)

warrior · ‎04-01-2019

there are three kafka brokers , in the cluster but one of the broker is not available in the zookeeper . In the cluster there are two role groups one is default role group where there are two kafka brokers and other one with other name with one broker available . What could be issue that lead to unavailibilty of one kafka broker in zookeeper.

warrior · ‎09-07-2018

There are very old files under /spark2-history and because of this spark2 web ui is getting stuck on loading history , I have gone thorough some of the documentations where it is mentioned setting below property will solve the issue .Please find below property : spark.history.fs.cleaner.enabled spark.history.fs.cleaner.maxAge spark.history.fs.cleaner.interval would these property be same for the spark2 as well ? also I do not see these property already existing , and in spark docs default property is mentioned there . https://spark.apache.org/docs/latest/monitoring.html

warrior · ‎09-06-2018

I am getting error when i try to access oozie workflow manager view : java.lang.RuntimeException: java.lang.NullPointerException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3036) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218) at org.apache.ambari.server.view.ViewURLStreamProvider.getHttpURLConnection(ViewURLStreamProvider.java:239) at org.apache.ambari.server.view.ViewURLStreamProvider.getInputStream(ViewURLStreamProvider.java:216) at org.apache.ambari.server.view.ViewURLStreamProvider.readFrom(ViewURLStreamProvider.java:103) at org.apache.ambari.server.view.ViewURLStreamProvider.readAs(ViewURLStreamProvider.java:117) at org.apache.ambari.server.view.ViewURLStreamProvider.readAsCurrent(ViewURLStreamProvider.java:131) at org.apache.oozie.ambari.view.AmbariIOUtil.readFromUrl(AmbariIOUtil.java:45) at org.apache.oozie.ambari.view.OozieDelegate.readFromOozie(OozieDelegate.java:152) at org.apache.oozie.ambari.view.OozieDelegate.consumeService(OozieDelegate.java:118) at org.apache.oozie.ambari.view.OozieDelegate.consumeService(OozieDelegate.java:111) at org.apache.oozie.ambari.view.OozieProxyImpersonator.handleGet(OozieProxyImpersonator.java:453) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at I have set below property in custum-oozie : oozie.service.ProxyUserService.proxyuser.ambari-server-AADSPRD.hosts=* oozie.service.ProxyUserService.proxyuser.ambari-server-AADSPRD.groups=* oozie.authentication.type=kerberos I followed below link for the resolving the same but still I am facing the issue : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-views/content/wfm-kerberos-setup.html

warrior · ‎03-16-2018

@Aditya Sirna Thanks Aditya , but is there a way instead of rest api , clients can be installed with the help of ambari .

warrior · ‎03-16-2018

I want to install clients for the spark and spark2 in all the nodes services are already installed . can some one help me to the procedure how to achieve the above purpose . when I click on the service tab and then I see services option but I can not proceed further ., since all services are pre-selected . I want to install clients in all nodes . please help me on this Thanks in advacne

warrior · ‎01-05-2018

@Jay Kumar SenSharma Qorg/apache/ambari/server/controller/AmbariServer.class class in file:/usr/lib/ambari-server/ambari-server-2.6.0.0.267.jar!/org/apache/ambari/server/controller/AmbariServer.class 05 Jan 2018 05:06:29,311 INFO [main] Configuration:3048 - Reading password from existing file 05 Jan 2018 05:06:29,320 INFO [main] Configuration:3542 - Hosts Mapping File null 05 Jan 2018 05:06:29,321 INFO [main] HostsMap:60 - Using hostsmap file null 05 Jan 2018 05:06:29,793 INFO [main] ControllerModule:224 - Detected POSTGRES as the database type from the JDBC URL 05 Jan 2018 05:06:29,842 INFO [main] ControllerModule:545 - Searching package org.apache.ambari.server for annotations matching [interface org.apache.ambari.server.EagerSingleton, interface org.apache.ambari.server.StaticallyInject, interface org.apache.ambari.server.AmbariService] 05 Jan 2018 05:06:30,128 INFO [main] ClasspathScannerUtils:54 - Checking package [org.apache.ambari.server] for binding candidates. 05 Jan 2018 05:06:30,664 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService 05 Jan 2018 05:06:30,671 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.AlertNoticeDispatchService 05 Jan 2018 05:06:30,673 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.AmbariServerAlertService 05 Jan 2018 05:06:30,674 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.CachedAlertFlushService 05 Jan 2018 05:06:30,827 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.MetricsRetrievalService 05 Jan 2018 05:06:30,829 INFO [main] ControllerModule:587 - Registering service class org.apache.ambari.server.state.services.RetryUpgradeActionService 05 Jan 2018 05:06:30,849 INFO [main] ControllerModule:615 - Searching package org.apache.ambari.server.notifications.dispatchers for dispatchers matching interface org.apache.ambari.server.notifications.NotificationDispatcher 05 Jan 2018 05:06:30,930 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.SNMPDispatcher 05 Jan 2018 05:06:30,937 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.EmailDispatcher 05 Jan 2018 05:06:30,944 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AlertScriptDispatcher 05 Jan 2018 05:06:30,945 INFO [main] ControllerModule:655 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AmbariSNMPDispatcher 05 Jan 2018 05:06:30,946 INFO [main] ControllerModule:679 - Searching package org.apache.ambari.server.checks for classes matching class org.apache.ambari.server.checks.AbstractCheckDescriptor 05 Jan 2018 05:06:31,111 INFO [main] ControllerModule:715 - Registered pre-upgrade check class org.apache.ambari.server.checks.HostsMasterMaintenanceCheck 05 Jan 2018 05:06:31,111 INFO [main] ControllerModule:715 - Registered pre-upgrade check class org.apache.ambari.server.checks.ServicesMaintenanceModeCheck tcp6 0 0 :::8080 :::* LISTEN 2710/java service iptables status : ● iptables.service - IPv4 firewall with iptables Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled) Active: inactive (dead) [root@namenode ~]# telnet 104.197.146.171 8080 Trying 104.197.146.171... telnet: connect to address 104.197.146.171: Connection timed out [root@namenode ~]# telnet namenode.c.famous-augury-181309.internal 8080 Trying 10.128.0.2... Connected to namenode.c.famous-augury-181309.internal. Escape character is '^]'. nOte : I am using google cloud platfrom vm instance to install cluster. [root@namenode ~]# telnet namenode.c.famous-augury-181309.internal 8080 Trying 10.128.0.2... Connected to namenode.c.famous-augury-181309.internal. Escape character is '^]'. if i open ambari web ui replacing ip address with hostname it shows : This site can’t be reached namenode.c.famous-augury-181309.internal’s server IP address could not be found. Search Google for namenode famous augury 181309 internal 8080 ERR_NAME_NOT_RESOLVED

warrior · ‎01-04-2018

I have started ambari server but when i open ambari web ui it shows me connection time out . I ran telnet command : telnet ambari_ipaddress 8080 trying .... telnet: connect to address Connection timed out I am beginner to hadoop please help me out Thanks in advance

warrior · ‎11-20-2017

hi , amabri -server status i have cehcked already it is in running state. when I try to access amabri web ui it shows me some DNS problem and if i repalce with ip it gives error as time out or more time taken, I have installed ambari server on machine that ip address i am taking as amabri server access.

warrior · ‎11-20-2017

Hi, i am new to all of this,, i have created my instances on google cloud but i after ambari server setup , i tried to access amabri through UI but i am not able to access

Online	Offline
Last Visited	‎03-30-2020 06:50 AM

Date Registered	‎04-01-2019 05:49 AM
Last Visited	‎03-30-2020 06:50 AM
Total Messages Posted	20

pig script failed to validate: org.apache.pig.back...

Unknown host exception on nameservice

variable [hbase] cannot be resolved

oozie is not picking up hbase common jar

kafka schema registry with HDP cluster

reading hbase table from spark and load into kafka...

creation of hive table in ORC format from avro fil...

Re: how does kerberos authentication work when edg...

how does kerberos authentication work when edge no...

Hbase snapshot cloning

while setting database connection with mysql faci...

one kafka broker not available in zookeeper

spark2 even logs cleaning issue

oozie workflow manager view error

Re: installation of clients in the cluster

installation of clients in the cluster

Re: ambari web ui not accessible

ambari web ui not accessible

Re: not able to access ambari ui

not able to access ambari ui