Hi,   I am using hortonworks sandbox HDP 3.0.1. Trying to execute Pig script from oozie workflow but its giving below error:   Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]   ERROR V2JobServlet:517 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET http://sandbox-hdp.horton works.com:11000/oozie/v2/job/0000000-191017065434484-oozie-oozi-W?show=auditlog] error, null java.lang.NullPointerException at org.apache.oozie.DagEngine.streamJobLog(DagEngine.java:400) at org.apache.oozie.BaseEngine.streamAuditLog(BaseEngine.java:217) at org.apache.oozie.servlet.V2JobServlet.streamJobAuditLog(V2JobServlet.java:292) at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:335) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:304) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.OozieXFrameOptionsFilter.doFilter(OozieXFrameOptionsFilter.java:48) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(OozieCSRFFilter.java:62) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503) at java.lang.Thread.run(Thread.java:748)   Can anyone provide me the solution ?   Thanks, Heta         ... View more

Hi, There are many etl tools in market. I did some research on Pentaho Data Integraion Tool. In which kettle is for ETL. How can we compare kettle with hadoop ? Does kettle(any ETL tool) replaces hadoop ? When do we need ETL tool on the top of hadoop ? Please someone help me to clear my doubts. Thanks, Heta ... View more

Hi, I have cluster of 4 nodes, each having 16 GB RAM and 2 core, 50 GB disk storage on master and 25 GB on each worker node. one is master node and 3 are worker nodes. i have stored 16 GB data on HDFS. I am trying to process this 16 GB data using PIG in map reduce mode. When i submit a pig job, the application master is assigned on one worker node from three, job starts its execution and gets stuck at 94%. After that i came to know that the user(application) cache(7.9 GB) is full on another 2 nodes. Is there any configuration to automatically clear the cache for running job ? Thanks, Heta ... View more

Hi, Suppose i have a server having 128 GB RAM and 32 cores. I am creating 4 VMs on this server each having 32 GB RAM and 8 cores. Will each VM works as worker node ? does it gives parallel data processing performance ? what will be the performance difference between 4 VMs as worker nodes on single server and 4 physical servers as worker nodes having same capacity(32 GB RAM and 8 cores) ? Can anyone help me to clear this doubts ? Thanks, Heta ... View more

Hi, I want to build a cluster in which everyday data input is 2 TB, which i want to process on daily basis. how can i decide howmany worker nodes we should have in a cluster and what hardware configuration for each node ? Suppose i have a server having 128 GB RAM and 32 cores. I am creating 4 VMs on this server each having 32 GB RAM and 8 cores. Will each VM works as worker node ? does it gives parallel data processing performance ? what will be the performance difference between 4 VMs on single server and 4 servers having same capacity(32 GB RAM and 8 cores) ? Thanks, Heta ... View more

Hi, I have two tables in hive, both table have a common field which i want to compare and based on that i want to put a value. a = load 'telecom.employee' using org.apache.hive.hcatalog.pig.HCatLoader(); b = load 'telecom.dept' using org.apache.hive.hcatalog.pig.HCatLoader(); c = foreach a {dp = foreach b generate (a.dept == b.deptname ? b.deptid:0); generate a.id as empid, dp as deptid,a.dept as deptname;}; whil executing statemnt c it throws an error: a = load 'telecom.employee' using org.apache.hive.hcatalog.pig.HCatLoader(); 2019-03-26 13:47:19,133 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <line 13, column 28> expression is not a project expression: (Name: ScalarExpression) Type: null Uid: null) Is there another way to compare hive table fields ? Thanks, Heta ... View more

Hi, In LDAP we create users and sync them with cluster, Ambari and Ranger, so same user can be used everywhere. I have configured my cluster for kerberos. How can i achieve centralized user authentication in kerberos enabled cluster as i mentioned above we can do in LDAP ? I do not have LDAP server. I want to do Authentication and Authorization both using Kerberos only. Thanks, Heta ... View more

Hi, I want to provide user authentication and resource/service authority to my cluster. I do not have LDAP server. I want to use Kerberos for my cluster security. How can I configure and start using kerberos without LDAP ? I have followed this link but i don't what is next step ? How can i integrate kerberos with Ambari ? For now i am trying on sandbox HDP 2.6.5. Thanks, Heta ... View more

Hi, I am using cloudbreak for Hortonworks Data Platform on Azure. How can i secure my cluster from hackers as well as unauthorized users ? There are 100s of user for my cluster. All users should not have admin access. How can i give limited access of hadoop cluster to users according to their role ? How to manage Authentication of user and authorization of services in hadoop cluster ? As per my research Kerberos and LDAP can be used but i am confused that these are alternative of each other or both works together ? also what will be the role of Ranger ? Can anyone helps me to secure my cluster ? Thanks, Heta ... View more

Hi, I have excel workbook in which there are 5 sheets. I want to read this workbook using PIG. How can I do this ? Thanks, Heta ... View more

Hi, I am using cloudbreak data platform on azure. My company has own authorization tool for user authorization. Is it possible to use this tool with cluster created on cloudbreak for user security management ? Same concern with HDinsight on Azure ? Thanks, Heta ... View more

Hi, I am using sandbox HDP 2.6.5 in virtualbox. I have configured Kylin 2.6.0. Kylin started successfully but i am not able to open Kylin Web UI. So i need to add Port 7070 to sandbox and for that i am following this article which is for HDP 2.5. As defined in the article to add port i have to edit start_sandbox.sh file under /root/start_scripts directory. But there is no directory named start_scripts under /root. I am facing same issue for HDP 3.0. So, how can i add new port in HDP 2.6.5/3.0 ? Thanks, Heta ... View more

Hi, I am using Hortonworks sandbox HDP 2.6.5. I have installed apache kylin 2.6.0-bin on sandbox. Kylin has started successfully. Kylin default Port 7070 is already forwarded in virtualbox network. Below is the output when I checked whether PORT 7070 is listening or not using below command: netstat -tupln | grep 7070  OUTPUT: ------- tcp        0      0 0.0.0.0:7070            0.0.0.0:*               LISTEN      4341/java In Kylin.properties file i have added below property for Web UI: kylin.web.dashboard-enabled=true   Still i'm not able to open Kylin UI. I'm not getting what configuration is missing. Can anyone help me solve this issue. Thanks, Heta ... View more

Hi, I am using Hortonworks sandbox HDP 2.6.5. I have installed apache kylin 2.6.0-bin on sandbox. Kylin has started successfully. But i am not able to open Kylin Web Interface. Port 7070 is registered and when i checked using below command it shows port is open: netstat -tupln | grep 7070 tcp        0      0 0.0.0.0:7070            0.0.0.0:*               LISTEN      4092/java When i enter http://localhost:7070/kylin in chrome it shows page can't be found. How can i resolve this ? Thanks ... View more

Hi, I need to design complete architecture for CDR analysis. Below is the basic flow. There are different telecom companies. They will give us data in any flat file format. The data can be batch data or real time This raw data needs to store somewhere for processing. After data gets processed we need to generate report. Everyday minimum 1TB data will be generated. How much storage will need. and how much computing resource will need to process data. How to decide cluster capacity Can I use Azure Blob storage for storage ? Or HDFS is compulsary I need help to design complete workflow for above requirement with which tools i can use at each step. ... View more

Hi, I have create cluster on Cloudbreak for hortonworks data platform on Azure. Initially i have created cluster with 1 master node and 1 worker node. The Hive was running perfectly from Hive view as well as Hive shell. After that when i added 2 additional worker nodes to the cluster, In Ambari Hive view is not opening and throws below error: Cannot open a hive connection with connect string jdbc:hive2://xx.xx.xx.xx.xx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2;hive.server2.proxy.user=admin When I try to insert data into Hive table from Hive shell(CLI) the job gets stuck. Query ID = cloudbreak_20190215071908_8e0a113e-2bfb-488a-95cd-8bacdb45abbd Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1550214881831_0001) How to resolve this error ? Thank you. ... View more

Hi, I am using cloudbreak for Hortonworks data platfom on Azure. Initially i have created cluster with 1 master node and 2 worker nodes. Now i am adding 2 more worker nodes. So, after cluster resize to 4 worker nodes, does data in hdfs gets distributed among 4 worker nodes ? also does data processing performance impacts after adding 2 more nodes ? I do not have much knowledge about how Auto-scalling works. Another thing, what is the performance difference between data processing on worker nodes and compute nodes ? Thanks you. ... View more

Hi, I have launched cloudbreak for hortonworks data platform on azure. Created cluster on top of cloudbreak. I want to connect Hive database with SSRS. So I am trying to connect using Hortonworks Hive ODBC Driver. But i am not able to connect. While entering configuration parameter i am confuse which value needs to enter like hostname, port, authentication mechanism, username-password and transport mode. can anyone help me to configure ODBC driver for Hive ? Thank you ... View more