About heta_desai

heta_desai · ‎04-26-2019

Hi, There are many etl tools in market. I did some research on Pentaho Data Integraion Tool. In which kettle is for ETL. How can we compare kettle with hadoop ? Does kettle(any ETL tool) replaces hadoop ? When do we need ETL tool on the top of hadoop ? Please someone help me to clear my doubts. Thanks, Heta

heta_desai · ‎04-11-2019

Hi, I have cluster of 4 nodes, each having 16 GB RAM and 2 core, 50 GB disk storage on master and 25 GB on each worker node. one is master node and 3 are worker nodes. i have stored 16 GB data on HDFS. I am trying to process this 16 GB data using PIG in map reduce mode. When i submit a pig job, the application master is assigned on one worker node from three, job starts its execution and gets stuck at 94%. After that i came to know that the user(application) cache(7.9 GB) is full on another 2 nodes. Is there any configuration to automatically clear the cache for running job ? Thanks, Heta

heta_desai · ‎03-27-2019

Hi, Suppose i have a server having 128 GB RAM and 32 cores. I am creating 4 VMs on this server each having 32 GB RAM and 8 cores. Will each VM works as worker node ? does it gives parallel data processing performance ? what will be the performance difference between 4 VMs as worker nodes on single server and 4 physical servers as worker nodes having same capacity(32 GB RAM and 8 cores) ? Can anyone help me to clear this doubts ? Thanks, Heta

heta_desai · ‎03-27-2019

Hi, I want to build a cluster in which everyday data input is 2 TB, which i want to process on daily basis. how can i decide howmany worker nodes we should have in a cluster and what hardware configuration for each node ? Suppose i have a server having 128 GB RAM and 32 cores. I am creating 4 VMs on this server each having 32 GB RAM and 8 cores. Will each VM works as worker node ? does it gives parallel data processing performance ? what will be the performance difference between 4 VMs on single server and 4 servers having same capacity(32 GB RAM and 8 cores) ? Thanks, Heta

heta_desai · ‎03-26-2019

Hi, I have two tables in hive, both table have a common field which i want to compare and based on that i want to put a value. a = load 'telecom.employee' using org.apache.hive.hcatalog.pig.HCatLoader(); b = load 'telecom.dept' using org.apache.hive.hcatalog.pig.HCatLoader(); c = foreach a {dp = foreach b generate (a.dept == b.deptname ? b.deptid:0); generate a.id as empid, dp as deptid,a.dept as deptname;}; whil executing statemnt c it throws an error: a = load 'telecom.employee' using org.apache.hive.hcatalog.pig.HCatLoader(); 2019-03-26 13:47:19,133 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <line 13, column 28> expression is not a project expression: (Name: ScalarExpression) Type: null Uid: null) Is there another way to compare hive table fields ? Thanks, Heta

heta_desai · ‎03-26-2019

Hi, In LDAP we create users and sync them with cluster, Ambari and Ranger, so same user can be used everywhere. I have configured my cluster for kerberos. How can i achieve centralized user authentication in kerberos enabled cluster as i mentioned above we can do in LDAP ? I do not have LDAP server. I want to do Authentication and Authorization both using Kerberos only. Thanks, Heta

heta_desai · ‎03-25-2019

@Jonathan Sneep After downloading jce8 i am not able to unzip it. It shows below error: Archive: jce_policy-8.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jce_policy-8.zip or jce_policy-8.zip.zip, and cannot find jce_policy-8.zip.ZIP, period. I am using this doc.

heta_desai · ‎03-25-2019

Hi, I want to provide user authentication and resource/service authority to my cluster. I do not have LDAP server. I want to use Kerberos for my cluster security. How can I configure and start using kerberos without LDAP ? I have followed this link but i don't what is next step ? How can i integrate kerberos with Ambari ? For now i am trying on sandbox HDP 2.6.5. Thanks, Heta

heta_desai · ‎03-19-2019

Hi, I am using cloudbreak for Hortonworks Data Platform on Azure. How can i secure my cluster from hackers as well as unauthorized users ? There are 100s of user for my cluster. All users should not have admin access. How can i give limited access of hadoop cluster to users according to their role ? How to manage Authentication of user and authorization of services in hadoop cluster ? As per my research Kerberos and LDAP can be used but i am confused that these are alternative of each other or both works together ? also what will be the role of Ranger ? Can anyone helps me to secure my cluster ? Thanks, Heta

heta_desai · ‎03-18-2019

Hi, I have excel workbook in which there are 5 sheets. I want to read this workbook using PIG. How can I do this ? Thanks, Heta

heta_desai · ‎03-16-2019

Hi, I am using cloudbreak data platform on azure. My company has own authorization tool for user authorization. Is it possible to use this tool with cluster created on cloudbreak for user security management ? Same concern with HDinsight on Azure ? Thanks, Heta

heta_desai · ‎03-14-2019

I thin docker is running. But there is no directory named start_scripts under /root didrectory.

heta_desai · ‎03-14-2019

@Jay Kumar SenSharma I am doing ssh to root@#localhost 2122 using putty. but still i am not able to do that.

heta_desai · ‎03-14-2019

@Jay Kumar SenSharma When i try to ssh on port 2122 it shows connection refused.

heta_desai · ‎03-14-2019

@Jay Kumar SenSharma I have same issue and not able to solve it. I have raised the question. Please help me.

heta_desai · ‎03-14-2019

Hi, I am using sandbox HDP 2.6.5 in virtualbox. I have configured Kylin 2.6.0. Kylin started successfully but i am not able to open Kylin Web UI. So i need to add Port 7070 to sandbox and for that i am following this article which is for HDP 2.5. As defined in the article to add port i have to edit start_sandbox.sh file under /root/start_scripts directory. But there is no directory named start_scripts under /root. I am facing same issue for HDP 3.0. So, how can i add new port in HDP 2.6.5/3.0 ? Thanks, Heta

heta_desai · ‎03-14-2019

@Jay Kumar SenSharma I can successfully login to Amabri in Incognito Mode in Gogle Chrome. I am running one HDP sandbox instance at a time, but i am using different versions so i think the issue i was facing is due to caching. Thank you so much.

heta_desai · ‎03-14-2019

@Jay Kumar SenSharma In browser when i try to login to Ambari using login credentials it shows "server error".

heta_desai · ‎03-14-2019

Hi, I have 3 versions of HDP in oracle virtual box. The versions are 2.5, 2.6.5 and 3.0. When i start VM for HDP 3.0 the Ambari UI works fine. But after that if i start VM for HDP lower versions 2.5 or 2.6.5 i can not open Ambari UI. It shows Server error. Based on the requirement i need to use different versions of HDP. How can i resolve this issue ? Thanks, Heta

heta_desai · ‎03-05-2019

#TO CHANGE KYLIN UI PORT 1. cd /usr/local/apache-kylin-2.6.0-bin/tomcat/conf 2. Open server.xml Change port in Connector Port argument: As shown below Connector Port 7070 is assigned. You can give another port instead of 7070. <Connector port="7070" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="7443" compression="on" compressionMinSize="2048" noCompressionUserAgents="gozilla,traviata" compressableMimeType="text/html,text/xml,text/javascript,application/javascript,application/json,text/css,text/plain" /> Thanks

heta_desai · ‎02-26-2019

@Gorka Zárate When i did same configuration on Cloudbreak for HDP on azure, Kylin UI is working.

heta_desai · ‎02-25-2019

Hi, I am using Hortonworks sandbox HDP 2.6.5. I have installed apache kylin 2.6.0-bin on sandbox. Kylin has started successfully. Kylin default Port 7070 is already forwarded in virtualbox network. Below is the output when I checked whether PORT 7070 is listening or not using below command: netstat -tupln | grep 7070 OUTPUT: ------- tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 4341/java In Kylin.properties file i have added below property for Web UI: kylin.web.dashboard-enabled=true Still i'm not able to open Kylin UI. I'm not getting what configuration is missing. Can anyone help me solve this issue. Thanks, Heta

heta_desai · ‎02-22-2019

@Gorka Zárate No, kerberos is not enable.

heta_desai · ‎02-21-2019

Hi, I am using Hortonworks sandbox HDP 2.6.5. I have installed apache kylin 2.6.0-bin on sandbox. Kylin has started successfully. But i am not able to open Kylin Web Interface. Port 7070 is registered and when i checked using below command it shows port is open: netstat -tupln | grep 7070 tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 4092/java When i enter http://localhost:7070/kylin in chrome it shows page can't be found. How can i resolve this ? Thanks

heta_desai · ‎02-16-2019

Hi, I need to design complete architecture for CDR analysis. Below is the basic flow. There are different telecom companies. They will give us data in any flat file format. The data can be batch data or real time This raw data needs to store somewhere for processing. After data gets processed we need to generate report. Everyday minimum 1TB data will be generated. How much storage will need. and how much computing resource will need to process data. How to decide cluster capacity Can I use Azure Blob storage for storage ? Or HDFS is compulsary I need help to design complete workflow for above requirement with which tools i can use at each step.

heta_desai · ‎02-15-2019

Hi, I have create cluster on Cloudbreak for hortonworks data platform on Azure. Initially i have created cluster with 1 master node and 1 worker node. The Hive was running perfectly from Hive view as well as Hive shell. After that when i added 2 additional worker nodes to the cluster, In Ambari Hive view is not opening and throws below error: Cannot open a hive connection with connect string jdbc:hive2://xx.xx.xx.xx.xx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2;hive.server2.proxy.user=admin When I try to insert data into Hive table from Hive shell(CLI) the job gets stuck. Query ID = cloudbreak_20190215071908_8e0a113e-2bfb-488a-95cd-8bacdb45abbd Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1550214881831_0001) How to resolve this error ? Thank you.

heta_desai · ‎02-14-2019

Hi, I am using cloudbreak for Hortonworks data platfom on Azure. Initially i have created cluster with 1 master node and 2 worker nodes. Now i am adding 2 more worker nodes. So, after cluster resize to 4 worker nodes, does data in hdfs gets distributed among 4 worker nodes ? also does data processing performance impacts after adding 2 more nodes ? I do not have much knowledge about how Auto-scalling works. Another thing, what is the performance difference between data processing on worker nodes and compute nodes ? Thanks you.

heta_desai · ‎02-12-2019

Hi, I have launched cloudbreak for hortonworks data platform on azure. Created cluster on top of cloudbreak. I want to connect Hive database with SSRS. So I am trying to connect using Hortonworks Hive ODBC Driver. But i am not able to connect. While entering configuration parameter i am confuse which value needs to enter like hostname, port, authentication mechanism, username-password and transport mode. can anyone help me to configure ODBC driver for Hive ? Thank you

heta_desai · ‎02-12-2019

When i use WASB as storage, while creating cluster i need to have Master node and compute nodes only right ? No need to have worker node as i am using WASB not HDFS ?

heta_desai · ‎02-12-2019

@lnardai It worked for me. What does it mean "wrap it as a service" ?

Online	Offline
Last Visited	‎04-30-2019 05:17 AM

Date Registered	‎03-21-2017 05:34 AM
Last Visited	‎04-30-2019 05:17 AM
Total Messages Posted	195
Total Kudos Received	6

Re: how to open druid UI ?

Re: after deleting user and group from openLDAP se...

Re: ambari-server sync-ldap: Exiting with exit co...

difference between hadoop and kettle etl

how to clear user cache for running pig job

horizontal and vertical scaling cluster difference...

how to decide the size of cluster

how to compare two different hive table fields usi...

centralized user authentication in kerberos

Re: kerberos configuration without AD/LDAP

kerberos configuration without AD/LDAP

Hadoop cluster authentication/authorization for us...

How Pig reads multiple sheets available in single ...

Does HDP supports third party authorization ?

Re: How to add new port to sandbox HDP 2.6.5 ?

Re: How to add new port to sandbox HDP 2.6.5 ?

Re: How to add new port to sandbox HDP 2.6.5 ?

Re: How can I add a new port?

How to add new port to sandbox HDP 2.6.5 ?

Re: can not open Ambari UI

Re: can not open Ambari UI

can not open Ambari UI

Re: Changing Apache Kyline port

Re: Unable to open Kylin Web interface

configuration for apache Kylin web interface

Re: Unable to open Kylin Web interface

Unable to open Kylin Web interface

CDR Analysis with Cloudbreak for HDP

can not open Hive view in Amabri

how auto-scalling works in HDP - cloudreak on azur...

How to connect Hortonworks Hive ODBC drver

Re: purpose of master, worker and compute node in ...

Re: Cloudbreak url is not working after VM restart