About Alexraj84

Alexraj84 · ‎06-20-2017

Perfect..saved my life!!! For cloudera user's like me, if you are using quickstart vm then issue the below command to get it working cd <zeppelin_dir> ./bin/zeppelin-daemon.sh stop sudo chown -R cloudera:cloudera webapps ./bin/zeppelin-daemon.sh start

Alexraj84 · ‎05-29-2017

I am loading hbase data into pig and the relation has a map of values. The values in the map are not common in each record. What I mean is, in some records maps, values will be present which may not be available in other records. Here is the sample.. <code>grunt>dump result; 06:177602927,[cloud_service#true,wvilnk#true,cmpgeo#true,cmplnk#true,webvid_standard#true,criteria_search#true,typeahead_search#true,aasgbr#true,lnkmin#false,aasdel#true,aasmcu#true,aasvia#true,lnkalt#false,aastlp#true,cmpeel#true,aasfsc#true,aasser#true,aasdhq#true,aasgbm#true,gboint#true,lnkupd#true,aasbig#true,webvid_basic#true,cmpelk#true] 06:177927527,[cloud_service#true,wvilnk#true,cmpgeo#true,cmplnk#true,webvid_standard#true,criteria_search#true,typeahead_search#true,aasgbr#false,lnkmin#false,aasdel#false,aasmcu#false,aasvia#false,lnkalt#false,aastlp#true,cmpeel#true,aasfsc#false,aasser#false,aasdhq#true,aasgbm#false,gboint#true,lnkupd#true,aasbig#false,webvid_basic#true,cmpelk#true,blake#true] I want to convert each map into individual records like below <code>177602927,cloud_service,true 177602927,wvilnk,true 177602927,cmpgeo,true 177602927,cmpgeo,true Would flatten work here as the tuples of records has uncommon values? In the second record blake#true is available where as in the first record it isn't.

Alexraj84 · ‎05-29-2017

If I create a plain target table would I be able to alter the table later? What I mean is, create a plain table and then alter the table to add the buckets? If yes, how?

Alexraj84 · ‎05-29-2017

I am trying to create a table with Bucketing from another table using select but it fails. Here is my query create table tmp CLUSTERED BY (key) INTO 256 BUCKETS as select * from link_table limit 10; And I get the below error FAILED: SemanticException [Error 10068]: CREATE-TABLE-AS-SELECT does not support partitioning in the target table link_table is already bucketed. I have also set the property to enforce bucketing. I am not sure if bucketing is supported with CTAS. Is there a way I can get this working?

Alexraj84 · ‎03-10-2017

"Issue Fixed" I talked with my DevOps later and found that the classpath for Java was not set in few datanodes in the Cluster. This was stopping the shell action to invoke the JVM at those datanodes. After fixing the Classpath, the job ran successfully

Alexraj84 · ‎03-09-2017

That is carriage return from the log while copying the content. I have created the JAVA_PATH accordingly.

Alexraj84 · ‎03-09-2017

I got the below error after those changes mentioned by you. <a href="http://10.241.1.164:8888/filebrowser/view=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/hadoop-hdfs/bin/hdfs%3A">/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hadoop-hdfs/bin/hdfs:</a> line 309: <a href="http://10.241.1.164:8888/filebrowser/view=/usr/java/jdk1.7.0_67/bin/java%3A">/usr/java/jdk1.7.0_67/bin/java:</a> No such file or directory /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hadoop-hdfs/bin/hdfs: line 309: exec: <a href="http://10.241.1.164:8888/filebrowser/view=/usr/java/jdk1.7.0_67/bin/java%3A">/usr/java/jdk1.7.0_67/bin/java:</a> cannot execute: No such file or directory ./test.sh: line 30: <a href="http://10.241.1.164:8888/filebrowser/view=/usr/java/jdk1.7.0_67/bin/java%3A">/usr/java/jdk1.7.0_67/bin/java:</a> No such file or directory

Alexraj84 · ‎03-09-2017

I am running a java program from a shell script through Oozie and I get the below error java: command not found When I run the shell script from the edge node I do not find any issues and the java class runs without any error and I get the desired output also. However it is the oozie job that fails to run the java command. All other actions in oozie are executed porperly but when it encounters the java line, it throws the afore said error. I understand that all the nodes in the Hadoop cluster will have Java installed, then why do I get this error? Below is the java command that I have in my shell script ... ... java -cp $LOCAL_DIR/libs/integration-tools.jar com.audit.reporting.GenerateExcelReport $LOCAL_DIR/input.txt $LOCAL_DIR/ ... Please provide your thoughts.

Alexraj84 · ‎03-07-2017

After a tireless research on the internet I was able to crack the solution for the issue. I have added a configuration to use the metastore server for the Hive job and it worked. Here is what I did to the Hive action. .... <hive xmlns='uri:oozie:hive-action:0.2'> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>hive.metastore.uris</name> <value>thrift://10.155.1.63:9083</value> </property> </configuration> <script>${dir}/gsrlQery.hql</script> <param>OutputDir=${jobOutput}</param> </hive> .... Note: replace the hive metatore ip accordingly if you are trying to fix a similar problem. To get the metastore details check the hive-site.xml file located in /etc/hive/conf dir. Credit: MapR

Alexraj84 · ‎03-07-2017

I get an error while running an Oozie workflow with Hive queries. Here is the workflow <workflow-app xmlns='uri:oozie:workflow:0.5' name='reporting_W_errorAuditHiveQueryExe'> <start to="hive_report_fork"/> <fork name="hive_report_fork"> <path start="hiveGSRLfile"/> <path start="hiveNGSRLfile"/> <path start="hiveNGsrlRAfile"/> </fork> <action name="hiveGSRLfile"> <hive xmlns='uri:oozie:hive-action:0.2'> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <script>${dir}/gsrlQery.hql</script> <param>OutputDir=${jobOutput}</param> </hive> <ok to="joining"/> <error to="joining"/> </action> <action name="hiveNGSRLfile"> <hive xmlns='uri:oozie:hive-action:0.2'> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <script>${dir}/nongsrlQuery.hql</script> <param>OutputDir=${jobOutput}</param> </hive> <ok to="joining"/> <error to="joining"/> </action> <action name="hiveNGsrlRAfile"> <hive xmlns='uri:oozie:hive-action:0.2'> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <script>${dir}/nongsrlRAQuery.hql</script> <param>OutputDir=${jobOutput}</param> </hive> <ok to="joining"/> <error to="joining"/> </action> <join name= "joining" to="Success"/> <action name="Success"> <email xmlns="uri:oozie:email-action:0.1"> <to>${failureEmailToAddress}</to> <subject>Success</subject> <body> The workflow ${wf:name()} with id ${wf:id()} failed [${wf:errorMessage(wf:lastErrorNode())}]. </body> </email> <ok to="end" /> <error to="fail" /> </action> <action name="failure"> <email xmlns="uri:oozie:email-action:0.1"> <to>${failureEmailToAddress}</to> <subject>Failure</subject> <body> The workflow ${wf:name()} with id ${wf:id()} failed [${wf:errorMessage(wf:lastErrorNode())}]. </body> </email> <ok to="end" /> <error to="fail" /> </action> <kill name="fail"> <message>Workflow failed</message> </kill> <end name="end"/> </workflow-app> And here is the oozie properties file oozie.wf.application.path=${deploymentPath}/workflows/errorAuditHiveQueryExe.xml deploymentPath=/user/amin/deploy_178 jobTracker=localhost:8032 nameNode=hdfs://nameservice1 dir=${deploymentPath}/data-warehouse/temp failureEmailToAddress=amin@dnb.com jobOutput=${dir} oozie.use.system.libpath=true Here is the error I get: FAILED: SemanticException [Error 10072]: Database does not exist: testnamespace Intercepting System.exit(10072) Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10072] However the namespace exist and I can query the tables inside it. What could be wrong here? Please help.

Alexraj84 · ‎02-10-2017

I have a HBase table that has data in Json format. I have a peculiar requirement where I need to add a new attribute to the existing one. Here is a column that I have queried from the HBase table COLUMN CELL c:601 timestamp=1, value={"value": "000000005295", "id": 601, "name": "ID", "type": "ATOMIC", "hash": "4 998c929c977272828456cd83aa0fc19635e674a61a67c96e6ccd226d3296db7", "attrs": {"timestamp": "2016-11-30T0919 27Z", "code": "805", "uuid": "c6d82240-87fd-469a-86a7-e8b1e5d1da4b", "source": "online", "appV ersion": "1.14.40.7", "cutoff": "1", "exportedAt": "20161020T144804Z"}, "globalAttrs": {"source": "online ", "timestamp": 1480497567000, "exportedAt": 1476974884000, "countryCode": 805, "appVersion": "1.14.4 0.7", "uuid": "c6d82240-87fd-469a-86a7-e8b1e5d1da4b", "index": null, "itemId": null, "publish": true}} For each column we have attributes denoted as attrs . Currently there are seven attributes and I want to add a new attribute called delete with value as yes . After altering the column it should like the below one. COLUMN CELL c:601 timestamp=1, value={"value": "000000005295", "id": 601, "name": "ID", "type": "ATOMIC", "hash": "4 998c929c977272828456cd83aa0fc19635e674a61a67c96e6ccd226d3296db7", "attrs": {"timestamp": "2016-11-30T0919 27Z", "code": "805", "uuid": "c6d82240-87fd-469a-86a7-e8b1e5d1da4b", "source": "online", "appV ersion": "1.14.40.7", "cutoff": "1", "exportedAt": "20161020T144804Z", "delete" : "yes"}, "globalAttrs": {"source": "online ", "timestamp": 1480497567000, "exportedAt": 1476974884000, "countryCode": 805, "appVersion": "1.14.4 0.7", "uuid": "c6d82240-87fd-469a-86a7-e8b1e5d1da4b", "index": null, "itemId": null, "publish": true}}

Alexraj84 · ‎02-09-2017

Where can I check property is Turned on and How?

Alexraj84 · ‎02-07-2017

I am not sure about Hue but from the terminal it can be fixed by exporting the correct oozie server. Use this command to export the oozie url. export OOZIE_URL=http://someip:11000/oozie To get this oozie url you need to use hue to connect you cluster and navigate to Workflows where you can find a tab called oozie . Inside this you should see gauges where a lot of properties will be listed. Look for the property oozie.servers.

Alexraj84 · ‎11-29-2016

Here is the ruby function. def count_table(tablename, args = {}) table = @shell.hbase_table(tablename) # Run the scanner scanner = table._get_scanner(args) count = 0 iter = scanner.iterator # Iterate results while iter.hasNext row = iter.next count += 1 end # Return the counter return count end

Alexraj84 · ‎11-24-2016

I have a custom function written in Ruby to count the number of columns. But I don't know how to add this function in HBase shell? Is there a way to add any custom functions of UDFs in HBase shell?

Alexraj84 · ‎10-18-2016

Still does not work. How can I add another filter (KeyOnlyFilter()) along with this filter? I have tried with AND but did not work.

Alexraj84 · ‎10-18-2016

When I query the HBase table using your query I get the same row again and again. I changed the chance argument to different floats as well but no luck.

Alexraj84 · ‎10-17-2016

I am querying HBase to get a set of key and value using the limit clause. Here is the query hbase(main):015:0> scan 'sample_table', {FILTER => "KeyOnlyFilter()",TIMESTAMP => 11, LIMIT => 2} and I get some output. If I repeat the same query I get the same output. What I need is a different output every time I execute the query. In Hive we can use rand() to get a different output every time we query with LIMIT clause. I want to know if we have something similar in HBase?

Alexraj84 · ‎09-13-2016

Thanks for your reply however I wanted to run it in a cluster directly and not in local mode.

Alexraj84 · ‎09-12-2016

I am using Eclipse to build spark applications and every time I need to export the jar and run it from the shell to test the application. I am using a VM running CDH5.5.2 quick start vm in it. I have my eclipse installed in my windows (Host) and I create spark applications which is then exported as Jar file from Eclipse and copied over to Linux(Guest) and then, I run the spark application using spark-submit. This is very annoying sometimes because if you miss something in your program and the build was successful, the application will fail to execute and I need to fix the code and again export the Jar to run and so on. I am wondering if there is a much simpler way to run the job right from eclipse(Please note that I don't want to run spark in local mode) where the input file will be in HDFS? Is this a better way of doing? What are the Industry standards that are followed to develop. test and deploying spark applications in Production?

Alexraj84 · ‎09-08-2016

We have HBase tables where the data is in in Binary Avro format. To query the HBase tables easily, everytime we are creating Hive Tables and then query it, which is a tedious process as the tables are taking a long time for creation and also AdHoc tasks goes for a toss. As Phoenix or Drill can be a best alternative to Hive, a question arouse in me, whether they will support the Avro file format. Will Phoenix or Drill make it in my case?

Alexraj84 · ‎08-08-2016

Hi Amit, I am using 1.6.0 that is installed in quick start vm from CDH 5.5.7

Alexraj84 · ‎08-05-2016

Great, that fixes the problem but another arises. scala> sqlContext.createDataFrame(sc.textFile("/user/cloudera/data/fruit_fixedwidth.txt").map { x => getRow(x) }, schema) <console>:31: error: package schema is not a value sqlContext.createDataFrame(sc.textFile("/user/cloudera/data/fruit_fixedwidth.txt").map { x => getRow(x) }, schema) ^ I am really getting excited now. What is the schema all about in this context?

Alexraj84 · ‎08-05-2016

Thanks Arun however I have a problem while creating getRow function. Not sure what exactly does it refers to. Here is the error <console>:26: error: not found: type Row def getRow(x : String) : Row={ ^ <console>:32: error: not found: value Row Row.fromSeq(columnArray)

Alexraj84 · ‎08-04-2016

I have a fixed length file ( a sample is shown below) and I want to read this file using DataFrames API in Spark(1.6.0). 56 apple TRUE 0.56 45 pear FALSE1.34 34 raspberry TRUE 2.43 34 plum TRUE 1.31 53 cherry TRUE 1.4 23 orange FALSE2.34 56 persimmon FALSE23.2 The fixed width of each columns are 3, 10, 5, 4 Please suggest your opinion.

Alexraj84 · ‎07-27-2016

I am using the vm version of the distribution in my laptop and hence there is no chance of any file deletion.

Alexraj84 · ‎07-27-2016

From the same directory I can fire spark-shell and that works without any hassle. Moreover I am using the VM and I am sure that the use I am using has the root access.

Alexraj84 · ‎07-27-2016

I am using CDH 5.5.7 quick start VM which has Spark 1.6.0 running. I am trying to connect to the spark-sql cli but it fails. According to this link by issuing spark-sql command I should be able to enter the cli but I get the below error. [cloudera@quickstart ~]$ spark-sql -bash: spark-sql: command not found I have also tried the below and getting the same error [cloudera@quickstart ~]$ ./bin/spark-sql -bash: ./bin/spark-sql: No such file or directory Any help is much appreciated.

Alexraj84 · ‎07-27-2016

Thanks for the reply. Let me try this will come back with a reply.

Alexraj84 · ‎07-27-2016

Hi @Sindhu, thanks for your followup. I was able to get the stats using the below query. Thank you again for your effort. analyze table sampletable partition (year) compute statistics noscan;

Online	Offline
Last Visited	‎06-01-2016 10:16 AM

Date Registered	‎05-31-2016 11:52 AM
Last Visited	‎06-01-2016 10:16 AM
Total Messages Posted	89
Total Kudos Received	14

Re: Oozie-- java: command not found - shell action

Re: FAILED: SemanticException [Error 10072]: Datab...

Re: How to select the last table from a list of hi...

Re: Warning message in Hive output after upgrading...

Re: How to check the namenode status?

Re: Zeppelin UI returns 503 error

Pig-Splitting maps into lines that has different m...

Re: Bucketing does now work with CTAS

Bucketing does now work with CTAS

Re: Oozie-- java: command not found - shell action

Re: Oozie-- java: command not found - shell action

Re: Oozie-- java: command not found - shell action

Oozie-- java: command not found - shell action

Re: FAILED: SemanticException [Error 10072]: Datab...

FAILED: SemanticException [Error 10072]: Database ...

How to add a new attribute to a Column in HBase Ta...

Re: How to update hive table

Re: hadoop - oozie job configuration App directory...

Re: Is there a way to add a custom function from H...

Is there a way to add a custom function from HBase...

Re: How to Randomly display the rows in HBase?

Re: How to Randomly display the rows in HBase?

How to Randomly display the rows in HBase?

Re: Is there a easy way to test spark applications...

Is there a easy way to test spark applications wit...

Does Apache Phoenix or Drill support Binary Avro f...

Re: how to read fixed length files in Spark

Re: how to read fixed length files in Spark

Re: how to read fixed length files in Spark

how to read fixed length files in Spark

Re: Unable to connect to spark-sql cli

Re: Unable to connect to spark-sql cli

Unable to connect to spark-sql cli

Re: Comparing hive tables with Spark.

Re: SHOW TABLE STATS Statement throws error in Hiv...