About PentaReddy

PentaReddy · ‎09-13-2018

PentaReddy · ‎09-13-2018

I tried with both the formats (Avro and Parquet), but no luck

PentaReddy · ‎09-13-2018

If we use the query, the problem is we might be doing the function to replace to only one column and so times these kind of characters can come in other columns as-well. I feel, at runtime with query, these kind of characters can come in from any column. I feel using an argument, will apply to entire table instead of just for a specific column. It would be nice if we can have argument applied to entire table. Thanks Praveen

PentaReddy · ‎09-13-2018

How to escape ^M character in data while doing sqoop import from the Oracle Database

PentaReddy · ‎08-17-2017

I am getting error which i am trying to check if the hdfs directory or not and i am trying check it through oozie fs action and below is the code, however, i am getting error. Appreciate any help on this. </action> <decision name="deleteFrompraveenPostCondition"> <switch> <case to="Export"> ${fs:exists(/dev/praveen/test/*)} </case> <default to="statusLog"/> </switch> </decision> --------------------------------- ERROR ---------------------------------------------------------- Encountered "/", expected one of [<INTEGER_LITERAL>, <FLOATING_POINT_LITERAL>, <STRING_LITERAL>, "true", "false", "null", "(", ")", "-", "not", "!", "empty", <IDENTIFIER>] Appreciate any help on this ......

PentaReddy · ‎08-17-2017

I am getting error which i am trying to check if the hdfs directory or not and i am trying check it through oozie fs action and below is the code, however, i am getting error. Appreciate any help on this. </action> <decision name="deleteFrompraveenPostCondition"> <switch> <case to="Export"> ${fs:exists(/dev/praveen/test/*)} </case> <default to="statusLog"/> </switch> </decision> --------------------------------- ERROR ---------------------------------------------------------- Encountered "/", expected one of [<INTEGER_LITERAL>, <FLOATING_POINT_LITERAL>, <STRING_LITERAL>, "true", "false", "null", "(", ")", "-", "not", "!", "empty", <IDENTIFIER>] Appreciate any help on this ......

PentaReddy · ‎08-17-2017

I am getting error which i am trying to check if the hdfs directory or not and i am trying check it through oozie fs action and below is the code, however, i am getting error. Appreciate any help on this. </action> <decision name="deleteFrompraveenPostCondition"> <switch> <case to="Export"> ${fs:ex ists(/dev/praveen/test/*)} </case> <default to="statusLog"/> </switch> </decision> --------------------------------- ERROR ---------------------------------------------------------- Encountered "/", expected one of [<INTEGER_LITERAL>, <FLOATING_POINT_LITERAL>, <STRING_LITERAL>, "true", "false", "null", "(", ")", "-", "not", "!", "empty", <IDENTIFIER>] Appreciate any help on this ......

PentaReddy · ‎07-21-2017

Hive - i would like to calculate percentage of column and based on the percentage i would like to load the data into another table(if the percentage of n is less 20%) or else not to load colA y y y n ------------------ Output: -- This is what i am expecting y 80% n 20%

PentaReddy · ‎05-17-2017

Can you eloberate more details as i am facing same issue and when i checked it i see the java-json.jar in the oozie shared lib path, However, i don't see it in the sqoop-client/lib path on the gateway.

PentaReddy · ‎04-27-2017

Replication is for the data-node failure, when Human deletes the data, data will lost where-ever it resides be it on any number of nodes. and this is moved into trash and if needed we can get it back within certain interval time.

PentaReddy · ‎04-25-2017

Thanks and Yes, i can re-write it, however i am looking options if there is any way to get it back and when i drop the table,immediately the commit will occur to the meta-store, which might be causing for not recovering the hive table schema back. Any other alternatives options ?

PentaReddy · ‎04-25-2017

I have a Hive external table and Unfortunately, the schema of the table got dropped and i want to get back the schema. Is there any ways to get it back ? I do understand that Hdfs is a file system, However, try to see if there are any possibilities.

PentaReddy · ‎03-10-2017

Thanks, This is done on the folder level of encryption, however i am looking on the fields level of encryption rather than entire file. I know ranger has this feature, however, this only help us on the hive column level of encryption when i query it, but eventually, when i look at the raw file, i could still see the sensitive data.

PentaReddy · ‎03-09-2017

I have a requirement, where i need to encrypt certain sensitive data before landing/ingestion into Hadoop. Just want to understand, how Hadoop process these kind of encrypted data(be it in hive or pig or any map-reduce). Do we need to write specific programs? to read this kind of encrypted files in hadoop or do we need to set any parameters on hive table or pig session to read these this kind of encrypt data ? Any ideas/thoughts or suggestions ?

PentaReddy · ‎02-16-2017

I found a solution to export this kind of data to any RDBS in the form of UTF8 or any other character set by giving the specific character set after the database/host name.

PentaReddy · ‎02-15-2017

Yes, It is displaying the special characters with good reading format after adding serilization encoding property, however,while i am exporting the data to teradata with sqoop statement as using a connection manager i getting as non-readable characters in teradata. Attached is the screen shot(teradat.png). I suspect sqoop is not reconizing the special chracters correctly or do i need to use any specific teradata jar's while exporting the data ? I have attached the ingested data(after-ingestion-data-into-hadoop.png) and the showed the data in hive after adding encoding property(after-adding-encoding-to-hive-table.png), where as the same data is not same in Teradata. I would like to see the same type of characters in teradata as-well. Any Help appreciated. )

PentaReddy · ‎02-14-2017

I have requirement to handle file which contains special characters (like trademarks, non-utf and so on..)

PentaReddy · ‎02-11-2017

Could you let me know if the data is not in Quto's, How to Handle it and below is the example column 1| column 2 first|second|last In the above example the first|second are actually one column. Could you let me know how to handle if the data is not in quoto's and if the delimiter is part of the data. Any suggestion or help is appreciated.

PentaReddy · ‎02-09-2017

Thanks and Yah .. Open Csv serde will do it.However, i am looking if there any other alternatives.

PentaReddy · ‎02-08-2017

In hive, One of my column data contains Pipe as the part of the data('|'), however, while exporting data from this table, we need to export the data as the pipe('|') as the delimiter between each fields, How to handle if the delimiters as part of the data while creating the flat file from the hive table.

PentaReddy · ‎02-08-2017

Just curious to find..... Why Sqoop will not allow us to create an external table while sqooping the data from RDBMS ?

PentaReddy · ‎02-08-2017

Here is work around, which i have implemented. For an external table, If you are trying to drop a partition and as-well would like to delete the data. This can be achieved as below. 1. Alter external table as internal table -- by changing the TBL properties as external =false 2. Drop the partitions -- when you drop the partitions, data pertained to the partitions will also be dropped as now this table is managed table 3. Alter back the table as external=True. By doing this, there more controlled on what we are deleting and drop the partitions rather than using hadoop rm command

PentaReddy · ‎02-08-2017

In a flat file, i have certain keywords, which are sensitive, i would like to identify these sensitive keywords row by row. These keywords could appear in any column of the flat file. Appreciate any help. Either in Hive or Pig anything is fine.

PentaReddy · ‎02-07-2017

In hive,One of my column data contains Pipe as the part of the data('|'),however, while exporting data from this table,we need to export the data as the pipe('|') as the delimiter between each fields,How to handle if the delimiters as part of the data while creating the flat file from the hive table.

PentaReddy · ‎01-27-2017

While i was trying to drop a partition based on date range, i unable to achieve it and below is what i am trying to do alter table X drop partition(partdate <= (select max(partdate) as from Y)); I am getting error, while i am executing above query -- cannot recognize input near '(' 'select' 'max' in constant Looks like the alter statement accepts only the constant value rather then a sub-query. Basicaly - Alter table drop partition ( partdate <= date -- and this date need to fetch from another table). Any help is appreciated.

PentaReddy · ‎01-26-2017

I would like to create a hive table for an xml files

PentaReddy · ‎01-25-2017

I have an external table which is created with partitions and i would like to delete/drop few partition along with data as i no longer require it. Appreciate any suggestions.

PentaReddy · ‎01-24-2017

I have overnight batch job, which loads the data into a hive table and the loading of data is done through dynamic partitions. Is there any way to find the number of paritions created by each batch job run ?

PentaReddy · ‎01-24-2017

I have an external table which is created with partitions and i would like to delete/drop few partition along with data as i no longer require it. Appreciate any suggestions.

PentaReddy · ‎01-24-2017

I have batch job, which run over the night and it create the dynamic partitions. How can i identify the number of partitions created by the batch job.

Online	Offline
Last Visited	‎10-25-2018 06:53 PM

Date Registered	‎12-21-2016 06:37 PM
Last Visited	‎10-25-2018 06:53 PM
Total Messages Posted	74
Total Kudos Received	5

Re: How to delete/drop a partition of an external ...

Re: Flume Authentication credentials are missing

How to escape ^M character in data while doing sqo...

Re: How to escape ^M character in data while doing...

Re: How to escape ^M character in data while doing...

How to escape ^M character in data while doing sqo...

Oozie fs action - Hdfs directory exists or not

Oozie fs action

Getting error when Oozie shell fs action - hdfs fi...

Hive - i would like to calculate percentage of col...

Re: Running oozie sqoop action fails with "Launche...

Re: Hive - external table schema got dropped unfor...

Re: Hive - external table schema got dropped unfor...

Hive - external table schema got dropped unfortuna...

Re: How to process encrypted data in Hadoop ?

How to process encrypted data in Hadoop ?

Re: How to handle trademark symbols in hive ?

Re: How to handle trademark symbols in hive ?

How to handle trademark symbols in hive ?

Re: How to handle delimiters, if they are part of ...

Re: How to handle delimiters, if they are part of ...

How to handle delimiters, if they are part of the ...

Why Sqoop will not allow us to create an external ...

Re: How to delete/drop a partition of an external ...

how to identify certain keywords from a flat file,...

How to handle delimiters, if they are part of the ...

Need to drop a hive partition based on date range ...

How to create hive table for a xml file

How to delete/drop a partition of an external tabl...

How to find the number of partition created for a ...

How to delete/drop a partition of an external tabl...

How to identify dynamic partitions created by a ba...