Use the event desearlizer  You can use BlobDeserializer  - if  you want to parse the whole file inside one event. or You can use Line  -  one event per line of text input.    Refer the link  https://flume.apache.org/FlumeUserGuide.html#event-deserializers ... View more

Yes, host IP. ... View more

In HDFS, you tell it which disk to use and it will fill up those disk. There is the ability to set how much space on those disks are reserved for non-DFS data but it doesn't actual prevent the disk from being filled up. The issue at hand is that the smaller disk will fill up faster, so at some point they will not allow any more write operations and the cluster will have no way to balance itself out. This causes issue with HDFS replication and placement, along with hotspotting in MR, Spark, and any other jobs. Say for instance if you primarily operation on the last days worth of data for 80% of your jobs. At some point you will hit critical mass were those jobs, are running mostly on the same set of nodes.   You could set the reserved non-DFS space to different values using Host Templates in CM.  This would then at least give you a warning when you are approaching filling up the smaller disk, but then at that point the larger disk would have free space that isn't getting used. This is why it is strongly encourage to not have different hardware. If possible upgrade the smaller set. A possible option would be to use Heterogeneous storage. With it you can designate pools, so the larger nodes would be in one pool and the smaller in the other. Each ingestion point would need to set which pool it would use and you can set how many replicas go to each. This is a big architectural change those and should be carefully reviewed to see if it benefits your use case(s) in anyway.   So, simply, use the same hardware or you will more than likely run into issues. ... View more

Thanks for your reply. Please find Hive ORC related config below. hive.exec.orc.skip.corrupt.data=false hive.exec.orc.default.row.index.stride=10000 ... View more

1. Check on the user perimission of the jar file.  2.when you add driver manully do this  --driver = org.postgresql.Driver  3. Please try the below jdbc driver version.    Optionally you can extract the jar file see the if there is    org.postgresql.Driver.class    curl -L 'http://jdbc.postgresql.org/download/postgresql-9.2-1002.jdbc4.jar' -o postgresql-9.2-1002.jdbc4.jar $ sudo cp postgresql-9.2-1002.jdbc4.jar /var/lib/sqoop/   ... View more

Hi Csguna,   Doing this for one or two columns is fine but doing for more than 200 columns is where I am stuck   Sqoop import ... --map-column-java id=String,value=Integer ... View more

@csguna   It is authorized_key   nothing to do with hdfs here. so it is user:linux group (instead of hdfs group)     ... View more

On the setting changes, stats, as stated will help with counts as that info is precalculates and stored in the metadata. The CBO and stats also help a lot with joins. It is possible that the OS cache is more to do with the improvement if this was a subsequent run with little activity. You could look at Hive on Spark for better consistent performance. Set hive.execution.engine = spark; On the times, the big impact between job submission and start is the the scheduler. That is a deep topic. It is best if you read up on them and review your settings and ask any specific questions that come up, preferably in a new topic. The other factor, not captured on the job stats, is the time it takes to return the results to the client. This will vary depending on the client and there isn't much to do about it. In general small result sets can be handle by the hive CLI. You can increase the client heap if needed. Otherwise use HS2 connections like beeline or HUE. ... View more