@Arunkumar Dhanakumar Since replication factor is by default 3, using Benjamin's approach for sizing, 10 GB will be per data node. That is very simplified and assumes big files. Additional to Benjamin's response, let's keep in mind that the block size matter. The calculation presented above is rough-order of magnitude and it does not account that data could be as many small files that may not fill the blocks. For example, your block size could be 256 MB and you stored 100 files of 1 KB. That could take 100 x 256 MB. Also, compression plays a role here. It depends the type of compression used, etc. Also, if you data is stored as ORC, you could have your 10 GB data reduced at 3 GB data even without compression. ... View more

@Leonid Zavadskiy You are dealing with this issue: https://issues.apache.org/jira/browse/HADOOP-3733 As a workaround you can set first the fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties then the URI would be S3:/mybucket/dest. Putting things on the command line is not very secure anyway. ... View more

@Rita McKissick Logging output is written to a target by using an appender. If no appenders are attached to a category nor to any of its ancestors, you will get the following message when trying to log: log4j:No appenders could be found for category (some.category.name). log4j:Please initialize the log4j system properly. Log4j does not have a default logging target. It is the user's responsibility to ensure that all categories can inherit an appender. This can be easily achieved by attaching an appender to the root category. You can find info on how to configure the root logger ( log4j.rootLogger ) in the log4j documentation, basically adding something as simple as this at the beginning of the file: log4j.rootLogger=debug, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=... This should clear those WARN messages you get on startup (make sure you don't already have an appender named stdout ; also be careful of what level you give the root logger, debug will be very verbose and every library in your app will start writing stuff to the console). In a nutshell, you're missing the log4j.properties or log4j.xml in your classpath. You can also bypass this by using BasicConfigurator.configure(); But beware this will ONLY log to System.out and is not recommended. You should really use one of the files above and write to a log file. A very simple example of log4j.properties would also be #Log to Console as STDOUT log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%-5p%c %3x-%m%n #Log to file FILE log4j.appender.file=org.apache.log4j.DailyRollingFileAppender log4j.appender.file.File=logfile.log log4j.appender.file.DatePattern='.'yyyy-MM-dd log4j.appender.file.append=true log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%-5p%c %3x-%m%n #RootLogger log4j.rootLogger=INFO, stdout, file ... View more

Multi-Polygon example: SELECT st_astext(ST_MultiPolygon('multipolygon (((0 0, 0 1, 1 0, 0 0)), ((2 2, 2 3, 3 2, 2 2)))')) from YourTable LIMIT 1; ... View more

@ripunjay godhani Just knowing the amount of data (250 GB) is insufficient for capacity planning. Intake and output is also necessary, data processing requirements. Workloads, concurrency, expected response time, resiliency and availability are also important factors. Those determine what CPU/RAM/Network/DiskIO you need. Account for how much data will be read from the disk vs from memory based on your SLA and design. This is all art of estimation. NN is best to use reliable hardware and RAID is a good option. Anyhow, it is usually good to begin small and gain experience by measuring actual workloads during a pilot project. We recommend starting with a relatively small pilot cluster, provisioned for a “ balanced ” workload. For pilot deployments, you can start with 1U/machine and use the following recommendations: Two quad core CPUs | 12 GB to 24 GB memory | Four to six disk drives of 2 terabyte (TB) capacity. Even you have only 250 GB, that is multiplied by a replication factor of 3, you need temporary space, plus room to grow. Multiple spindles will also give enough IOPS for disk operations. Don't think only in matter of "I need 250 GB storage". The minimum requirement for network is 1GigE all-to-all and can be easily achieved by connecting all of your nodes to a Gigabyte Ethernet switch. In order to use the spare socket for adding more CPUs in future, you can also consider using either a six or an eight core CPU. For more check the following references: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_cluster-planning-guide/content/balanced-workloads-deployments.1.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Sys_Admin_Guides/content/ch_clust_capacity.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Sys_Admin_Guides/content/ch_hbase_io.html If you find this response helpful, accept it as a best answer. ... View more