Member since
09-18-2015
3274
Posts
1159
Kudos Received
426
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2137 | 11-01-2016 05:43 PM | |
6506 | 11-01-2016 05:36 PM | |
4147 | 07-01-2016 03:20 PM | |
7102 | 05-25-2016 11:36 AM | |
3442 | 05-24-2016 05:27 PM |
04-15-2016
09:23 AM
@Marco Gaido Marco, Take a look on this thread https://community.hortonworks.com/questions/4067/snappy-vs-zlib-pros-and-cons-for-each-compression.html I would test the same task with zlib to check the different behavior.
... View more
04-15-2016
09:09 AM
4 Kudos
Original Post
Calcite is a highly customizable engine for parsing and planning queries on data in a wide variety of formats. It allows database-like access, and in particular a SQL interface and advanced query optimization, for datanot residing in a traditional database.
Apache Calcite is a dynamic data management framework.
It contains many of the pieces that comprise a typical database management system, but omits some key functions: storage of data, algorithms to process data, and a repository for storing metadata.
Calcite intentionally stays out of the business of storing and processing data. As we shall see, this makes it an excellent choice for mediating between applications and one or more data storage locations and data processing engines. It is also a perfect foundation for building a database: just add data. Source
Tutorial https://calcite.apache.org/docs/tutorial.html
Demo:
Read DEPT and EMPS table
Create a test table based on existing csv example. Read the tutorial link to understand the model.json and schema.
In the demo, you can see that I am running explain plan on the queries and then I used smart.json to change the plan.
Watch the demo and then read the following links
model.json https://calcite.apache.org/docs/tutorial.html#schema-discovery
Query tuning https://calcite.apache.org/docs/tutorial.html#optimizing-queries-using-planner-rules
Calcite https://calcite.apache.org/
This page describes the SQL dialect recognized by Calcite’s default SQL parser.
Adapters
JDBC driver
Calcite is embedded in Drill, Hive and Kylin.
... View more
Labels:
04-01-2016
12:03 PM
@Usman Shahid See this https://www.digitalocean.com/community/tutorials/how-to-install-solr-on-ubuntu-14-04
... View more
03-30-2016
12:44 PM
1 Kudo
@Rainer Geissendoerfer See if it helps https://community.hortonworks.com/content/kbentry/1620/how-to-increase-sandbox-disk-space-virtualbox.html
... View more
03-29-2016
04:27 PM
@Vadim See if this is helpful.
... View more
03-28-2016
06:23 PM
@marksf Do this: 1) run top as root 2) locate top pid and then see what process those pid belongs to. Once you are 100% sure that mysql causing the issue and those sql are not related to hadoop or ambari then follow this
... View more
03-22-2016
09:41 PM
1 Kudo
@mkataria You can use Apache Falcon http://hortonworks.com/hadoop/falcon/ or see this https://community.hortonworks.com/articles/9933/apache-nifi-aka-hdf-data-flow-across-data-center.html
... View more
03-20-2016
11:35 AM
@Mohamed Ashiq Please see this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_RelNotes/content/ch_relnotes_v240.html HDP 2.4 comes with Apache Kafka 0.9.0 -- This is the highest version of Kafka as of now in HDP stack
... View more
03-19-2016
09:44 PM
2 Kudos
@gopal
... View more
03-19-2016
02:12 AM
4 Kudos
@Mahesh Deshmukh See this https://cwiki.apache.org/confluence/display/Hive/Tutorial It is also a good idea to bucket the tables on certain columns so that efficient sampling queries can be executed against the data set. If bucketing is absent, random sampling can still be done on the table but it is not efficient as the query has to scan all the data. The following example illustrates the case of the page_view table that is bucketed on the userid column: The following example illustrates the case of the page_view table that is bucketed on the userid column: CREATE TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
ip STRING COMMENT 'IP Address of the User')
COMMENT 'This is the page view table'
PARTITIONED BY(dt STRING, country STRING)
CLUSTERED BY(userid) SORTED BY(viewTime) INTO 32 BUCKETS
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '1'
COLLECTION ITEMS TERMINATED BY '2'
MAP KEYS TERMINATED BY '3'
STORED AS SEQUENCEFILE; In the example above, the table is clustered by a hash function of userid into 32 buckets. Within each bucket the data is sorted in increasing order of viewTime. Such an organization allows the user to do efficient sampling on the clustered column - in this case userid. The sorting property allows internal operators to take advantage of the better-known data structure while evaluating queries with greater efficiency.
... View more