Member since
06-07-2016
923
Posts
322
Kudos Received
115
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3301 | 10-18-2017 10:19 PM | |
3668 | 10-18-2017 09:51 PM | |
13355 | 09-21-2017 01:35 PM | |
1366 | 08-04-2017 02:00 PM | |
1780 | 07-31-2017 03:02 PM |
09-16-2020
03:27 PM
I believe this will fail if you stop your job today and run it tomorrow.. now will change to other day and you will miss the data...
... View more
07-28-2020
11:31 PM
"I highly recommend skimming quickly over following slides, specially starting from slide 7. http://www.slideshare.net/Hadoop_Summit/w-235phall1pandey" This slide is not there at the path
... View more
07-19-2020
07:37 AM
Here we have listed a few ETL tools both, traditional and Open source you can have a look at them and see for yourself which one suits your use case. 1. Panoply: Panoply is the main cloud ETL supplier and data warehouse blend. With 100+ data connectors, ETL and data ingestion is quick and simple, with only a couple of snaps and a login among you and your recently coordinated data. In the engine, Panoply is really utilizing an ELT approach (instead of conventional ETL), which makes data ingestion a lot quicker and progressively powerful, since you don't need to trust that change will finish before stacking your data. What's more, since Panoply fabricates oversaw cloud data warehouses for each client, you won't have to set up a different goal to store all the data you pull in utilizing Panoply's ELT procedure. On the off chance that you'd preferably utilize Panoply's rich arrangement of data gatherers to set up ETL pipelines into a current data warehouse, Panoply can likewise oversee ETL forms for your Azure SQL Data Warehouse. 2. Stitch: Stitch is a self-administration ETL data pipeline. The Stitch API can reproduce data from any source, and handle mass and gradual data refreshes. Stitch additionally gives a replication motor that depends on various techniques to convey data to clients. Its REST API underpins JSON or travel, which empowers programmed recognition and standardization of settled report structures into social constructions. Stitch can associate with Amazon Redshift engineering, Google BigQuery design, and Postgres design - and incorporates with BI apparatuses. Stitch is normally intended to gather, change and burden Google examination data into its own framework, to naturally give business bits of knowledge on crude data. 3. Sprinkle: Sprinkle is a SaaS platform providing ETL tool for organisations.Their easy to use UX and code free mode of operations makes it easy for technical and non technical users to ingest data from multiple data sources and drive real time insights on the data. Their Free Trial enables users to first try the platform and then pay if it fulfils the requirement. Some of the open source tools include 1. Heka: Heka is an open source programming framework for elite data gathering, investigation, observing and detailing. Its principle part is a daemon program known as 'hekad' that empowers the usefulness of social occasion, changing over, assessing, preparing and conveying data. Heka is written in the 'Go' programming language, and has worked in modules for contributing, disentangling, separating, encoding and yielding data. These modules have various functionalities and can be utilized together to assemble a total pipeline. Heka utilizes Advanced Message Queuing Protocol (AMQP) or TCP to transport data starting with one area then onto the next. It tends to be utilized to stack and parse log records from a document framework, or to perform constant investigation, charting and inconsistency recognition on a data stream. 2. Logstash: Logstash is an open source data handling pipeline that ingests data from numerous sources at the same time, changing the source data and store occasions into ElasticSearch as a matter of course. Logstash is a piece of an ELK stack. The E represents Elasticsearch, a JSON-based hunt and investigation motor, and the K represents Kibana, which empowers data perception. Logstash is written in Ruby and gives a JSON-like structure which has a reasonable division between inner items. It has a pluggable structure highlighting more than 200 modules, empowering the capacity to blend, coordinate and arrange offices over various information, channels and yield. This instrument can be utilized for BI, or in data warehouses with bring, change and putting away occasion capacities. 3. Singer: Singer's open source, order line ETL instrument permits clients to assemble measured ETL pipelines utilizing its "tap" and "target" modules. Rather than building a solitary, static ETL pipeline, Singer gives a spine that permits clients to interface data sources to capacity goals. With a huge assortment of pre-constructed taps, the contents that gather datapoints from their unique sources, and a broad choice of pre-fabricated focuses on, the contents that change and burden data into pre-determined goals, Singer permits clients to compose succinct, single-line ETL forms that can be adjusted on the fly by trading taps and focuses in and out.
... View more
07-07-2020
04:34 AM
Solr includes the specified file terms in an index. Indexing in Solr would be similar to creating an index at the end of a book that includes the words that appear in that book and their location, so basically we would take an inventory of the words that appear in the book and an inventory of the pages where said words appear That is, by including content in the index, we make said content available for search by Solr. This type of index, called an inverted index, is a way of structuring the information that will be retrieved by a search engine. You may find a longer answer of the way the information is stored and retrieved by solr in https://www.solr-tutorial.com/indexing-with-solr.html
... View more
06-12-2020
08:41 AM
Hello, Does COBRIX support Python ? I see only Scala api's..at https://github.com/AbsaOSS/cobrix Please advice. Thanks Sreedhar Y
... View more
03-31-2020
08:56 AM
For example I want to transform the zabbix payload from v4.0 to v4.4: Zabbix Json Payload v4.0 (INPUT) {
"hosts": [
"Host B",
"Zabbix Server"
],
"groups": [
"Group X",
"Group Y",
"Group Z",
"Zabbix servers"
],
"tags": [
{
"tag": "availability",
"value": ""
},
{
"tag": "data center",
"value": "Riga"
}
],
"name": "Either Zabbix agent is unreachable",
"clock": 1519304285,
"ns": 123456789,
"eventid": 42,
"value": 1
} the JOLT transform: [
{
"operation": "shift",
"spec": {
"hosts": {
"*": [
"hosts.[&].host",
"hosts.[&].name"
]
},
"*": "&"
}
}
] The result ( Zabbix v4.4) {
"hosts" : [ {
"host" : "Host B",
"name" : "Host B"
}, {
"host" : "Zabbix Server",
"name" : "Zabbix Server"
} ],
"groups" : [ "Group X", "Group Y", "Group Z", "Zabbix servers" ],
"tags" : [ {
"tag" : "availability",
"value" : ""
}, {
"tag" : "data center",
"value" : "Riga"
} ],
"name" : "Either Zabbix agent is unreachable",
"clock" : 1519304285,
"ns" : 123456789,
"eventid" : 42,
"value" : 1
}
... View more
02-25-2020
01:41 PM
Hi @DLEdwards,
As this thread is older and was marked 'Solved' in 2016 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity for you to provide details specific to your version of HDP, Storm and Metron that could aid others in providing a more accurate answer to your question.
... View more
11-22-2019
10:56 PM
Hi @mqureshi , you have explained beautifully. But how the replication of blocks will impact this calculation? Please explain. Regards.
... View more
11-20-2019
08:48 AM
To clarify for others, I believe the reason the following doesn't work is that it was executed in a bash shell, i.e. using a typical terminal. hive --hivevar rdate=112211 -e "select 9${hivevar:rdate}9" In double quoted strings, the '$' tells bash to treat "{hivevar:rdate}" as a variable but it isn't defined so it returns an empty string before being passed in to hive due to the -e flag. i.e. "9${hivevar:rdate}9" is evaulated to "99" even before passing it into hive. In contrast within single quotes there is no substitution in bash, so '9${hivevar:rdate}9' is passed into hive as is, so the following executes as the poster expected. hive --hivevar rdate=112211 -e 'select 9${hivevar:rdate}9'
... View more
11-06-2019
10:09 AM
In order to run a balancer on only one data node hdfs balancer -include -f <#datanode name can be specified> this would balance the data load on that particular DN.
... View more