NiFi doesnot YET have a CDC kind of processor - as-in the processor that would look into logs to determine the changed rows in a given time span. However, there is a processor "QueryDatabaseTable" which essentially returns the rows that have been updated since last retrieval - but the problem with this processor is that it scans the whole table to find the changes values, and this could pose a performance bottle neck if your source table is really big. Here is the documentation for QueryDatabaseTable - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/ (esp pay attention to the property 'Maximum value columns') Here is the blog that walks you through setting up a CDC using QueryDatabaseTable - https://hortonworks.com/blog/change-data-capture-using-nifi/ Lastly, specific to your question, should you go down this route, below are the nifi processors that you probably need: QueryDatabaseTable ConvertAvroToJson PublishKafka PutHiveQL / PutHiveStreaming As an alternate to this you may also look into Attunity which has a CDC capability Hopefully this helps, if it does, please remember to upvote and 'accept' the answer. Thank you! ... View more

Working further with our support team and customer, it was determined that this issue was coming mostly from Postgres side. The reason being the ticket caching was not enabled for the PG side, and the customer is currently working on enabling the same. This document link should talk about enabling caching from Postgres - http://jpmens.net/2012/06/23/postgresql-and-kerberos/ As far as the above question on multiple requests on the same session goes - Yes, Hive metastore does caching my default, and the multiple commands executed within the same HS2 session is translated to a single auth request due to caching at the HMS level ... View more

@ripunjay godhani I also answered to your another post on changing the block size and why you should refrain from doing so. So here I will simply address the other ways you can overcome this small file problem. The primary questions that need to be asked when picking up the data archive strategy are How am I going to access this data? How often am I going to access this archived data? Am I going to be bound by some stringent SLAs? The answer to these questions will lead you to figure out if you need some kind of low density spinning disk or some kind of SSDs from hardware perspective, or am I going to put this data to HBase (memory intensive) or just a plain old file. You put into Hbase when you have a very stringent SLAs - like sub-second response, and have the luxury of clustering lot of nodes with high memory (RAM) - this doesn't seem to be the case from your explanation above. So here are my two suggestions (in order of preference): Put data into Hive. There are ways to put xml data into hive. At a very dirty level you have an xpath udf to work on xml data in Hive, or you can package it luxuriously by converting xml to avro and then using serde to map the filelds to column names. (let me know if you want to go over this in more detail and I can help you there) Combine bunch of files, zip it up and upload to hdfs. This option is good, if your access is very cold (once in a while) and you are going to access the files physically (like hadoop fs -get) Let me know if you have further questions. Lastly, if you find this answer to be helpful, please upvote and accept my answer. Thank you!! ... View more

@ripunjay godhani Here is the general answer - reducing the default block size will result in creation of too many blocks which results an overhead on Name Node. By architecture each node (in newer architecture it will be each storage type per node - but that conversation is for a different time) on the Hadoop cluster will report a storage report and block report back to the Name Node, which will then be used when retrieving/accessing the data at a later time. So, as you would imagine this will increase the chattiness between name node and data node, as well as increase the meta data on the Name node iteself. Also, when you start hitting 100's of millions of file range, then your Name node will start filling up the memory and may result in going through a major garbage collection, which is a stop the world operation and may result in your whole cluster being down for few minutes.. there are ways around this - like increasing the memory size of NN or changing the gc, but none of those are economical or easy. These are all the down sides of reducing the block size - or even a small file problem, in general. And now coming to your specific use case - why do you think you have so many smaller files? Is there a way you can merge multiple of those into a larger file? I know one of my customer had similar issue while storing tick symbols - they mitigated this by combining the tick data on a hourly basis. Another customer had a source file FTP-ed that is quite small and they mitigated by gzipping bunch of those file into a really large one. Also, archiving data to Hive is another option. The bottom line being the small file issue on the hadoop must be viewed as a combination of technical + business problem, and you will be best off by looking to ways to eliminate this situation from business standpoint as well. Simply playing around the block size is not going to give you the most mileage. Lastly, if you felt this answer to be helpful, please upvote and accept the answer. Thank you! ... View more

@vpemawat If you are not using log4j: If you are looking to delete the files for good then there is not many options available other than rm -rf; however there are few tweaks that you can do to make it faster you can perhaps run multiple rm scripts in parallel (multiple threads) In order to do this, you should be able to logically separate the log files either by folder or name format Once you have done that, you can run multiple rm commands in background like something below nohup rm -fr app1-2016* > /tmp/nohup.out 2>&1 & nohup rm -fr app1-2015* > /tmp/nohup.out 2>&1 & If using log4j: You should probably be 'DailyRollingFileAppender" with 'maxBackupIndex' - this will essentially limit the max file size of your log and then purge the older contents. More details here: http://www.codeproject.com/Articles/81462/DailyRollingFileAppender-with-maxBackupIndex Outside of this, you should consider the below 2 things for future use cases Organize the logs by folder (normally broken down like /logs/appname/yyyy/mm/dd/hh/<log files> Have a mechanism that will either delete the old log files, or archive it to a different log archive server Hopefully this helps. If it does, please 'accept' and 'upvote' the answer. Thank you!! ... View more

Go to your Ambari >> Kafka >> Configs and look for the port the Kafka broker listens on If it is HDP sandbox 2.4, it will most probably be on 6667 and therefore you should be running the below command instead ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic test1 Let me know if this works. Else, post with the proper exception and we can look deeper. If this answer helps you, please don't forget to upvote / accept this answer ... View more

Before I answer the question specifically, let me address (based on my research) the fault tolerance for each of the components within Storm: 1) Nimbus - A stateless daemon which sits on master node which deploys the job (Topology) and keeps track of it. Now there are two scenarios. First, If Nimbus goes down after you submit the topology it will not have any adverse affect on current topology as it is running on worker nodes (and not on the master node), Now if you have kept this process under supervision the process will restart and when your nimbus comes back as it is fail-fast it will retrieve all the meta information of all the active topologies on the cluster from Zookeeper and start tracking them again. Second, If Nimbus goes down before you submit the topology you will simply have to restart it. 2) Supervisor - This daemon is responsible to keep track of worker processes (JVM Process) on the node he sits and coordinate the state with Nimbus through Zookeeper. If this daemon goes down your worker process will not be affected and it will keep on running unless it doesn't get crashed, once it comes back (due to supervisord or monit) it will collect the state from Zookeeper and resume tracking the worker processes. If timeout occurs Nimbus will reschedule the topology on different worker node. 3) Worker Processes (JVM Processes) - These container processes actually execute your topology’s components (Spouts + Bolts), If it goes down Supervisor will simply restart them on different ports(on same worker node) and if it is out of ports it will notify Nimbus and then it will reschedule the process on different worker node. 4) Worker Node (Supervisor + Worker Process) - During this scenario Nimbus will stop receiving the heartbeats (due to timeout) from the Worker Node and then the Nimbus will simply reassign the work to different Worker Node(s) in the cluster. 5) Zookeeper (Zk) - Okay! From all the above you might have infered that all the state gets stored on Zk, Hmmh! What if it goes down or can it go down? Zk is again not a single node process, it has its own cluster and the state stored in Zk is constantly replicated, so even if a single Zk node goes down, the new node will be elected as a leader which will start communicating with Apache Storm. Now, going back to the specific question: When a supervisor with 4 slots (ports) go down The very first thing the Nimbus will try to do is to restart the processes on the SAME worker node on the available ports. And for those processes that it does not have port, it will be reassigned to a different worker node - so yes, it will increase the executor threads on these worker nodes. And from design perspective itself, you should not necessarily counter for redundant ports as Nimbus is designed to take care of this by either restarting the processes on that port or by re-distributing among other worker nodes. ... View more

@gkeys The mergecontent processor has 2 properties that I normally use to determine the output file size Minimum number of entries Minimum group size For your question as how do i increase the file size to reach a desired file size (say 1gb)? - Set the minimum group size to the size that you would like (i.e 1 gb) AND set the minimum number of entries to 1. This will merge the content to the 1 gb before it writes out to the next processor Can you clarify a little more about your other question on how do i double the size on existing setting?? - do you mean double the size of incoming file? - this will be direct. Just set the minimum number of entries to 2 and minimum group size to 0 b ... View more