Hi @Ya ko, Why not considering the new ORC. https://www.slideshare.net/Hadoop_Summit/orc-improvement-in-apache-spark-23-95295487 Then you will get the best performance when querying from hive. And yes you have to define your table with all heh fields. The slide 20 show how to specify the new orc library, you will have to just all the location setting to point where your data will be stored in hdfs. Michel ... View more

Hi @rajat puchnanda, Based on your example, you are trying to do a "join". Nifi is not an ETL tool but more a flow manager, it allow to move data accros system and to do some very simple transformation like csv to avro. You should not do computation or join with Nifi. For you usecase it would be better to use another tools like hive, spark,... Best regards, Michel ... View more

Hi @Zack Riesland, Indeed increasing the the number of bucket will increase the parallelism to write to hdfs (then to the disk). If I was you I would have a look at the disk/iops usage, if you try to load a lot of data and you have only one disk it can take a long time. generally its recommended to have multiple disk per node to avoid iops congestion. Whats the exact query that you are doing to insert the data? does it contain some casting? whats the size of your data? Also as a good optimisation is to use ORC table and not avro. you the loading face it should not change a lot but when you are going to query your data that will make the difference, Michel ... View more

Hi @rajat puchnanda, If by merging you means doing an union, you can use the processor mergecontent if the two csv have the same structure. Best regards, Michel ... View more

HI, Usually timeout happen because the cluster is undersized or no dedicated node for hbase or the ingestion is so quick that hbase need to do a lot of split of region. - Do you manage a lot of data with hbase? if yes, idd you pre-split your table? - If i was you I would also have a look to the cpu, memory and io disk usage. If you dont have anydedicated nodes for hbase other hadoop component like spark, hive, etc can have an impact. As a general best practice, you should have dedicated node for hbase with enough cpu and several disk Mest regards, Michel ... View more

Hi @Ashish Singh, Can you show the command that you used to submit your spark application? Michel ... View more

Hi @Ennio Sisalli, Before running your query that save the result in HDFS, can you try to set the following parameter: set hive.cli.print.header=true; Best regards, Michel ... View more

Hi @Simon Jespersen, Did you restarted the nifi once you added the new nifi property or when you modified the file? Nifi need to be restarted in order to load the new parameter. Michel ... View more

Hi @Gayathri Devi, I can't give you more idea than in my previous comment. Because It depends on the system specification that you have, the other load on the cluster, size of the data. size of each lines, etc... The % that I gave you is based on benchmark that I made in previous project and blog/forum that I read in the past. The best that you can do is a test. I would recommend you to do a test with compression and another without to see the impact that it have on your environnement. Moreover, be careful with Hive on top of hbase. You might have bad performance because often it start a full scan of the hbase table, which is an expensive operation. Michel ... View more

Hi @Gayathri Devi, The operation of compression/decompression will increase the cpu load of around 5-10 % For the memory, is will decrease the disk space by around 70%, more over the size on the disk will be smaller then you will need less iops. Because of that you should see a general improvement of your performance. Michel ... View more

Hi @Gobi Subramani, Is it normal that in your code?: String node = "x.x.x.x:6667"; I think it should be an ip or hostname. Michel ... View more

@Hemant, You said that you were able to interact with hdfs from the host that has nifi. How did you get the ticket to interact wit hdfs? Are you able to create a ticket with the user and keytab mentionned in the configuration or the processor? (Just to be sure that the key tab is working well ... View more

@Hemant for the user do you have this structure: hive/FQDN@MY_REALM ? ... View more

For info, I think that once you configure that property, you need to restart nifi ... View more

Hi @Hemant, Did you configure the nifi.kerberos.krb5.file in your nifi.properties? ... View more

Hi @Hemant, No Nifi doesn't need to be kerberized but you need to install the kerberos client on the os (where nifi is installed) in order to be able to request a ticket. Michel ... View more

Hi @Rahul Gupta, Did you managed this? if yes can you accept the answer? 🙂 Thanks, Michel ... View more

hi @Piyush Chauhan, Did you need more info? If not can you accept the answer? 🙂 Michel ... View more

Hi @Nagesh Gollapudi, Can you do a screenshot of the configuration of your different processor in Nifi? 🙂 Michel ... View more

Hi @Piyush Chauhan, In the hortonworks stack you have the following: - HDFS ACL: you manage by your self the right access on HDFS, it can be very quickly become a huge amount of work if you have a lot of user. More over it protect the access only to hdfs - HDFS TDE (encryption): This is a feature of HDFS that encrypt in a completly transparent manner all the file in a folder. This provide strong protection to any data store on HDFS. It can be from Hive, HBase, etc... - Ranger: The most interesting part! It's a tool that help you to configure that to manage the access to the different hadoop component. For example, you can create policy for a specific group of user define in your enterprise AD to have only read or write or access denied to HDFS folder. It can also restrict access to: Hive, Hbase, Solr, kafka, etc... Ranger is really powerfull and help to manage the security by reduce the time needed to do it. More over it provide an audi feature, if it's enable you can see who had access to what and when (you can also see if the permission was denied). - Knox: you can see Know as a kind of proxy. Every request from every user will be sent to the Knox server and he will redirect the request to the correct service/server. It's useful if you don't want that your user know the network topology of your cluster and if you don't want that they have a direct access to the server hosting the service (like Hive database) I would recommend to use the combination ranger+ Knox and if you have the need to use TDE enrcyption. Best regards, Michel ... View more