@anvanna  @jitendra03  That's obvious and by design, an MV rebuild will take time depending on the changes in the underlying tables the MV is built on even incremental as it has to check each table to compare whether there was a change or not. Something else also you should look at tuning the memory remember behind the scenes it's Map Reduce working and you won't expect it to run in seconds with all the different map and reduces steps.   You should have a good understanding of the MV rebuild mechanism below and take that into consideration. Hive supports incremental view maintenance, i.e., only refresh data that was affected by the changes in the original source tables. Incremental view maintenance will decrease the rebuild step execution time. In addition, it will preserve LLAP cache for existing data in the materialized view.   By default, Hive will attempt to rebuild a materialized view incrementally, falling back to full rebuild if it is not possible. The current implementation only supports incremental rebuild when there were INSERT operations over the source tables, while UPDATE and DELETE operations will force a full rebuild of the materialized view. To execute incremental maintenance, the following conditions should be met: The materialized view should only use transactional tables, either micromanaged or ACID. If the materialized view definition contains a Group By clause, the materialized view should be stored in an ACID table, since it needs to support MERGE operation. For materialized view definitions consisting of Scan-Project-Filter-Join, this restriction does not exist. A rebuild operation acquires an exclusive write lock over the materialized view, i.e., for a given materialized view, only one rebuild operation can be executed at a given time.   I would think you should trigger the rebuild during off-peak hours like when you have no users logged on and running queries in your DW I would think you should trigger the rebuild during off-peak hours like when you have no users logged on and running queries in your DWH @jitendra03   I think the initial problem of the MV rebuild failure has been achieved now we are talking of a performance problem, Since performance is a new issue unrelated to the question asked in the original subject, I kindly ask that you start a new question for performance-specific help I would be happy to help.  Asking multiple questions in the same thread makes a thread harder to follow for other users of this community.  Hope that gives you a better picture of the behind the scenes tasks   Happy hadooping ... View more

@sow    Impala does not allow binary data. What you can do is use a serialize-deserialize methodology. This means you convert your image to a String format that still contains all the information necessary to transform it back. Once you need to retrieve an image on HDFS you will need to deserialize, meaning converting the string to the original format. Found this example using Python it  would work like this:   import base64 def img_to_string(image_path): with open(image_path, "rb") as imageFile: image_string= base64.b64encode(imageFile.read()) print image_string   def string_to_img(image_string): with open("new_image.png", "wb") as imageFile: imageFile.write(str.decode('base64'))       ... View more

@anvanna  Hello there just trying to follow up on this thread, did the response help you resolve the issue or do you still need further help.   It would be a good idea to update so you get the help needed. ... View more

@saivenkatg55    I see in the hadoop-yarn-nodemanager-w0lxdhdp05.ifc.org.log errors pointing to "Unable to start NodeManager: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir/libleveldbjni-64-1-6279667856305652637.8 (Permission denied)]   My suspicion:   Please verify that /tmp on the host does not have the noexec option set. You can verify this by running /bin/mount and checking the mount options. If you are able to, remount /tmp without noexec and try starting the NodeManager again. I am sure its issue with noexec on /tmp. See my sample output [root@tokyo ~]# /bin/mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=7167976k,nr_inodes=1791994,mode=755) ....... ... systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15609) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) /dev/sda1 on /boot type ext4 (rw,relatime,data=ordered) /dev/sda5 on /opt type ext4 (rw,relatime,data=ordered) /dev/sda8 on /home type ext4 (rw,relatime,data=ordered) /dev/sda11 on /u02 type ext4 (rw,relatime,data=ordered) /dev/sda6 on /var type ext4 (rw,relatime,data=ordered) /dev/sda10 on /u01 type ext4 (rw,relatime,data=ordered) /dev/sda9 on /tmp type ext4 (rw,relatime,data=ordered) This issue occurs when the user running the Hadoop [Nodemanager start] process does not have the necessary rights and cannot generate temporary files under the /tmp directory.   Solution - Allow the user running node manager startup process read/write/execute access on /tmp - Remove the noexec parameter when mounting /tmp - Change the execution rights on /tmp. ie: sudo chmod 777 /tmp   In the /var/log/messages I  can also see Jan 2 05:14:23 w0lxdhdp05 abrt-server: Package 'ambari-agent' isn't signed with proper key Jan 2 05:14:23 w0lxdhdp05 abrt-server: 'post-create' on '/var/spool/abrt/Python-2020-01-02-05:14:22-11897' exited with 1 Jan 2 05:14:23 w0lxdhdp05 abrt-server: Deleting problem directory '/var/spool/abrt/Python-2020-01-02-05:14:22-11897'   Please edit /etc/abrt/abrt-action-save-package-data.conf change the value for OpenGPGCheck should be changed from yes to no. OpenGPGCheck = no It might also be necessary to change the value of limit coredumpsize: limit coredumpsize unlimited After editing the file restart the process with the following command: # service abrtd restart Restart the node manager and share your joy ! ... View more

@pra_big  hbase user is the admin user of hbase one connects to a running instance of HBase using the hbase shell command, located in the bin/ directory of your HBase install. Here the version information that is printed when you start HBase Shell has been omitted. The HBase Shell prompt ends with a > character. As hbase user $ ./bin/hbase shell hbase(main):001:0> All the below methods will give you access to the HBase shell as the admin user [hbase] If you have root access # su - hbase It will give you the same above If you have sudo privileges # sudo su hbase -l I don't see the reason for changing to bash or didn't I understand your question well? ... View more

@ssk26  A queueMaxAMShareDefault and maxAMShare are mutually exclusive as its overridden by maxAMShare element in each queue. Can you decrease it to queueMaxAMShareDefault or  maxAMShare to 0.1 and weight to 2.0 For the spark create the fairscheduler.xml from the fairscheduler.xml.template your path might be different due to version 3.1.x.x.x. # cp /usr/hdp/3.1.x.x-xx/etc/spark2/conf/fairscheduler.xml.template fairscheduler.xml   Please check the file permission Then set spark.scheduler.allocation.file property in your SparkConf or either by putting a file named fairscheduler.xml on the classpath. Note if no pools configured in the XML file will simply get default values for all settings (scheduling mode FIFO, weight 1, and minShare 0).   Here there are 2 default pools in fairscheduler.xml.template notably production and test using FAIR and FIFO <allocations> <pool name="production"> <schedulingMode>FAIR</schedulingMode> <weight>1</weight> <minShare>2</minShare> </pool> <pool name="test"> <schedulingMode>FIFO</schedulingMode> <weight>2</weight> <minShare>3</minShare> </pool> </allocations>   Without any intervention, newly submitted jobs go into a default pool, but jobs’ pools can be set by adding the spark.scheduler.pool “local property” to the SparkContext in the thread that’s submitting them. This is done as follows:   // Assuming sc is your SparkContext variable to pick the FAIR sc.setLocalProperty("spark.scheduler.pool", "production")   Please let me know  ... View more

@sheelstera    There is a great YARN tuning spreadsheet here that will help you calculate correctly your YARN settings. It applies to YARN clusters only, and describes how to tune and optimize YARN for your cluster   Please revert ... View more

@saivenkatg55  Sorry festive period, can you do the following. Delete old messages in /var/log/messages all that have the extension /var/log/messages.x that should leave you with only one /var/log/messages then truncate that file so you will have only new entries # truncate --size 0 /var/log/messages Do the same for /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<node_name>.log.x and also truncate the /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<node_name>.log # truncate --size 0 /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<node_name>.log Start manually the node manager # su -l yarn -c "/usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh start nodemanager" Then share the latest files created below /var/log/messages /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-<node_name>.log /var/lib/ambari-agent/data/errors-xxx.txt Please revert  ... View more

@Nawaz    Can you run   HBase hbck command-line tool that checks for region consistency and table integrity problems and repairs corruption? Please see the Checking and Repairing HBase Tables  that should repair the tables and you should be fine thereafter. Please let me know. ... View more

@ssk26  I successfully configured the fair scheduler on the below HDP version Original scheduler          YARN UI Default capacity scheduler after deployment of HDP   Pre-emption enabled  before the change to fair-scheduler Grabbed the template fair-scheduler.xml  fair-scheduler here   I  then changed a few values  for testing  purposes but ensured the is valid using the XML using XML Validator  I then copied the fair-scheduler.xml to the $HADOOP_CONF directory and changed the user & permission   # cd /usr/hdp/3.1.0.0-78/hadoop/conf # chown hdfs:hadoop fair-scheduler.xml # chmod 644 fair-scheduler.xml Changed the Scheduler class in the yarn-site.xml see the attached screenshot. From : yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler To: yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler   Added these new values in Custom yarn-site using the relative path default to /usr/hdp/3.1.0.0-78/hadoop/conf yarn.scheduler.fair.allocation.file=fair-scheduler.xml Custom yarn-site.xml Changed the below mandatory parameter to enable the ReservationSystem in the ResourceManager is not enabled by default yarn.resourcemanager.reservation-system.enable=true   Disable pre-emption Set the below  properties as shown The yarn-site.xml file contains parameters that determine scheduler-wide options. These properties include the below properties if they don't exist add them in the custom yarn-site    Note The below property was available so I didn't add yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled=false     Properties to verify For my testing I didn't add the below properties, you will notice that above that despite disabling the pre-emption in the Ambari UI  the fair schedule shows it's enabled [True]  and my queues ain't showing I need to check my fair-scheduler.xml  attached is the template I used yarn.scheduler.fair.assignmultiple=false yarn.scheduler.fair.sizebasedweight=false yarn.scheduler.fair.user-as-default-queue=true yarn.scheduler.fair.preemption=false            Note: Do not use preemption when FairScheduler                                                                                                                    DominantResourceFairness is in                                                                                                   use and node labels are present. All in all, this shows the fair-scheduler configuration is doable and my RM is up and running !!   I also noticed that the above fair-scheduler was template  overwritten when I checked the YARN Queue Manager so that can now allow me to configure a new valid fair-scheduler Happy Hadooping     ... View more

@Nawaz  The files that were accidentally deleted and moved to .Trash is not altered by any process, so be assured that they are exactly the same files prior t the deletion. Just to prove that fire up the HBase it shouldn't complain of fail. HTH ... View more

@Prakashcit    There is a Jira https://issues.apache.org/jira/browse/HIVE-16575 last updated on 05/Dec/19 Hive does not enforce foreign keys to refer to primary keys or unique keys. In your previous thread, I explained what a NOVALIDATE constratriant is "A NOVALIDATE constraint is basically a constraint that can be enabled but for which hive will not check the existing data to determine whether there might be data that currently violate the constraint" The difference between a UNIQUE constraint and a Primary Key is that per table you may only have one Primary Key but you may define more than one UNIQUE constraint. Primary Key constraints are not nullable.UNIQUE constraints may be nullable. Oracle also implements the NOVALIDATE constraint here is a write-up by Richard Foote When you create a UNIQUE constraint, the database automatically creates a UNIQUE index. For RDBMS databases, a PRIMARY KEY will generate a unique CLUSTERED INDEX. A UNIQUE constraint will generate a unique NON-CLUSTERED INDEX. ... View more

@Prakashcit  To ensure data from multiple data sources are ingested to discover at a later stage business insights, usually we dump everything. Comparison of source data with data ingested to simply validate that all the data has been pushed and verifying that correct data files are generated and loaded into HDFS correctly into the desired location.   A smart data lake ingestion tool or solution like kylo should enable self-service data ingestion, data wrangling, data profiling, data validation, data cleansing/standardization,see attached architecture /landing_Zone/Raw_data/                                                                     [ Corresponding to stage1] /landing_Zone/Raw_data/refined                                                      [ Corresponding to stage2] /landing_Zone/Raw_data/refined/Trusted Data                       [ Corresponding to stage3] /landing_Zone/Raw_data/refined/Trusted Data/sandbox   [ Corresponding to stage4] The data lake can be used also to feed upstream systems for a real-time monitoring system or long storage like HDFS or hive for analytics Data quality is often seen as the unglamorous component of working with data. Ironically, it’s usually the component that makes up the majority of our time of data engineers. Data quality might very well be the single most important component of a data pipeline, since, without a level of confidence and reliability in your data, the dashboard and analysis generated from the data is useless.   The challenge with data quality is that there are no clear and simple formulas for determining if data is correct this is a continuous data engineering task as more data sources are incorporated to the data pipeline.   Typically hive plugged on stage 3 and tables are created  after the data validation of stage 2 this ensures that data scientists have cleansed data to run their models and analysts using BI tools at least this has been the tasks I have done all through many projects HTH ... View more

@hud  When you run nifi as a microservice then you can configure PVC [Persistent Volume Claims] using helm in AKS or Kubernetes which will ensure that evenif the nifi pods restarts it will always have the same volume mounted.   Under the persitence configuration the parameter persistence.enabled should be set to true see Helm Chart for Apache Nifi    HTH ... View more

@saivenkatg55  I have tried to analyze your situation but with access to the Linux box it rather difficult,but I think there is a workaround.   The chattr linux command makes important files IMMUTABLE (Unchangeable).   The immutable bit [ +i  ] can only be set by superuser (i.e root) user or a user with sudo privileges can be able to set. This will prevent the file from being deleted forcefully, renamed or change the permissions, but it won’t be allowed says 'Operation not permitted“' # ls -al /var/run/hadoop-yarn/yarn/ total 8 . .. -rw-r--r-- 1 yarn hadoop 0 Dec 24 09:34 hadoop-yarn-nodemanager.pid   Set immutable bit # chattr +i hadoop-yarn-nodemanager.pid   Verify the attribute with command the below command # lsattr ----i--------e-- ./hadoop-yarn-nodemanager.pid   The normal ls command shows no difference # ls -al /var/run/hadoop-yarn/yarn/ total 8 drwxr-xr-x 2 root root 4096 Dec 24 09:34 . drwxr-xr-x 3 root root 4096 Dec 24 09:34 .. -rw-r--r-- 1 yarn hadoop 0 Dec 24 09:34 hadoop-yarn-nodemanager.pid   Deletion protection # rm -rf /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid rm: cannot remove ‘/var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid’: Operation not permitted   Permission change  protected # chmod 755  /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid chmod: changing permissions of ‘/var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid’: Operation not permitted How to unset attribute on Files # chattr -i /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid After resetting permissions, verify the immutable status of files using lsattr command # lsattr ---------------- ./var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid   Please do that and revert      ... View more

@Prakashcit  That by design: A NOVALIDATE constraint is basically a constraint that can be enabled but for which hive will not check the existing data to determine whether there might be data that currently violate the constraint. This is useful if we know there’s data that violates the constraint but we want to quickly put on a constraint to prevent further violations, with the intention to clean up any possible violations at some future point in time. It’s also potentially useful if we know the data is clean and so want to prevent the potentially significant overheads of hive having to check all the data to ensure there are indeed no violations. ... View more

@GrahamB    No you don't need to wait for 24 hours to destroy a kerberos ticket you will need to run on the Kerberos server  as the user   Check valid ticket To list all of the entries in the default credentials cache $ klist  You should have some out here To delete the default credentials cache for the user $  kdestroy Then to obtain a ticket-granting ticket with a lifetime of 10 hours, which is renewable for five days, type: $ kinit -l 10h -r 5d your_principal   HTH   ... View more