@budati    For this case define your avro schema(with one field) to read incoming flowfile with some delimiter that doesn't exist in flowfile.   So that whole row will be read as string then we can filter out the records by using not like (or) using regex operator in apache calicite.   Select * from flowfile where col1 not like 'SKIP' Now output flowfile will not having any records that have SKIP in them and this solution will work dynamically for any number of columns. ... View more

@ranger  There are three modes for Hive Metastore deployment:            Embedded Metastore Not recommended for production.             Local Metastore This mode allows us to have many Hive sessions i.e. many users can use the metastore at the same time. It's achieved by using any JDBC compliant like MySQL. In this case, the javax.jdo.option.ConnectionURL property is set to jdbc:mysql://host/dbname? createDatabaseIfNotExist=true, and javax.jdo.option.ConnectionDriverName is set to com.mysql.jdbc.Driver. The JDBC driver JAR file for MySQL  must be on Hive’s classpath,               Remote Metastore In this mode, metastore runs on its own separate JVM, not in the Hive service JVM. If other processes want to communicate with the metastore server they can communicate using Thrift Network APIs here you have the ability to have one more metastore servers in this case to provide High availability. having said that it seems you are trying to use embedded Metastore. What I advice you to do is create one as the root through the Ambari ui, it will ask you for the DBName and Host which would be where you installed the MySQL database else pre-create the metastore. The Hive database must be created before loading the Hive database schema that explains why you are getting the startup error.                                                        Using Hive with MySQL On the Ambari Server host, stage the appropriate MySQL connector for later deployment. On the Ambari Server host, Download the MySQL Connector/JDBC driver from MySQL. Runambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar Confirm that mysql-connector-java.jar is in the Java share directory. ls /usr/share/java/mysql-connector-java.jar Make sure the .jar file has the appropriate permissions - 644. Execute the following command: ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Create a user for Hive and grant it permissions. using the MySQL database admin utility: # mysql -u root -p CREATE USER '[HIVE_USER]'@'localhost' IDENTIFIED BY '[HIVE_PASSWORD]'; GRANT ALL PRIVILEGES ON *.* TO '[HIVE_USER]'@'localhost'; CREATE USER '[HIVE_USER]'@'%' IDENTIFIED BY '[HIVE_PASSWORD]'; GRANT ALL PRIVILEGES ON *.* TO '[HIVE_USER]'@'%'; CREATE USER '[HIVE_USER]'@'[HIVE_METASTORE_FQDN]' IDENTIFIED BY '[HIVE_PASSWORD]'; GRANT ALL PRIVILEGES ON *.* TO '[HIVE_USER]'@'[HIVE_METASTORE_FQDN]'; FLUSH PRIVILEGES; Where [HIVE_USER] is your desired  Hive user name,  [HIVE_PASSWORD] is your desired Hive user password and  [HIVE_METASTORE_FQDN] is the Fully Qualified Domain Name of the Hive Metastore host. Create the Hive database. The Hive database must be created before loading the Hive database schema. # mysql -u root -p CREATE DATABASE [HIVE_DATABASE] Where [HIVE_DATABASE] is your desired  Hive database name. After the above step then in the Ambari UI when you reach the Hive Metastpre configuring stage  use the same credentials and the "test"  should succeed and that should fire up when you start all the HDP components  HOpe that helps ... View more