@Mark Use csv serde to escape quote characters in csv file, ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( "separatorChar" = ",", "quoteChar" = "\"") Example:- input data:- a, quick,"brown,fox jumps",over,"the, lazy" Crete table statement:- create table hcc(field1 string, field2 string, field3 string, field4 string, field5 string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( "separatorChar" = ",", "quoteChar" = "\""); Select the data in table hive> select * from hcc; +---------+---------+------------------+---------+------------+--+ | field1 | field2 | field3 | field4 | field5 | +---------+---------+------------------+---------+------------+--+ | a | quick | brown,fox jumps | over | the, lazy | +---------+---------+------------------+---------+------------+--+ 1 row selected (0.055 seconds) So in our create table statement we have mentioned quote character as " and seperator as ,. When we query table hive considers all the data enclosing quotes as one filed. ... View more

@Surya RS Data for comm column will be available in HDFS table location and it won't create any error because Hive works "schema on read" when we query the table it will show the data for all the available columns only. If you reload(insert overwrite) data for the emp table with new set of columns(id,name,sal) then the data for comm column will be lost because we are going to overwrite the all existing data(having data for comm column) with new columns. After reloading with new data if you add comm column to the table it will show null values for comm column because there is no data existed for comm column in hdfs location. Example:- i have emp external table with same schema as you have mentioned in the question hive# select * from emp; +-----+-------+-------+------------+--+ | id | name | sal | comm | +-----+-------+-------+------------+--+ | 1 | hcc | 1000 | empsalary | | 1 | hdp | 2000 | hdpsalary | | 3 | hdf | 3000 | hdfsalary | +-----+-------+-------+------------+--+ Now i'm going to remove comm column by using replace columns hive# ALTER TABLE emp REPLACE COLUMNS(id int,name string,sal int); Selecting the data from emp table again hive# select * from emp; +-----+-------+-------+--+ | id | name | sal | +-----+-------+-------+--+ | 1 | hcc | 1000 | | 1 | hdp | 2000 | | 3 | hdf | 3000 | +-----+-------+-------+--+ comm column has been removed from table and there will be no issues when you query the table. If you specify comm column in select statement then hive returns error. hive# select comm from emp; Error: Error while compiling statement: FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias or column reference 'comm': (possible column names are: id, name, sal) (state=42000,code=10004) Hive returns error as the specified column is not in table. If you check HDFS directory the data for comm column will be still exists, when we add again comm column to the table it will returns the data for comm column again. Adding comm column to the table again:- hive# ALTER TABLE emp add COLUMNS(comm string); hive# select * from emp; +-----+-------+-------+------------+--+ | id | name | sal | comm | +-----+-------+-------+------------+--+ | 1 | hcc | 1000 | empsalary | | 1 | hdp | 2000 | hdpsalary | | 3 | hdf | 3000 | hdfsalary | +-----+-------+-------+------------+--+ After adding comm column again to the table we are able to see the data again.Hive works on schema on read it will displays the data as it matches with schema and datatypes(if datatype doesn't match it shows as nulls). If you are trying to access this table using spark then spark gets table info from hcatalog server and loads the data as per your schema. Loading data into spark after comm column gets removed. >>> hc.table("default.emp").show(10,False) +---+----+----+ |id |name|sal | +---+----+----+ |1 |hcc |1000| |1 |hdp |2000| |3 |hdf |3000| +---+----+----+ Adding comm column again and loading into Spark >>> hc.table("default.emp").show(10,False) +---+----+----+---------+ |id |name|sal |comm | +---+----+----+---------+ |1 |hcc |1000|empsalary| |1 |hdp |2000|hdpsalary| |3 |hdf |3000|hdfsalary| +---+----+----+---------+ . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Kashif Amir Cast the column p as null (or) some value and data types of the column p needs to match on both tables. Example:- hive# select cast("ta" as string)p union select cast("tb" as string)p; +--------+--+ | p | +--------+--+ | ta | | tb | +--------+--+ In the example i'm having 2 tables having same column name(p) and datatype(String) after doing union we are getting merged results. (or) select "i" op union select "ji" p; +---------+--+ | op | +---------+--+ | i | | ji | +---------+--+ ... View more

@Varun R After split text processor use extract text processor and add new property with matching regex, then the extracted attribute will be added as the flowfile attribute. Example:- After split text you are having each address in a flowfile as like this http://aaa.com/q="bigdata"≈i_key="", now you want to know which query param value(ex:bigdata) has been used. Then use extract text add new property query_values q="(.*?)" output flowfile:- Once the flowfile processed by extract text processor it matches the regex and adds the attribute query_values to the flowfile. By this way you are going to know which query param values are used to get response. . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Daminee Sao, By using restApi search url you can get the id of the process group RestApi URL:- curl -i -X GET http://localhost:9090/nifi-api/flow/search-results?q=<name(or)id>; Example:- I'm having flowfiledrop as the name of process group in my NiFi and to get the id of the flowfiledrop process group , RestApi url would be curl -i -X GET http://localhost:9090/nifi-api/flow/search-results?q=flowfiledrop As you can see in i've mentioned flowfiledrop process group name in q parameter $ curl -i -X GET http://localhost:9090/nifi-api/flow/search-results?q=flowfiledrop HTTP/1.1 200 OK Date: Mon, 05 Feb 2018 18:10:03 GMT X-Frame-Options: SAMEORIGIN Cache-Control: private, no-cache, no-store, no-transform Content-Type: application/json Vary: Accept-Encoding Vary: User-Agent Content-Length: 336 Server: Jetty(9.4.3.v20170317) {"searchResultsDTO":{"processorResults":[],"connectionResults":[],"processGroupResults":[{"id":"619a2801-0161-1000-a4c2-95e4430e977f","groupId":"3b738dba-0161-1000-c808-f7d38f21fcab","name":"flowfiledrop","matches":["Name: flowfiledrop"]}],"inputPortResults":[],"outputPortResults":[],"remoteProcessGroupResults":[],"funnelResults":[]}} the id of the flowfile drop process group is 619a2801-0161-1000-a4c2-95e4430e977f. . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors. ... View more