@Jim Barnett Not able to recreate the same scenario on my end(using hive 1.2.1). Wondering how count(distinct key) is generated just one row of count as result ,but in your case query is giving results as select key,count(*) from default.dummy_data group by key; ----- Check is there any data that already exists in hdfs directory that table is pointing to, If yes clear off the data from the hdfs directory and recreate the table again. Example: hive> CREATE TABLE default.dummy_data     > AS     > SELECT row_nbr as key     > FROM (     > SELECT row_number() OVER (partition by '1') as row_nbr     > FROM (     > select explode(split(repeat("x,", 1000000-1), ",")) -- 1,024 distinct     > ) AS x     > ) AS y; hive> select count(distinct key) c from default.dummy_data; 1000000 hive> select key,count(*)cnt from default.dummy_data group by key order by cnt desc limit 10; --ordering desc and limiting 10 Result: key     cnt 10      1 9       1 1000000 1 7       1 6       1 5       1 4       1 3       1 999999  1 1       1 ... View more

@Juhyeon Yun Configure directory as /user/paxata/job and keep recursive sub directories to true will list all the files in the sub directories. Similar question reported in this HCC thread. ... View more

@Bora Özkan One thing i observed in the article is author created hive external table but PutHiveStreaming expects internal table. Create hive internal table CREATE TABLE OLYMPICS(CITY STRING,EDITION INT,SPORT STRING,SUB_SPORT STRING,ATHLETE STRING,COUNTRY STRING,GENDER STRING,EVENT STRING,EVENT_GENDER STRING,MEDAL STRING) CLUSTERED BY (EDITION)INTO 3 BUCKETS ROW FORMAT DELIMITED STORED AS ORC  LOCATION  '/user/hive/olympics' TBLPROPERTIES('transactional'='true'); -> PutHiveStreaming processor expects input data in Avro format. Use ConvertRecord processor to convert data from other formats(csv..etc) to AVRO format. Then feed the avro format data to PutHiveStreaming processor. ... View more

@Karthikeyyan Subbiah I think your attribute is not coming as attribute of the flowfile, if your attribute is in content of flowfile then UpdateAttribute processor won't work. We need to use UpdateRecord processor for that and give the path as /<attribute_name> and add your logic to get the desired format of timestamp. ... View more

@Veera Pavan This job will work fine in Hive but in Spark follow these steps: write the data to temporary table first. then select from temporary table insert overwrite the final table. Check this similar thread regards to similar case. - If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂 ... View more

@Carlo Bustos Could you confirm that DistributedMapCacheClientService is defined and enabled ? Please refer this and this links for defining distributed map cache service. ... View more

@Veerendra Nath Jasthi In UpdateAttribute add new attribute as ts value as ${now():format("yyyy_MM_dd_HH_mm_ss_SSS")} Example: ts attribute will have value as 2019_06_12_20_42_26_762 Then in PutHDFS processor configure directory as /<path>/${ts} (or) You can skip UpdateAttribute processor and directly use directory name as /<path>/${now():format("yyyy_MM_dd_HH_mm_ss_SSS")} In PutHDFS processor. This will create a directory in HDFS with current timestamp value. You can change the format of the timestamp using NiFi expression language. ... View more

@James Willson Try with below spec: [{     "operation": "shift",     "spec": {       "rows": {         "*": {           "value": "[&1].date",           "data": {             "*": "[#3].data"           }         }       }     } } ] Gives output that you are looking for: [ {   "date" : "00:00 2019-06-03",   "data" : 120 }, {   "date" : "05:00 2019-06-08",   "data" : 98 }, {   "date" : "23:00 2019-06-09",   "data" : 172 } ] - If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂 ... View more

@jingyong zou The issue is with the flowfile format that is passed through the processor. As ConvertAvroToJson processor accepts only Avro format but i think you are passing Json format to the processor which is causing java.io.IOException: Not a data file error. ... View more

@petri koski You can create a shell script(ssh into) with all your commands that you wish to run on GPU machine and trigger the shell script using NiFi( Execute Script,ExecuteStreamCommand..etc ) processors. ... View more

@Jean Paul Barddal TEZ initializes session when we run some hive job and based on the tez configurations we will have one YARN application initialized (as you can see in the Resource manager). The TEZ session will be same for all the queries that we are running until the session gets closed, In TEZ view we can see same application id for all the queries that we are running in Hive but DAG_ID will be different for all. In case of MapReduce we are not going to hold any of the resources like TEZ but we are going to run seperate applications for each query. - If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂 ... View more

@Shrikant BM Try increasing the container size set hive.tez.container.size=10240; set hive.tez.java.opts=-Xmx10g; if you are still facing the issue, try increasing the size and run the statements again. ... View more

@James Willson Goto Scheduling tab of GenerateFlowfile processor and choose Timer Driven strategy and keep 300 sec Now processor triggers one flowfile after 300 sec. (or) Using Cron strategy you can keep cron expression as 0 0/5 * 1/1 * ? * Now processor runs and triggers flowfile for every 5 mins. - If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂 ... View more

@James Willson I just downloaded NiFi-1.9.2 from this link. GenerateFlowFile processor is included in NiFi-1.9.2 version. Not sure why you are not able to have GenerateFlowFile processor in 1.9.2. If i try to search for generate then i was to see GenerateFlowfile processor and GenerateTableFetch Processors. ... View more

@Shrikant BM 1. How to move the data from the source ACID table to a non-ACID table. hive> create table <db_name>.<non_acid_table> stored as orc as select * from <acid_table>; Now you can copy the non_acid_table data in HDFS from Prod to pre_prod using Distcp command. 2. How to move the data from non-ACID table to ACID table. 2.1. Create your acid_table if not exists. 2.2. hive> Insert into acid_Table select * from non_acid_table; - If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂 ... View more

@Rosa Negra Yes, you are in right track we need to use Validate Record processor to perform this check and while defining avro schema you need to use Avro logical types for timestamp type and we need to keep the format of the timestamp field also. Refer to this template for defining avro schema with logical types. ... View more

@3nomis ConvertRecord processor is still running as we can see running threads as one at top right corner of the processor. If it is running for ever (or) you are not convinced with performance then you can try splitting the 2GB file into smaller chunks by using SplitRecord processor then use MergeRecord processor to merge them back by using Defragment as merge strategy. ... View more