About ss_anticipation

ss_anticipation · ‎02-04-2019

@Matt Burgess, Hello Matt, need your help on the above. I tried multiple things and nothing seems to be working. I am using nifi 1.8 when i use JSONTOSQL, i get insert into aaa values (?,?,?,?) how do i pass the values here?

ss_anticipation · ‎02-01-2019

@Shu, Hi Shu, need your help. I have to ingest csv to hive and below is the format getFile -->inferavroschema-->convercsvtoavro-->putHDFS-->ReplaceText-->PutHiveQL I am getting data till putHDFS and it is in avro format. I am using replacetext to create INSERT script its not fetching any records. avrofile looks like this [{"name":"field_0","type":"string","doc":"Type inferred from 'ID'"},{"name":"field_1","type":"string","doc":"Type inferred from 'CITY_NAME'"},{"name":"field_2","type":"string","doc":"Type inferred from 'ZIP_CD'"},{"name":"field_3","type":"string","doc":"Type inferred from 'STATE_CD'"}] using below insert command INSERT INTO aaa (field_0, field_1, field_2, field_3) VALUES ('${field_0}', '${field_1}', '${field_2}', '${field_3}') but its not working. sql statement generated is, insert into aaa values (,,,)

ss_anticipation · ‎01-28-2019

Hello All, I want to ingest csv file into Hive table. But, csv file format changes frequently i.e I get different file each time as a csv. I would like to develop a workflow which handles any kind of csv file. Does nifi provides any such options? if yes, what should be the process that needs to be followed?

ss_anticipation · ‎01-23-2019

Thanks Shu. makes sense. Let me checkout the feasibility and in case I get into some issues, I'll let you know. Thanks again for taking time and replying to my question

ss_anticipation · ‎01-21-2019

Hello Team, I have 5 source systems, 2 are RDBMS systems and 3 are files that need to be picked up from some server X.we do not have Kafka. I cannot use Flume, since it doesn't have DB connectors and using for file movement is not a good choice. I can use NiFi, but my question is, do i need to create 5 workflows for each source system and schedule them? or does NiFi has something similar to Kafka topic, where i can create 5 topics in a directory and respective target system can consume from those topics

ss_anticipation · ‎01-15-2019

Perfect Matt. makes sense. I changed the directory to /tmp and i was able to write the files.

ss_anticipation · ‎01-13-2019

@Matt Clarke, would you please help me in resolving this issue? I still do not understand why permission 777 is unable to resolve the issue.

ss_anticipation · ‎01-11-2019

Hi Matt, Thank you for the response. For directory /home/xxx/receive I have given 777 permission and that should resolve the permission issue right? I ran grep command and it gave me a huge output. I looked for "/usr/hdf/current/nifi-registry/bin/nifi-registry.sh" and owner and group was "nifiregistry".

ss_anticipation · ‎01-11-2019

Hello All, I have installed Nifi(HDF) on server A. I connect to Ambari and Launch NiFi UI. Everything looks good till here. FIrst i tried to copy a file from 1 directory to another using GetFile. It did not recognize the directory and got the following error "input directory invalidated because input directory does not exist while creating getfile in nifi". I am on the same server where Nifi is Launched. I checked all permissions and everything looks good but not sure why it complains directory does not exist. I tried to generate a file and write to a directory, files are getting generated but getting below error. "failure due to java.nio.file.AccessDeniedException: /home/xxx/receive: java.nio.file.AccessDeniedException" can someone please help me in fixing this issue.

ss_anticipation · ‎01-04-2019

Hello Experts, can someone please help me?

ss_anticipation · ‎12-28-2018

Hello Team, I have .dat files ranging from 2GB - 500GB that are present in SAS server. I need to get these files and load it into Hive table. How can this be automated? I considered few options but they do not transfer the files fast. Flume and Nifi: These are meant for realtime data ingestion. Though i can use them, but it will be an overkill. They are simply not meant for file transfer. distcp: It is between HDFS cluster.On one server, i do not have HDFS scp/winscp: These would be very slow over the network and for files >500G, I dont think anyone would recommend this approach. Sqoop: I will not have access to tables, If i manage to get aceess, will i be able to transfer files by increasing number of mappers? Any thoughts/suggestions are welcome and appreciated.

Online	Offline
Last Visited	‎02-11-2019 07:45 AM

Date Registered	‎12-28-2018 04:17 AM
Last Visited	‎02-11-2019 07:45 AM
Total Messages Posted	11

Re: Nifi use case for ingesting data in Hive

Re: Nifi use case for ingesting data in Hive

Nifi use case for ingesting data in Hive

Re: Data pipeline options

Data pipeline options

Re: HDF NiFi issue. Not recognizing directories

Re: HDF NiFi issue. Not recognizing directories

Re: HDF NiFi issue. Not recognizing directories

HDF NiFi issue. Not recognizing directories

Re: Moving .dat file into HDFS

Moving .dat file into HDFS