About stevenmatison

stevenmatison · ‎04-13-2020

@Rups Your GetMongo should have a Limit, Batch Size, and Results Per FlowFile. Investigate using these to get the # of results you want per FlowFile. If you do not see these configurations investigate nifi 1.9 or greater. I would recommend NOT using SplitJson to split results. For putFile you need to update the filename to something unique per FlowFile to avoid the duplicate filename. For example something like: filename = ${filename:append(now():format('yyyy-mm-dd hh:mm:ss'))} or use the FlowFile's Filename: filename = ${Filename}

stevenmatison · ‎04-13-2020

@Eric_B Are the tables Presto cannot read owned by NiFi? The error you share seems like a permissions issue to the underlying files. Also if you can, please share screen shots of your processor configurations.

stevenmatison · ‎04-13-2020

@Rod_B_2020 have you tried finding the driver on the open internet? Also have you tried using the 6 version?

stevenmatison · ‎04-13-2020

@ForrestGump No, out of the box NiFi should work with all Processors and Controller Services. The stock configuration of NiFi should work without any issues. You should not see any inconsistencies or stability issues unless you are exceeding resources available on the NiFi node(s). If you are seeing specific issues with " Regex, prepend, and other items" they each should give a very specific error. Sometimes the errors are not shown in the UI. You have to tail the nifi app logs in order to see full error informations.

stevenmatison · ‎04-13-2020

@fizadi Here are some ways to get files to hdfs: From shell to hdfs: su - hdfs hdfs fs -copyFromLocal /local/path/file.ext /hdfs/path/ hdfs dfs -copyToLocal /hdfs/path/file.ext /local/path/ Using HDFS Files View - there is an upload feature here Using HUE - Hue UI has an HDFS Plugin Option #1 assumes the shell is within the cluster (master, worker, edge).

stevenmatison · ‎04-13-2020

@Hive_SQL The solution here is to create a hive table of New Format (|). Next Select/Insert Old Format(/t) into New Format. For example: select * from old_format INSERT INTO new_format

stevenmatison · ‎04-13-2020

@Raj2cool16 I was looking to see the Avro Schema in the Record Reader. I think you have another post on this same issue. I also asked there to see screen shots of the full config and schemas. When we can see that info in the post, the community can better understand if there is an adjustment needed to resolve the original issue.

stevenmatison · ‎04-13-2020

@AndyL Can you update original post or reply with screenshots of the Processor Configuration and Record Reader Configuration (including avro schemas)?

stevenmatison · ‎04-13-2020

@Raj2cool16 Can you update the original post, or send a reply here, showing the processor configuration and associated reader configurations? I believe there are some settings in the Processor and the Reader to effect your issue.

stevenmatison · ‎04-13-2020

@Kamran1993 I took a look at the documentation you referenced. I think that the documentation is showing you how to setup hdp or hdf. Not both. In my reply above I am referencing only what I see is the problem to get you moved forward. I am not directly familiar with that sandbox, but I am familiar with what you need to do go get HDP and HDF to work. In your above posts you are having problems with both HDP & HDF ambari trying to access same port on same node. Even if you change the ambari configuration port for HDF (for example 8081), other services' ports will still conflict on the same node. I also notice above you mention 22gb ram and only having 16gb available. I do not think this is an issue but as I said before its hard to run HDP and HDF on same node. Can you make two 8 gb VMs and use 1 for HDP and 1 for HDF? Trying to get them both running on same machine will be a bigger challenge than the twitter tutorial.

stevenmatison · ‎04-12-2020

RE: " when i put the IP address: in my case 192.18.*.***:8080 either HDF or HDP " You can't have both hosts mapped to the same IP and expect ambari to come up on both hosts with the same port (impossible) 192.168.0.107 localhost sandbox-hdp.hortonworks.com sandbox-hdf.hortonworks.com I would recommend a different IP for each host. You can also change port for HDF or HDP, in its ambari.properties so they are not on same port, but running both in same node is a challenge for resources. Also, if you are having issues with ambari loading on full fqdn (and the ip works), you should confirm the configs are setup using the fqdn and not "localhost" or the local ip (192.168.0.107)

stevenmatison · ‎04-10-2020

@Raj2cool16 Can you share your schema?

stevenmatison · ‎04-09-2020

@bhara You do not have to use LDAP. You can create the users in HUE admin, using the first admin user you created. If you want to configure LDAP please see official documentation here: https://docs.gethue.com/administrator/configuration/server/#ldap You will need to make the LDAP hue.ini changes via ambari in the HUE->Config->Advanced->Advanced Hue-Ini and restart hue after each change. Your error above are 2 issues I notice: SSL Configuration for HDFS. Your HUE truststore must have the ssl certs for hdfs hosts https://gethue.com/configure-hue-with-https-ssl/ (bottom section) https://docs.cloudera.com/documentation/enterprise/5-11-x/topics/cm_sg_ssl_hue.html (top section) HDFS Configuration - Doc Here The SSL Example links above are not specific to your case (HDP) but still apply. Also I am assuming you have hdfs secure. The links I share for SSL outline the fundamentals required to put the right HDFS and SSL settings in hue.ini for secure access to hdfs. The HDFS Configuration link is official gethue.com documentation for HDFS. You will need to make the SSL hue.ini changes via ambari in the HUE->Config->Advanced->Advanced Hue-Ini and restart hue after each change.

stevenmatison · ‎04-08-2020

@ForrestGump there must be some configuration difference then. I created a simple flow and was able to get below output using exact proc I screen shot above on your Before data: Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_var,Time_btwn_observation_sec,Field_magnutude_average_nT,BY_nT(GSM),BZ_nT_(GSM),RMS_SD_B_scalar_nT,RMS_SD_field_vector_nT,Speed_km/s,Alfven_mach_number,Magnetosonic_Mach_number,BSN_location_Xgse_Re,2019,1,0,0,51,100,2788,164,0.12,999999,5.11,2.00,2.73,0.08,1.03,451.0,9.8,6.5,13.15 2019,1,0,1,51,100,2810,159,0.12,37,5.10,2.33,2.58,0.11,1.04,451.3,9.8,6.5,13.10 2019,1,0,2,51,80,2852,109,0.09,18,4.86,2.37,2.56,0.12,0.56,454.7,10.3,6.7,13.07 2019,1,0,3,51,67,2951,66,0.06,-39,4.78,2.21,2.55,0.03,0.33,452.3,11.0,6.8,13.00 2019,1,0,4,51,100,3025,7,0.00,-13,4.80,2.17,2.37,0.03,0.14,451.4,11.2,6.8,13.00 2019,1,0,5,99,80,2973,111,0.09,111,4.94,2.68,2.39,0.13,0.55,99999.9,999.9,99.9,13.19 2019,1,0,6,51,67,3074,20,0.02,-40,4.88,2.54,2.01,0.02,0.28,451.0,9.8,6.5,13.27 2019,1,0,7,51,50,3114,9,0.00,19,4.82,2.37,2.93,0.02,0.14,451.0,9.9,6.5,13.29 2019,1,0,8,99,999,999999,999999,99.99,999999,9999.99,9999.99,9999.99,9999.99,9999.99,99999.9,999.9,99.9,9999.99 2019,1,0,9,99,100,3036,0,0.00,999999,5.16,3.34,2.44,0.00,0.00,99999.9,999.9,99.9,13.24 2019,1,0,10,99,100,3036,0,0.00,60,5.16,3.34,2.43,0.00,0.00,99999.9,999.9,99.9,13.24 I have dropped the template for you here: https://github.com/steven-dfheinz/NiFi-Templates/blob/master/Replace_Text_Demo.xml

stevenmatison · ‎04-08-2020

@bhara Change the line 193 below and try to start again. File: /var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/params.py dfs_namenode_http_address = config['configurations']['hdfs-site']['dfs.namenode.http-address'] to dfs_namenode_http_address = 'localhost' That will give dfs_namenode_http_address a value and get past the error.

stevenmatison · ‎04-08-2020

@Pe If you have a new question, you can post it and reference the article. However, since im already responding, I will be brief. Yes, all dev, test,prod to usually Prod NiFi Registry. Or sometimes all to Dev, your preference. To connect to existing keystores, would just add the new certs from the nifi registry host to all the other existing keystores/truststores.

stevenmatison · ‎04-08-2020

Notice my settings for All Properties and I use Regex. If you don't want to use Regex, change Replacement Strategy. I would recommend doing it as I have shown, with regex on complete flow file. Not line by line. Also, you can append and prepend more text and transformations in same replace too. For Example: PREPEND ${'$1':replace('"',''):replace(',',' '):replace('\n',',\n')} APPEND

stevenmatison · ‎04-08-2020

@ForrestGump I believe you are using ReplaceText. I use it like this:

stevenmatison · ‎04-08-2020

@unrec I would recommend and absolute path outside of the nifi folder. Then provide the full path inside the processor not using "." which is causing your conflict. This folder should be writeable and owned to nifi.

stevenmatison · ‎04-08-2020

The creation of large database schemas can be a very complicated task. In this article, I am going to share how I used NiFi to fully automate a monstrous task. For my project, I needed to create very large Avro Schemas, and corresponding Hive Tables for five or more Data Sources, each having from 400-500+ different CSV columns. Doing this manually would have been a nightmare just to manage the initial schema creations. Managing schema changes an even bigger task over time. My answer was the Schema Generator API using NiFi and Schema Registry. Please reference the following NiFi Template: Schema Generator API Demo Template Schema Generator API The above NiFi template provides the following capabilities: A NiFi API is capable of the following: Accepting POST for a new table, given the tables data columns and data types is in CSV Format Creating Schema Registry Entity (POST: create) Creating Schema Registry Avro Schema (POST: parse) Creating Hive HQL Schema Executing Hive HQL Statement A sample call to create Schema Registry Entity (demo). A sample call to parse Data Columns (22 string columns). Lots of helpful labels with notes. The following are the template setup instructions: Download, Upload Template, and Drag Template to your NiFi Pallette. Make sure a Schema Registry is Setup within reach of NiFi. Edit the following Schema Generator Demo Process Group's Variables: schemaGeneratorApiHost schemaGeneratorApiPort schemaRegistryUrl hiveDatabaseName hiveDatabaseConnectionUrl (jdbc string) hiveConfigurationResources (path to hive-site.xml) Enable controller services in Schema Generator API process group: StandardHttpContextMap for HandleHttpRequest & Response HiveConnectionPool for PutHiveQl Start Schema Generator API Processor group. Navigate to samples and execute Sample Call 1, then 2 by switching appropriate GenerateFlowFile On/Off. These 2 proc are disabled by default as you should switch them On and Off immediately. These are the only 2 proc that should not always run. Disable them again when done. This is just a basic demonstration to get you started with Schema Registry and Data Source Schema Automation. Parts of this template are also helpful for anyone who needs to automate creating Avro Schemas and/or Hive Schemas for large CSVs which could still be done without Schema Registry. The demo above has been tested up to 500 columns and includes mapping various different column types to hive data types. Important Information The template is built and tested on NiFi 1.9, Single Node Nifi Cluster, with local Schema Registry Installed. The Schema Registry UI doesn't have full capability. Learn the API to work with your Schemas Directly. For Example: Delete. See my previous post Using the Schema Registry API for detailed API info. Versioning Schema Forward and Backward can be very problematic. Be Warned. Use a proper and consistent table and column naming conventions. Complicated column names will break Avro and Hive. Example characters include but not limited to: spaces, /, \, $, *, [, ], (, ), etc. Schema Registry Entities and Associated Avro Schemas can be used in NiFi Record Readers, using HortonworksSchemaRegistry, and other Controller Services.

stevenmatison · ‎04-08-2020

Description: A continuation of 2.x Ambari Hue Service for HDP 3.x. Repo Info: See GitHub ReadMe for detailed information. Repo URL: https://github.com/steven-dfheinz/HDP3-Hue-Service Account Name: steven-dfheinz Repo Name: HDP3-Hue-Service This is now tested and operating in HDP 3.1.4 but should work in any flavor of HDP 3.x I do not recommend using this in HDP 2.x. If you want to work in a flavor of HDP 2.x, please start with the original repo (here). However, you must know there are differences required to get the old repo to work. You can message me if you want to know what those differences are or just compare the repo file sets for differences. I will be working on this repository to include Hue 4.x as time permits.

stevenmatison · ‎04-08-2020

Description Install Hue with Ambari using this management pack created by DFHz. Repo Info Repo URL: https://github.com/steven-dfheinz/dfhz_hue_mpack Account Name: steven-dfheinz Repo Name: dfhz_hue_mpack

stevenmatison · ‎04-07-2020

In a recent Use Case of NiFi + ELK, we route all failures for ELK Processors to a Replay Process Group. If any failures, we have a notification, and collect them in a replay queue that is disabled. Once the notification is seen, and or some corrective action taken to make sure there are no issues, a user logs in and enables queue, which routes back to original processor to execute again. I hope a similar setup can help resolve your issue.

stevenmatison · ‎04-03-2020

I have taken the (now broken) EsharEditor Repo and made it work for HDP2 & 3, with Hue 3 & 4, and bundled into an ambari management pack. Details here: https://github.com/steven-dfheinz/dfhz_hue_mpack

stevenmatison · ‎04-03-2020

@bhara No problem. Glad I could help and I appreciate your feedback too. When you do get it started, the setup and configuration is going to need similar attention. Please keep me updated and/or reach out in Private Messages. You can always open new Questions here and tag me in them too.

stevenmatison · ‎04-03-2020

I have now completed the DFHz Hue Management Pack which you can now use to install Hue 3 or Hue 4 in HDP 2 or HDP3. Just be careful in selecting your correct version(s) and getting the correct mpack install command. They are each very specific. Check out the repo: https://github.com/steven-dfheinz/dfhz_hue_mpack

stevenmatison · ‎04-03-2020

@bhara This morning I did a 3 node HDP 2.6.5 cluster with NameNode HA (the point you had conflict in). The install worked from management pack without issues. Hue install and started. No problems. In my early morning testing, I had installed Hue before doing NameNode HA. So for the HA test, I stopped hue, executed 2 rms below, deleted service from ambari, and did a re-install from ambari. 14 rm -rf /usr/local/hue 15 rm -rf /usr/hdp/current/hue-server If you continue to have issues, you will need to debug using method I showed you before: edit the files that are throwing errors. Looking at the conflicts, and commenting/debugging python is how I even got this to work at all. In my original article I discuss some important information about debugging the service files directly, and using the retry button during the install or start section of the Ambari Add Service Wizard. If you do find a solution that needs to be changed, please report it back so that I can get the repo updated. I am doing an update now to include the rms above, making an install/reinstall from ambari easier.

stevenmatison · ‎04-03-2020

@sivakumarss My recommendation would be to start with making some empty (empty process group) templates and study the content. Next make a template which starts to contain processors and controller services. Work in small steps comparing content of each. You should then be able to very easily figure out what is required.

stevenmatison · ‎04-03-2020

@Gubbi I think these topics are related? The idea you need to accomplish here is getting all the files. Then once all the files are in nifi, check the details of the filename to get the date from there instead of using now() and comparing the filename to some version of "now" or "now minus X days". Once you have the files you just work off the filename to get the date: date = ${filename:replace('.csv',''):substringAfter('_')} ^^ I did not test this, so you may have to adjust the nifi expression language to get the exact date match Then use the ${date} in the path when you store the file. This method removes the requirement to have different routes based on the date matching, and everything goes the same route with the dynamic path based on ${date}. Hope this helps.

stevenmatison · ‎04-03-2020

@Gubbi You need to figure out how to use ListFile to get everything, then check the dates/filenames/folders after the files arrived. Regardless of what folders the files are in. This way if a folder from 3 days ago appears tomorrow, you will still download the files. Can you scan /user/data/ and take all folders and files?

Online	Offline
Last Visited	‎05-26-2020 11:11 AM

Name	Steven Matison
Location	Florida
Personal web page	https://github.com/steven-dfheinz
Date Registered	‎07-19-2018 04:45 PM
Last Visited	‎05-26-2020 11:11 AM
Total Messages Posted	434
Total Tags	52
Total Kudos Received	69

Re: How to create a json request for invokehttp pr...

Re: InvokeHTTP randomly hangs

Re: CSV with embedded JSON to JSON - NiFi

Re: AVRO Schema Registry Namespace

Re: HDP 3 public repo

Re: Multiple JSON to JSON Array Output file

Re: Presto (Prestosql) can't read ORC ACID transac...

Re: JDBC driver Rhel 7

Re: Configuring Nifi

Re: Web shell management

Re: How do to show query results as tab delimited ...

Re: Error: Writing Parquet data to PosgreSQL using...

Re: PutDatabaseRecord and ARRAY type

Re: ERROR while loading CSV to PostgreSQL using Pu...

Re: HDP AND HDF Twitter Sentiment Analysis Applica...

Re: HDP AND HDF Twitter Sentiment Analysis Applica...

Re: Error: Writing Parquet data to PosgreSQL using...

Re: Error while configuring HUE for HDFS

Re: Replacing spaces in Ascii file (Nifi)

Re: Getting Error while starting HUE services

Re: Setting Up a Secure NiFi to Integrate with a S...

Re: Replacing spaces in Ascii file (Nifi)

Re: Replacing spaces in Ascii file (Nifi)

Re: NiFi PutFile processor doesn't save file to a ...

How to automate creation of Avro and Hive Schemas ...

Hue 3.11.0 on HDP 3.x

Ambari Management Pack to install Hue 3.x or Hue 4...

Re: NiFi ScrollElasticsearchHttp - continue from a...

Re: How to install Hue on the HDP 2.6.1. sandbox (...

Re: Instalation of HUE using Ambari for hdp 2.6.5

Re: How to install Hue 3.11 in HDP 3.1

Re: Instalation of HUE using Ambari for hdp 2.6.5

Re: XSD schema for template XML.

Re: Transfer files based on date present in filena...

Re: Monitor Historical files and transfer to desti...