About stevenmatison

stevenmatison · ‎03-30-2020

@Former Member The solution is to use variables (in older versions (<= 1.9 and on) and parameters (1.10 and on).

stevenmatison · ‎03-30-2020

@MPraveen Hello friend! I dont your sql schema but the error is wanting the real time version of the data (without the /). Glad you were able to solve it. Accept your own answer and get a solution credit!!

stevenmatison · ‎03-29-2020

@mahfooz The mysql jar should be in the class path for the sqoop client. Also make sure the file ha the correct permissions.

stevenmatison · ‎03-29-2020

@saaga119 The solution here is to create the external table in the format you need to query the raw xml data. Seems like you have this done already. Next create a native hive table. It could still be external too. The new table should be the schema for the fields you want with required ORC lines. You can also choose some other text format; for example: csv. Last INSERT INTO new_table SELECT * FROM external_table. The main idea here is to use external and staging tables, then INSERT INTO SELECT to fill the final table you want. Final table could then be optimized for performance (ORC). This kind of idea also allows multiple external/raw/staging tables able to be combined (select w/ join) into a single final optimized table (orc, compression, partitions, buckets, etc).

stevenmatison · ‎03-27-2020

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route? I would recommend to simply the match in RouteOnAttribute: ${filename:endsWith(${today})} If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

stevenmatison · ‎03-27-2020

@Savi I think you only need one date format in your nifi expression language. Next, it seems you know what format it is upstream (source), and what it needs to be for downstream. Based on your expression need, you should play the flow up to the csv reader processor and then List Queue on the FlowFiles and inspect the attributes. Adjust the expression, and retest until the attribute format meets the required format. Inspecting the dates upstream won't show you what the date looks like after expression and before the reader. Often times I find myself testing a flow step by step, each step confirming the attributes needed are correct for the next step. Never moving forward to the next step until I am sure the attributes are correct after visual inspection.

stevenmatison · ‎03-27-2020

@murali2425 It sure is possible. I believe anything is possible in NiFi!! The excel processor is: ConvertExcelToCSVProcessor I have also put an Excel to SQL template on my GitHub: https://github.com/steven-dfheinz/NiFi-Templates This template gets an excel file that contains product inventory data (sku, name, price, description, quantity, etc and routes the contents of the excel file to ConvertsExceltoCSV. Then it splits the csv file line by line, parses each line for quantity and sku and uses those attributes to build an insert query, which is finally executed via PutSQL. There are many different ways to deliver the Excel file too ConvertExceltoCsv, different ways to process the CSV, and different ways to insert into SQL. The above is just one way I have used in the past.

stevenmatison · ‎03-26-2020

Update for Hue 4.x: I have finished initial required changes to for branch Hue.4.6.0 and successfully installed Hue 4.6.0 in HDP 3.1.4. There ended up being some big differences in the hue.ini for 4.x. Once these were resolved the only manual work needed for install in the node is creating the hue user and group and running the python to ignore_groupsusers_create. User Commands: useradd -g hue hue usermod -a -G wheel hue chown -R hue:hue /home/hue Reset/Remove Commands: rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/ rm -rf /usr/local/hue rm -rf /usr/hdp/current/hue-server rm -rf /usr/local/hue-4.6.0/ Once installed it is possible to update/manage changes to hue.ini via ambari to enable HDFS, Hive, RDBMS, enable/disable Hue Plugins, or further configure Hue as necessary.

stevenmatison · ‎03-25-2020

@ParthiCyberPunk In HDP3 the only out of box option for hive is beeline. Additionally many third Party Tools can access hive via jdbc connections. I have been working on Hue with HDP3 and have had some success with Hue 3.x and HDP 3.x in production already. Here is article: https://community.cloudera.com/t5/Community-Articles/How-to-install-Hue-3-11-in-HDP-3-1/ta-p/291280 Here is the repo: https://github.com/steven-dfheinz/HDP3-Hue-Service I am currently workjng on hue 4.x and a management pack for HDP 2.x - 3.x

stevenmatison · ‎03-24-2020

Repo Branch created for Hue 4.6.0. I am working on this right now but should have it operating here shortly. If you want Hue 4.6.0 be sure to download Branch Hue.4.6.0.

Online	Offline
Last Visited	‎06-01-2022 03:47 PM

Name	Steven Matison
Location	Florida
Member Since	‎07-19-2018 04:45 PM
Last Visited	‎06-01-2022 03:47 PM
Posts	613
Kudos received	101

Cloudera Community

Re: Apache nifi - how to convert a file .txt into ...

Re: Apache Nifi - Using PutParquet, the HDFS file ...

Re: How to extract csv column record and used it f...

Re: Could not connect to Distributed Map Cache ser...

Re: NiFi InvokeHTTP POST JSON

Re: sensitive value access key, secret key and pas...

Re: PUTSQL which cannot be converted to a timestam...

Re: -libjar option is not working in sqoop import ...

Re: Hive and XML parsing and saving table as ORC o...

Re: Transfer files based on the YYYYMMDD in their ...

Re: Date is not parsed as expected in Csv Reader P...

Re: Is it possible to create a table in mysql usin...

Re: How to install Hue 3.11 in HDP 3.1

Re: why HDP-3.0 version not showing Hive view..? ...

Re: How to install Hue 3.11 in HDP 3.1