About mustafa_qizilba

mustafa_qizilba · ‎11-06-2018

@Scott Shaw @James Dinkel We have implement HDP with LLAP enabled, followed all the tutorials and able to connect Tableau with Hive using ODC driver and using port: 10500. HDP 3.0.0: Single Node - 64 GB RAM - 32 Cores HDF 3.2.0: Single Node - 32 GB RAM - 32 Cores Tableau Desktop: Single Node - Windows - 64 GB RAM Problem Statement: (unable to using both port: 10000 and 10500 (LLAP) in parallel) When we connect Tableau using port: 10500 (ODBC Driver) and refresh dashboards, we can not load or query data from beeline/ hive using port: 10000, if we do query hangs. If we first query using beeline/ hive using port: 10000 and then try to run dashboards in Tableau using port: 10500, Tableau hangs Will really appreciate quick help here. Looking forward. Cheers.

mustafa_qizilba · ‎10-28-2018

@Geoffrey Shelton Okot @Matt Burgess @Shu

mustafa_qizilba · ‎10-26-2018

Scenaria: Run a query on a table e.g. emp in Hive using port: 10,000 i.e. non-llap HiveServer2 Connect and pull data in Tableau from same table e.g. emp using port: 10,500 i.e. llap HiveServer2 Environment: HDP 3.0 (Hive 3) Tableau Desktop (latest) Error/ Issue: Run query in Hive (using port: 10,000) then Tableau dashboards (using port: 10,500) doesn't refresh and hangs. Then close hive and refresh Tableau dashboards (using port: 10,500), its refreshes fine but now when run query in hive (using port: 10,000), hive hangs. Conclusion: Issue is, we are unable to connect with hive and tableau as the same time. Please advice.

mustafa_qizilba · ‎09-29-2018

@Jay Kumar SenSharma So we moved ahead and used another IP: XX.XX.XX.51 (which has internet access) to install HDP and then reverted HDP back to its original IP: XX.XX.XX.160 (which does't have internet access) and did the same to install HDF using IP: XX.XX.XX.51 (which has internet access) and then reverted HDF back to its original IP: XX.XX.XX.161 (which doesn't has internet access). When HDP and HDF were on IP: XX.XX.XX.51, all the services were up and running and restart-able but the moment those were reverted back i.e. HDP and HDF to those original IPs, everything got hung. Now services are in start mode as were when were on IP: XX.XX.XX.51 and we can't stop & restart and other things like hive is not accessible, NiFi UI does't appear etc. Please advice.

mustafa_qizilba · ‎09-23-2018

@Jay Kumar SenSharma Yup permissions are good. The problem is, there is no index.html file present in/repo/repodata directory.

mustafa_qizilba · ‎09-23-2018

@Jay Kumar SenSharma Hi, Apologies for delay. There you go with WebServer Log. -------------------------------------------------------------------------------------------------------------------------------------------- [root@ufm httpd]# tail error_log [Sun Sep 23 16:18:17.112853 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path [Sun Sep 23 16:18:19.399155 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/ambari/centos7/index.html denied (filesystem path '/var/www/html/repo/ambari/centos7/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ [Sun Sep 23 16:18:20.584500 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:22.762022 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:23.613018 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:41.563016 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53473] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 18:11:29.286927 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53653] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path [Sun Sep 23 18:13:02.298494 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53660] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP/ [Sun Sep 23 18:14:16.053361 2018] [core:error] [pid 1780] (13)Permission denied: [client10.200.40.51:53663] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-UTILS/ [Sun Sep 23 18:15:56.767202 2018] [core:error] [pid 1781] (13)Permission denied: [client10.200.40.51:53666] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-GPL/ -------------------------------------------------------------------------------------------------------------------------------------------- Note: As this is offline line installation, we don't have index.html any where. Please advice.

mustafa_qizilba · ‎09-18-2018

@SANJAY RAI Any resolve on this issue, as i am facing same?

mustafa_qizilba · ‎09-18-2018

@Jay Kumar SenSharma @Geoffrey Shelton Okot It seems, repodata/repomd.xml is the culprit. Please advice its uses and content with no-internet access as its content is; <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> I feel above mentioned URLs should not go on internet.

mustafa_qizilba · ‎09-18-2018

@Jay Kumar SenSharma We moved ahead by disabling rest the repos and enabling ambari one but got stuck at cluster building. Note: When we paste this URL: http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml it does work. Following failed at installing this service: Timeline Service V1.5 Install failure: repodata/repomd.xml from HDP-3.0: [Errno 256] No more mirrors to try. http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable 2018-09-18 16:05:51,775 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {} 2018-09-18 16:05:51,805 - call returned (0, '') 2018-09-18 16:05:51,805 - The 'hadoop-yarn-timelineserver' component did not advertise a version. This may indicate a problem with the component packaging. Command failed after 1 tries I AM SURE, WE DO NOT NEED INTERNET as a mandatory pre-requisite?

mustafa_qizilba · ‎09-18-2018

@Geoffrey Shelton Okot @Shu @Matt Burgess Hi All, We are facing issue installing using Local Repositories as don't have internet access in our environment. This issue is pending since last 2 weeks and we are unable to crack this via any forum or community. Error: Cannot find a valid baseurl for repo: base/7/x86_64 ------------------------------------------------- Environment: VMWare - ESXI - done CenOS 7 - done Java 8+ - done MySQL 5.7 - done Files updated - done /etc/hosts /etc/sysconfig/network /etc/hostname /etc/selinux/config /etc/yum/pluginconf.d/refresh-packagekit.conf /etc/profile .bash_profile Repo Directories and rpm placement - done /var/www/html/repo/ambari /var/www/html/repo/HDP-UTILS /var/www/html/repo/HDP /var/www/html/repo/HDP-GPL Repo files creation - done /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/hdp.repo /etc/yum.repos.d/hdp.gpl.repo Services - done systemctl stop firewalld systemctl start ntpd systemctl start httpd systemctl status mysqld.service (all required users and databases are created along with also ran: Ambari-DDL-MySQL-CREATE.sql in DB: Ambari Amabri Server: Pending Ambari Agent: Pending HDP 3.0: Pending ----------------------------------------------------------------------------------- Issue: Using local repositories, whenwe run this command: yum install ambari-server following error hit us. Error: [root@ufm yum.repos.d]# yum install ambari-server Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was 14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error" One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: base/7/x86_64 ----------------------------------------------------------------------------------- Note: Above whole installation was achieved on Local VM enviroments but not on ESXI (this is our first time on EXSI environment) Bare Metal Servers Limitation: We do not have internal access in our environment, so have to use local repositories. Looking forward. Cheers

mustafa_qizilba · ‎09-15-2018

@Shu pmresult-67109368-60-201804260000-201804260100-ori.zip (Please unzip, its a csv file) sample-xml.xml Same result :-(. Also attaching XML and Source data file for testing. Empty Queue Avro Schema: { "type":"record", "name":"jazz", "fields":[ {"name":"hw_3g_result_time", "type":["null","string"]}, {"name":"hw_3g_granularity_period", "type":["null","string"]}, {"name":"hw_3g_bsc_name", "type":["null","string"]}, {"name":"hw_3g_bsc_type", "type":["null","string"]}, {"name":"hw_3g_cell_name", "type":["null","string"]}, {"name":"hw_3g_cell_id", "type":["null","string"]}, {"name":"hw_3g_cell_index", "type":["null","string"]}, {"name":"hw_3g_reliability", "type":["null","string"]}, {"name":"vsrabattestabamr", "type":["null","string"]}, {"name":"vsrabsuccestabcsamr", "type":["null","string"]}] }

mustafa_qizilba · ‎09-15-2018

@Shu It got successful but out out was empty. Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR CSVWriter Properties Avro Schema UpdateRecord Property Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR

mustafa_qizilba · ‎09-14-2018

@Shu UpdateRecord Error: Can not write to schema, schema is unknow. UpdateRecord Properties CSV Reader CSV Writer

mustafa_qizilba · ‎09-14-2018

@Shu Data coming in ConvertRecord from CSV to AVRO schema. Avro_Schema Outgoing Data As you can see, incoming data have column name in different format. I changed the Avro Schema to reflect the same column name format but then it loaded NULL values in all the columns.

mustafa_qizilba · ‎09-13-2018

@Shu We have headers coming in multiple sized like in lower and upper case. As we are converting CSV to Json, we are first converting the csv schema into Avro Schema where avro is not accepting upper case column name. Kindly advice how to convert upper case into lower case. Thanks in advance.

mustafa_qizilba · ‎09-13-2018

Many thanks, it worked.

mustafa_qizilba · ‎09-13-2018

@Shu Followed and implemented but getting the error at the same place where you are showing in your snapshot. Kindly advice how to fix. UpdateAttribute_Header UpdateAttribute_Data

mustafa_qizilba · ‎09-13-2018

@Shu Thanks, the merged worked but header came as a last row. How to prioritize the header FlowFile to come top as data header (as column names)

mustafa_qizilba · ‎09-13-2018

@Shu @Matt Burgess Hi, We have 2 files coming from different locations. 1) First one is coming containing Header (column names only) 2) Second one is having Data, in the same column sequence. Aim: We need to merge both in one output file, where Header comes on top (in first row) and data from 2 row on wards. Looking forward. Cheers

mustafa_qizilba · ‎09-08-2018

@Shu @Matt Burgess @Geoffrey Shelton Okot Hi, Requirement: How to find and then replace multiple words using ONE ReplaceText processor Scenario: We have a .csv file coming from source with 2500 columns and, NOTE, the placement of columns are not fix. We need to change names of the columns We have achieved it by using one ReplaceText processor, each, to search for each column name, as it can come any where, and then replace it with requirement column name. Aim: Aim is to use ONE ReplaceText processor to achieve above i.e find all the columns and replace all in one go. Looking forward for a quick solution. Cheers

mustafa_qizilba · ‎09-02-2018

@Steven Matison These SerDe(s) are not working with Hive3 in HDP3.0 Did you tested these with Hive3 or previous versions?

mustafa_qizilba · ‎08-26-2018

@Shu @Matt Burgess @Geoffrey Shelton Okot Scenario: We have N number of files, which wont be fixed, but with the help of naming convention we pick few files following files as 1 file type. For example, we have 4 files with structures like below; and are called as 1 type of files. File1: Col1 (mandatory and fix location), Col2, Col3, Col4 File2: Col1 (mandatory and fix location), Col2, Col3, Col4, Col5 File3: Col1 (mandatory and fix location), Col2, Col5, Col4, Col3 File4: Col1 (mandatory and fix location), Col2, Col3, Col6 Means, Col1 (Pkey columns) will always be first but rest of the Columns can change its position, can come or can be missed out in different coming files. We need to come-up with a solution where we should be able to pull 1 type of files in one go and able to load all in 1 target Hive table. We can create Hive table, having all the possible columns, upfront like from Col1 to Col6. Our process should write all the files, having same file type, into 1 target file or 1 target hive table. Option of Implementation; We merge all the files in 1 file type and create 1 target file, in local directory, with all the columns and data as well and then load that 1 file into hive table or in HDFS as file which can then be read via Hive external table. We pull all the files in 1 file type, SPARK and place those in RDD/ data frames, and then create 1 target RDD which will keep a copy in SPARK and also push the data into Hive table as well. Not sure how to achieve that using NiFi, via SCALA scripting we have achieved it. If anyone can help us to convert SCALA script into NiFi, that will be of great help also. Ideal option is to us NiFi which pulls all the files of 1 file type and on the fly, it do the thing and load into Hive table or in HDFS as file which can then be read iva Hive external table. Looking forward.

mustafa_qizilba · ‎08-26-2018

@Shu @Geoffrey Shelton Okot @Matt Burgess Hi, We have JSON and XML files in HDFS and need both SerDe(s), so we can use Hive External Tables to load into Hive Managed Partitioned Tables via NiFi. Environment: HDP 3.0 (hive3) HDF 3.2 Tableau Looking forward.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

@Shu Can it be a permission issue to create Partition directories by Nifi as loading in non-partition table is not an issue. The only issue we are facing after loading in non-partition table is, we are unable to see the data from command line where can see data in HDFS hive directory, well we will raise this issue in different thread. As far as this thread is concern, NiFi --> PutHiveStreaming is unable to load in Hive Partitioned table. Please advice what to check next, really appreciate your help here @Shu.

Online	Offline
Last Visited	‎11-06-2018 09:39 AM

Date Registered	‎01-03-2018 02:41 PM
Last Visited	‎11-06-2018 09:39 AM
Total Messages Posted	89
Total Kudos Received	3

Anyone is using LLAP with Tableau: Issue Viewing D...

Re: Tableau vs LLAP & Non-LLAP Connection Error

Tableau vs LLAP & Non-LLAP Connection Error

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: Gettring Error while Installing Hortonworks HD...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

HDP 3.0 Error: Cannot find a valid baseurl for rep...

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

NiFi: Convert data into lower case

Re: NiFi: Merge Header and Data coming from differ...

Re: NiFi: Merge Header and Data coming from differ...

Re: NiFi: Merge Header and Data coming from differ...

NiFi: Merge Header and Data coming from different ...

NiFi: How to find and then replace multiple words ...

Re: Hive3: JSON and XML SerDe?

NiFi: Merge and Append Files (unique requirement)

Hive3: JSON and XML SerDe?

Re: NiFi: How to load data into Hive Partitioned T...

Re: Failed connecting to EndPoint while using PutH...

Re: NiFi: Pull data from Hive External table and l...

Re: PutHiveStreaming Nifi processor; various error...

Re: NiFi processor putHiveStreaming exception.

Re: Permission Error: To load into Hive using NiFi

Re: Permission Error: To load into Hive using NiFi