Re: Tableau vs LLAP & Non-LLAP Connection Error

mustafa_qizilba · ‎10-28-2018

@Geoffrey Shelton Okot @Matt Burgess @Shu

mustafa_qizilba · ‎10-26-2018

Scenaria: Run a query on a table e.g. emp in Hive using port: 10,000 i.e. non-llap HiveServer2 Connect and pull data in Tableau from same table e.g. emp using port: 10,500 i.e. llap HiveServer2 Environment: HDP 3.0 (Hive 3) Tableau Desktop (latest) Error/ Issue: Run query in Hive (using port: 10,000) then Tableau dashboards (using port: 10,500) doesn't refresh and hangs. Then close hive and refresh Tableau dashboards (using port: 10,500), its refreshes fine but now when run query in hive (using port: 10,000), hive hangs. Conclusion: Issue is, we are unable to connect with hive and tableau as the same time. Please advice.

mustafa_qizilba · ‎09-29-2018

@Jay Kumar SenSharma So we moved ahead and used another IP: XX.XX.XX.51 (which has internet access) to install HDP and then reverted HDP back to its original IP: XX.XX.XX.160 (which does't have internet access) and did the same to install HDF using IP: XX.XX.XX.51 (which has internet access) and then reverted HDF back to its original IP: XX.XX.XX.161 (which doesn't has internet access). When HDP and HDF were on IP: XX.XX.XX.51, all the services were up and running and restart-able but the moment those were reverted back i.e. HDP and HDF to those original IPs, everything got hung. Now services are in start mode as were when were on IP: XX.XX.XX.51 and we can't stop & restart and other things like hive is not accessible, NiFi UI does't appear etc. Please advice.

mustafa_qizilba · ‎09-23-2018

@Jay Kumar SenSharma Yup permissions are good. The problem is, there is no index.html file present in/repo/repodata directory.

mustafa_qizilba · ‎09-23-2018

@Jay Kumar SenSharma Hi, Apologies for delay. There you go with WebServer Log. -------------------------------------------------------------------------------------------------------------------------------------------- [root@ufm httpd]# tail error_log [Sun Sep 23 16:18:17.112853 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path [Sun Sep 23 16:18:19.399155 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/ambari/centos7/index.html denied (filesystem path '/var/www/html/repo/ambari/centos7/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ [Sun Sep 23 16:18:20.584500 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:22.762022 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:23.613018 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 16:18:41.563016 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53473] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/ [Sun Sep 23 18:11:29.286927 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53653] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path [Sun Sep 23 18:13:02.298494 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53660] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP/ [Sun Sep 23 18:14:16.053361 2018] [core:error] [pid 1780] (13)Permission denied: [client10.200.40.51:53663] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-UTILS/ [Sun Sep 23 18:15:56.767202 2018] [core:error] [pid 1781] (13)Permission denied: [client10.200.40.51:53666] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-GPL/ -------------------------------------------------------------------------------------------------------------------------------------------- Note: As this is offline line installation, we don't have index.html any where. Please advice.

mustafa_qizilba · ‎09-18-2018

@SANJAY RAI Any resolve on this issue, as i am facing same?

mustafa_qizilba · ‎09-18-2018

@Jay Kumar SenSharma @Geoffrey Shelton Okot It seems, repodata/repomd.xml is the culprit. Please advice its uses and content with no-internet access as its content is; <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> I feel above mentioned URLs should not go on internet.

mustafa_qizilba · ‎09-18-2018

@Jay Kumar SenSharma We moved ahead by disabling rest the repos and enabling ambari one but got stuck at cluster building. Note: When we paste this URL: http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml it does work. Following failed at installing this service: Timeline Service V1.5 Install failure: repodata/repomd.xml from HDP-3.0: [Errno 256] No more mirrors to try. http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable 2018-09-18 16:05:51,775 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {} 2018-09-18 16:05:51,805 - call returned (0, '') 2018-09-18 16:05:51,805 - The 'hadoop-yarn-timelineserver' component did not advertise a version. This may indicate a problem with the component packaging. Command failed after 1 tries I AM SURE, WE DO NOT NEED INTERNET as a mandatory pre-requisite?

mustafa_qizilba · ‎09-18-2018

@Geoffrey Shelton Okot @Shu @Matt Burgess Hi All, We are facing issue installing using Local Repositories as don't have internet access in our environment. This issue is pending since last 2 weeks and we are unable to crack this via any forum or community. Error: Cannot find a valid baseurl for repo: base/7/x86_64 ------------------------------------------------- Environment: VMWare - ESXI - done CenOS 7 - done Java 8+ - done MySQL 5.7 - done Files updated - done /etc/hosts /etc/sysconfig/network /etc/hostname /etc/selinux/config /etc/yum/pluginconf.d/refresh-packagekit.conf /etc/profile .bash_profile Repo Directories and rpm placement - done /var/www/html/repo/ambari /var/www/html/repo/HDP-UTILS /var/www/html/repo/HDP /var/www/html/repo/HDP-GPL Repo files creation - done /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/hdp.repo /etc/yum.repos.d/hdp.gpl.repo Services - done systemctl stop firewalld systemctl start ntpd systemctl start httpd systemctl status mysqld.service (all required users and databases are created along with also ran: Ambari-DDL-MySQL-CREATE.sql in DB: Ambari Amabri Server: Pending Ambari Agent: Pending HDP 3.0: Pending ----------------------------------------------------------------------------------- Issue: Using local repositories, whenwe run this command: yum install ambari-server following error hit us. Error: [root@ufm yum.repos.d]# yum install ambari-server Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was 14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error" One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: base/7/x86_64 ----------------------------------------------------------------------------------- Note: Above whole installation was achieved on Local VM enviroments but not on ESXI (this is our first time on EXSI environment) Bare Metal Servers Limitation: We do not have internal access in our environment, so have to use local repositories. Looking forward. Cheers

mustafa_qizilba · ‎09-15-2018

@Shu pmresult-67109368-60-201804260000-201804260100-ori.zip (Please unzip, its a csv file) sample-xml.xml Same result :-(. Also attaching XML and Source data file for testing. Empty Queue Avro Schema: { "type":"record", "name":"jazz", "fields":[ {"name":"hw_3g_result_time", "type":["null","string"]}, {"name":"hw_3g_granularity_period", "type":["null","string"]}, {"name":"hw_3g_bsc_name", "type":["null","string"]}, {"name":"hw_3g_bsc_type", "type":["null","string"]}, {"name":"hw_3g_cell_name", "type":["null","string"]}, {"name":"hw_3g_cell_id", "type":["null","string"]}, {"name":"hw_3g_cell_index", "type":["null","string"]}, {"name":"hw_3g_reliability", "type":["null","string"]}, {"name":"vsrabattestabamr", "type":["null","string"]}, {"name":"vsrabsuccestabcsamr", "type":["null","string"]}] }

mustafa_qizilba · ‎09-15-2018

@Shu It got successful but out out was empty. Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR CSVWriter Properties Avro Schema UpdateRecord Property Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR

mustafa_qizilba · ‎09-14-2018

@Shu UpdateRecord Error: Can not write to schema, schema is unknow. UpdateRecord Properties CSV Reader CSV Writer

mustafa_qizilba · ‎09-14-2018

@Shu Data coming in ConvertRecord from CSV to AVRO schema. Avro_Schema Outgoing Data As you can see, incoming data have column name in different format. I changed the Avro Schema to reflect the same column name format but then it loaded NULL values in all the columns.

mustafa_qizilba · ‎09-13-2018

@Shu We have headers coming in multiple sized like in lower and upper case. As we are converting CSV to Json, we are first converting the csv schema into Avro Schema where avro is not accepting upper case column name. Kindly advice how to convert upper case into lower case. Thanks in advance.

mustafa_qizilba · ‎09-13-2018

Many thanks, it worked.

mustafa_qizilba · ‎09-13-2018

@Shu Followed and implemented but getting the error at the same place where you are showing in your snapshot. Kindly advice how to fix. UpdateAttribute_Header UpdateAttribute_Data

mustafa_qizilba · ‎09-13-2018

@Shu Thanks, the merged worked but header came as a last row. How to prioritize the header FlowFile to come top as data header (as column names)

mustafa_qizilba · ‎09-13-2018

@Shu @Matt Burgess Hi, We have 2 files coming from different locations. 1) First one is coming containing Header (column names only) 2) Second one is having Data, in the same column sequence. Aim: We need to merge both in one output file, where Header comes on top (in first row) and data from 2 row on wards. Looking forward. Cheers

mustafa_qizilba · ‎09-02-2018

@Steven Matison These SerDe(s) are not working with Hive3 in HDP3.0 Did you tested these with Hive3 or previous versions?

mustafa_qizilba · ‎08-26-2018

@Shu @Geoffrey Shelton Okot @Matt Burgess Hi, We have JSON and XML files in HDFS and need both SerDe(s), so we can use Hive External Tables to load into Hive Managed Partitioned Tables via NiFi. Environment: HDP 3.0 (hive3) HDF 3.2 Tableau Looking forward.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.

mustafa_qizilba · ‎08-21-2018

@Shu Can it be a permission issue to create Partition directories by Nifi as loading in non-partition table is not an issue. The only issue we are facing after loading in non-partition table is, we are unable to see the data from command line where can see data in HDFS hive directory, well we will raise this issue in different thread. As far as this thread is concern, NiFi --> PutHiveStreaming is unable to load in Hive Partitioned table. Please advice what to check next, really appreciate your help here @Shu.

mustafa_qizilba · ‎08-21-2018

@Shu Still same error. Done following as per your direction. Reduced Table Name From: UFM_Intra_Cell_Handover_Measurement_Per_Cell To: ufmichmpc There is only 1 row in the source table for testing, so SelectHiveQL processor Max rows flowfile set to 0 is not an issue, anymore. There is only 2 columns in target table for testing Column#1: HW_2G_Result_Time Column#2: HW_2G_BSC_Name (this columns is partition column) Converted all columns data types to string, so now source and target columns datatypes both are string. SelectHiveQL processor has this query: select HW_2G_Result_Time, HW_2G_BSC_Name from ext_UFM_Intra_Cell_Handover_Measurement_Per_Cell. Please advice.

mustafa_qizilba · ‎08-21-2018

Dear @Shu Please find screenshots.

mustafa_qizilba · ‎08-21-2018

We are trying to load data from Hive External table to Hive Managed table and facing Hive permission issues. NiFi uses user: nifi to load into hive that's where its crashing. Note: Avro files are been created after pulling from source Hive External table. Please refer to the screenshot attached.

Online	Offline
Last Visited	‎11-06-2018 09:39 AM

Date Registered	‎01-03-2018 02:41 PM
Last Visited	‎11-06-2018 09:39 AM
Total Messages Posted	89
Total Kudos Received	3

Re: Tableau vs LLAP & Non-LLAP Connection Error

Tableau vs LLAP & Non-LLAP Connection Error

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: Gettring Error while Installing Hortonworks HD...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

Re: HDP 3.0 Error: Cannot find a valid baseurl for...

HDP 3.0 Error: Cannot find a valid baseurl for rep...

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

Re: NiFi: Convert data into lower case

NiFi: Convert data into lower case

Re: NiFi: Merge Header and Data coming from differ...

Re: NiFi: Merge Header and Data coming from differ...

Re: NiFi: Merge Header and Data coming from differ...

NiFi: Merge Header and Data coming from different ...

Re: Hive3: JSON and XML SerDe?

Hive3: JSON and XML SerDe?

Re: NiFi: How to load data into Hive Partitioned T...

Re: Failed connecting to EndPoint while using PutH...

Re: NiFi: Pull data from Hive External table and l...

Re: PutHiveStreaming Nifi processor; various error...

Re: NiFi processor putHiveStreaming exception.

Re: Permission Error: To load into Hive using NiFi

Re: Permission Error: To load into Hive using NiFi

Re: Permission Error: To load into Hive using NiFi

Re: Permission Error: To load into Hive using NiFi

Permission Error: To load into Hive using NiFi