Member since
01-03-2018
89
Posts
3
Kudos Received
0
Solutions
10-28-2018
01:02 PM
@Geoffrey Shelton Okot @Matt Burgess @Shu
... View more
10-26-2018
11:30 AM
Scenaria:
Run a query on a table e.g. emp in Hive using port: 10,000 i.e. non-llap HiveServer2 Connect and pull data in Tableau from same table e.g. emp using port: 10,500 i.e. llap HiveServer2 Environment:
HDP 3.0 (Hive 3) Tableau Desktop (latest) Error/ Issue:
Run query in Hive (using port: 10,000) then Tableau dashboards (using port: 10,500) doesn't refresh and hangs. Then close hive and refresh Tableau dashboards (using port: 10,500), its refreshes fine but now when run query in hive (using port: 10,000), hive hangs. Conclusion: Issue is, we are unable to connect with hive and tableau as the same time. Please advice.
... View more
Labels:
- Labels:
-
Apache Hive
09-29-2018
11:03 PM
@Jay Kumar SenSharma So we moved ahead and used another IP: XX.XX.XX.51 (which has internet access) to install HDP and then reverted HDP back to its original IP: XX.XX.XX.160 (which does't have internet access) and did the same to install HDF using IP: XX.XX.XX.51 (which has internet access) and then reverted HDF back to its original IP: XX.XX.XX.161 (which doesn't has internet access). When HDP and HDF were on IP: XX.XX.XX.51, all the services were up and running and restart-able but the moment those were reverted back i.e. HDP and HDF to those original IPs, everything got hung. Now services are in start mode as were when were on IP: XX.XX.XX.51 and we can't stop & restart and other things like hive is not accessible, NiFi UI does't appear etc. Please advice.
... View more
09-23-2018
03:32 PM
@Jay Kumar SenSharma Yup permissions are good. The problem is, there is no index.html file present in/repo/repodata directory.
... View more
09-23-2018
02:12 PM
@Jay Kumar SenSharma Hi, Apologies for delay. There you go with WebServer Log. -------------------------------------------------------------------------------------------------------------------------------------------- [root@ufm httpd]# tail error_log
[Sun Sep 23 16:18:17.112853 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path
[Sun Sep 23 16:18:19.399155 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/ambari/centos7/index.html denied (filesystem path '/var/www/html/repo/ambari/centos7/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/
[Sun Sep 23 16:18:20.584500 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/
[Sun Sep 23 16:18:22.762022 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/
[Sun Sep 23 16:18:23.613018 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53472] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/
[Sun Sep 23 16:18:41.563016 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53473] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://10.200.40.160/repo/ambari/
[Sun Sep 23 18:11:29.286927 2018] [core:error] [pid 28765] (13)Permission denied: [client10.200.40.51:53653] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path
[Sun Sep 23 18:13:02.298494 2018] [core:error] [pid 29345] (13)Permission denied: [client10.200.40.51:53660] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP/
[Sun Sep 23 18:14:16.053361 2018] [core:error] [pid 1780] (13)Permission denied: [client10.200.40.51:53663] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-UTILS/
[Sun Sep 23 18:15:56.767202 2018] [core:error] [pid 1781] (13)Permission denied: [client10.200.40.51:53666] AH00035: access to /repo/repodata/index.html denied (filesystem path '/var/www/html/repo/repodata/index.html') because search permissions are missing on a component of the path, referer:http://ufm.hdp01.com/repo/HDP-GPL/ -------------------------------------------------------------------------------------------------------------------------------------------- Note: As this is offline line installation, we don't have index.html any where. Please advice.
... View more
09-18-2018
02:53 PM
@SANJAY RAI Any resolve on this issue, as i am facing same?
... View more
09-18-2018
02:50 PM
@Jay Kumar SenSharma @Geoffrey Shelton Okot It seems, repodata/repomd.xml is the culprit. Please advice its uses and content with no-internet access as its content is; <?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> I feel above mentioned URLs should not go on internet.
... View more
09-18-2018
11:31 AM
@Jay Kumar SenSharma We moved ahead by disabling rest the repos and enabling ambari one but got stuck at cluster building. Note: When we paste this URL: http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml it does work. Following failed at installing this service: Timeline Service V1.5 Install failure: repodata/repomd.xml from HDP-3.0: [Errno 256] No more mirrors to try.
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
http://ufm.hadoop.com/repo/HDP/centos7/3.0.0.0-1634/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
2018-09-18 16:05:51,775 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
2018-09-18 16:05:51,805 - call returned (0, '')
2018-09-18 16:05:51,805 - The 'hadoop-yarn-timelineserver' component did not advertise a version. This may indicate a problem with the component packaging.
Command failed after 1 tries I AM SURE, WE DO NOT NEED INTERNET as a mandatory pre-requisite?
... View more
09-18-2018
05:55 AM
@Geoffrey Shelton Okot @Shu @Matt Burgess Hi All, We are facing issue installing using Local Repositories as don't have internet access in our environment. This issue is pending since last 2 weeks and we are unable to crack this via any forum or community. Error: Cannot find a valid baseurl for repo: base/7/x86_64 ------------------------------------------------- Environment: VMWare - ESXI - done CenOS 7 - done Java 8+ - done MySQL 5.7 - done Files updated - done /etc/hosts /etc/sysconfig/network /etc/hostname /etc/selinux/config /etc/yum/pluginconf.d/refresh-packagekit.conf /etc/profile .bash_profile Repo Directories and rpm placement - done /var/www/html/repo/ambari /var/www/html/repo/HDP-UTILS /var/www/html/repo/HDP /var/www/html/repo/HDP-GPL Repo files creation - done /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/hdp.repo /etc/yum.repos.d/hdp.gpl.repo Services - done systemctl stop firewalld systemctl start ntpd systemctl start httpd systemctl status mysqld.service (all required users and databases are created along with also ran: Ambari-DDL-MySQL-CREATE.sql in DB: Ambari Amabri Server: Pending Ambari Agent: Pending HDP 3.0: Pending ----------------------------------------------------------------------------------- Issue: Using local repositories, whenwe run this command: yum install ambari-server following error hit us. Error: [root@ufm yum.repos.d]# yum install ambari-server Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error" One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: base/7/x86_64 ----------------------------------------------------------------------------------- Note: Above whole installation was achieved on Local VM enviroments but not on ESXI (this is our first time on EXSI environment) Bare Metal Servers Limitation: We do not have internal access in our environment, so have to use local repositories. Looking forward. Cheers
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
09-15-2018
12:07 PM
@Shu pmresult-67109368-60-201804260000-201804260100-ori.zip (Please unzip, its a csv file) sample-xml.xml Same result :-(. Also attaching XML and Source data file for testing. Empty Queue Avro Schema: {
"type":"record",
"name":"jazz",
"fields":[
{"name":"hw_3g_result_time", "type":["null","string"]},
{"name":"hw_3g_granularity_period", "type":["null","string"]},
{"name":"hw_3g_bsc_name", "type":["null","string"]},
{"name":"hw_3g_bsc_type", "type":["null","string"]},
{"name":"hw_3g_cell_name", "type":["null","string"]},
{"name":"hw_3g_cell_id", "type":["null","string"]},
{"name":"hw_3g_cell_index", "type":["null","string"]},
{"name":"hw_3g_reliability", "type":["null","string"]},
{"name":"vsrabattestabamr", "type":["null","string"]},
{"name":"vsrabsuccestabcsamr", "type":["null","string"]}]
}
... View more
09-15-2018
10:34 AM
@Shu It got successful but out out was empty. Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR CSVWriter Properties Avro Schema UpdateRecord Property Source Header: Aim is to convert mentioned last 2 columns in lower case. hw_3g_result_time,hw_3g_granularity_period,hw_3g_bsc_name,hw_3g_bsc_type,hw_3g_cell_name,hw_3g_cell_id,hw_3g_cell_index,hw_3g_reliability,VSRABAttEstabAMR,VSRABSuccEstabCSAMR
... View more
09-14-2018
06:09 PM
@Shu UpdateRecord Error: Can not write to schema, schema is unknow. UpdateRecord Properties CSV Reader CSV Writer
... View more
09-14-2018
11:51 AM
@Shu Data coming in ConvertRecord from CSV to AVRO schema. Avro_Schema Outgoing Data As you can see, incoming data have column name in different format. I changed the Avro Schema to reflect the same column name format but then it loaded NULL values in all the columns.
... View more
09-13-2018
10:36 PM
@Shu We have headers coming in multiple sized like in lower and upper case. As we are converting CSV to Json, we are first converting the csv schema into Avro Schema where avro is not accepting upper case column name. Kindly advice how to convert upper case into lower case. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
09-13-2018
10:25 PM
Many thanks, it worked.
... View more
09-13-2018
09:14 PM
@Shu Followed and implemented but getting the error at the same place where you are showing in your snapshot. Kindly advice how to fix. UpdateAttribute_Header UpdateAttribute_Data
... View more
09-13-2018
06:38 PM
@Shu Thanks, the merged worked but header came as a last row. How to prioritize the header FlowFile to come top as data header (as column names)
... View more
09-13-2018
04:23 PM
@Shu @Matt Burgess Hi, We have 2 files coming from different locations. 1) First one is coming containing Header (column names only) 2) Second one is having Data, in the same column sequence. Aim: We need to merge both in one output file, where Header comes on top (in first row) and data from 2 row on wards. Looking forward. Cheers
... View more
Labels:
- Labels:
-
Apache NiFi
09-02-2018
07:25 AM
@Steven Matison These SerDe(s) are not working with Hive3 in HDP3.0 Did you tested these with Hive3 or previous versions?
... View more
08-26-2018
05:00 AM
@Shu @Geoffrey Shelton Okot @Matt Burgess Hi, We have JSON and XML files in HDFS and need both SerDe(s), so we can use Hive External Tables to load into Hive Managed Partitioned Tables via NiFi. Environment: HDP 3.0 (hive3) HDF 3.2 Tableau Looking forward.
... View more
Labels:
- Labels:
-
Apache NiFi
08-21-2018
04:36 PM
Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
04:36 PM
Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
04:35 PM
Issue Resolved for me. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
04:34 PM
Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
04:33 PM
In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
04:33 PM
Issue Resolved. In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL. Cheers.
... View more
08-21-2018
02:18 PM
@Shu Can it be a permission issue to create Partition directories by Nifi as loading in non-partition table is not an issue. The only issue we are facing after loading in non-partition table is, we are unable to see the data from command line where can see data in HDFS hive directory, well we will raise this issue in different thread. As far as this thread is concern, NiFi --> PutHiveStreaming is unable to load in Hive Partitioned table. Please advice what to check next, really appreciate your help here @Shu.
... View more
08-21-2018
02:01 PM
@Shu Still same error. Done following as per your direction. Reduced Table Name From: UFM_Intra_Cell_Handover_Measurement_Per_Cell To: ufmichmpc There is only 1 row in the source table for testing, so SelectHiveQL processor Max rows flowfile set to 0 is not an issue, anymore. There is only 2 columns in target table for testing Column#1: HW_2G_Result_Time Column#2: HW_2G_BSC_Name (this columns is partition column) Converted all columns data types to string, so now source and target columns datatypes both are string. SelectHiveQL processor has this query: select HW_2G_Result_Time, HW_2G_BSC_Name from ext_UFM_Intra_Cell_Handover_Measurement_Per_Cell. Please advice.
... View more
08-21-2018
11:19 AM
Dear @Shu Please find screenshots.
... View more
08-21-2018
04:07 AM
We are trying to load data from Hive External table to Hive Managed table and facing Hive permission issues. NiFi uses user: nifi to load into hive that's where its crashing. Note: Avro files are been created after pulling from source Hive External table. Please refer to the screenshot attached.
... View more
Labels:
- Labels:
-
Apache NiFi