Getting this Error : PXF service could not be reached. PXF is not running in the tomcat container (libchurl.c:878) when trying to read data from Hive table in Hawq through PXF

[#1]

Hi,

I'm getting the below error when trying to read data from a Hive table in Hawq using pxf:

PXF service could not be reached. PXF is not running in the tomcat container (libchurl.c:878)

Please note tat I've installed pxf-service on all the nodes successfully and verified that the the pxf service is running on the hosts on port 51200.

Please suggest what could be the possible reason for this error.

gpadmin=# select count(*) from hcatalog.default.sample_07;

ERROR: remote component error (404) from

'10.131.137.16:51200': PXF service could not be reached. PXF is not running in the tomcat container (libchurl.c:878) LINE 1: select count(*) from hcatalog.default.sample_07;

gpadmin=# select * from hcatalog.default.sample_07 limit 1;

ERROR: remote component error (404) from '10.131.137.16:51200': PXF service could not be reached. PXF is not running in the tomcat container (libchurl.c:878) LINE 1: select * from hcatalog.default.sample_07 limit 1;

I checked that the pxf service is running on all the hosts as below:

Last login: Mon Feb 13 18:08:53 2017 from in-cppy5z1.in.kronos.com

[root@kvs-in-hadoop04 ~]# netstat -anp | grep 51200

tcp6 0 0 :::51200 :::* LISTEN 519018/java

[root@kvs-in-hadoop04 ~]# ps -ef | grep 519018

pxf 519018 1 0 15:57 ? 00:00:31 /usr/jdk64/jdk1.8.0_77/bin/java -Djava.util.logging.config.file=/var/pxf/pxf-service/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx512M -Xss256K -Djava.endorsed.dirs=/var/pxf/pxf-service/endorsed -classpath /var/pxf/pxf-service/bin/bootstrap.jar:/var/pxf/pxf-service/bin/tomcat-juli.jar -Dcatalina.base=/var/pxf/pxf-service -Dcatalina.home=/var/pxf/pxf-service -Djava.io.tmpdir=/var/pxf/pxf-service/temp org.apache.catalina.startup.Bootstrap start root 662021 661908 0 20:41 pts/0 00:00:00 grep --color=auto 519018

I'm also attaching the segment log files from the host where Hive is installed.hawq-2017-02-13-170046.txt

@njayakumar @Greg Keys @Gagan Brahmi @Artem Ervits

[#2]

I'm seeing the below lines on the host where Hive and Hawq segment is installed under the segment logs in pg_log:

2017-02-13 21:56:39.098974 IST,,,p572743,th-427955904,,,,0,,,seg-10000,,,,,"LOG","00000","Resource manager discovered local host IPv4 address 127.0.0.1",,,,,,,0,,"network_utils.c",210, 2017-02-13 21:56:39.099010 IST,,,p572743,th-427955904,,,,0,,,seg-10000,,,,,"LOG","00000","Resource manager discovered local host IPv4 address 10.131.137.16",,,,,,,0,,"network_utils.c",210,

I'm wondering why is it showing 127.0.0.1

Thoughts anyone?

[#3]

Hi Shikhar ,

Can you confirm the below

1.Are you able to query a normal pxf table (other than hctalog or hive one ) .You can create an external table in hdfs

2.Please stop the pxf services in all the nodes and kill if there are any orphan processes to the 51200 port .We need to see why the tomcat server is not spawning up .

3.Once you clear all the processes start pxf services in all the nodes and query the table back .

4.Please make sure you have installed hive-clients in all the pxf nodes to access hive data .

5.Also see whether you need to set pxf_service_address point to the hive metstore

[#4]

> 5.Also see whether you need to set pxf_service_address point to the hive metstore

Shouldn't this be pointed at the namenode host:port or namenode host:51200?

[#5]

Hi Prathneesh,

Thanks for your help

Please find my responses below:

1) Yes able to query a normal external table created using a HDFS file

2) Checked and killed all pxf processes on all servers. Ambari showed PXF agents as down, then restarted all the PXF agents via ambari.

3) Done, same error repeated

4) Yes a total of 11 PXF and 11 Hive clients installed on all the nodes.

5) pxf_service_address is set to hivenode:51200 and also hcatalog_enable=true. Both these properties set under custom hawq-site.xml.

I've followed all the above steps but not able to get past the error.

Any suggestions/ pointers are welcomed.

Thanks

Shikhar

[#6]

If necessary, set the pxf_service_address global configuration property to a hostname or IP address and port where you have installed the PXF Hive plug-in. By default, the value is set to localhost:51200.

SET pxf_service_address TO "hivenode:51200"

[#7 · Accepted solution]

Try removing /var/pxf, then run "service pxf-service init" on every HAWQ/PXF host

[#8]

Hi Kyle,

This worked for me!

But could you please throw some light on how did you get to this solution and what was the actual issue.

Thanks

Shikhar

[#9]

Shikhar - I actually experienced the exact same issue - for me it was due to an incomplete PXF upgrade. Initially I noticed (on my system) the PXF RPMs were at a different build version than HAWQ was - this was the first thing I fixed. I then explored a number of dead ends, ultimately noticing the timestamps of the files in /var/pxf could not possibly be the correct ones, given when I upgraded HAWQ to a newer version. After I confirmed these files were created at runtime, I removed the entire directory and re-ran the "service pxf-service init", which re-created that folder and nested files from the correct RPM version. After doing that, all was well in PXF land. 🙂

[#10]

Hello! I also encountered a similar problem with you, I would like to ask you, how did you finally solve it? Thank you very much!

,,