I have a table in Phoenix used for ETL auditing. This table has a composite key of three columns. I have recently add a secondary index on a fourth column to speed up queries on this column. However I have noticed an odd behaviour: on all rows that were in the table before the index was created, columns other than the key columns and the index are <null>. For all rows inserted into the table after the creation of the index, all columns are populated. Since this is a write heavy table (~500 000 rows inserted per day), I have used a local index. I have two questions: 1) Is this expected behaviour for a Phoenix index, or does it indicate an error on my part? 2) How can I cause the <null> fields to be populated? I am willing to recreate the index if necessary ... View more

I am currently experiencing an issue with HDFS storage. We have (intentionally) deleted the majority of the data on our cluster. According to hdfs du, the total usage on HDFS is approximately 1TB. However, Ambari reports that the DFS used is 238.9TB. I can understand a small discrepancy here, for blocks that have not been deleted and such. However such a huge difference is worrying. On top of this, a huge number of the underlying disks are at 100% full, and no amount of HDFS balancing changes this. HDFS has been incredibly unstable over the past few weeks, and it's possible that this is the underlying cause. Is there any way I can safely clear this space? I don't mind losing data (we are repurposing this cluster) as long as HDFS remains stable. The issue with the full disks is my bigger concern, but improving that should also clear a lot of the falsely reported space on HDFS. Any advice will be appreciated. ... View more

Our cluster recently started having a problem with one of our ZooKeeper servers - this server will consistently refuse any connections made to ZooKeeper, which has lead to some problems in our cluster. This is not a network issue, as all other connections are successful. Our investigation has identified that ZooKeeper CPU usage on this server is absurdly high - ranging from 300% to 2200% CPU usage. For comparison, the other servers in the Quorum rarely show ZooKeeper in top at all, and when the do the CPU usage is <1%. A misconfiguration of this ZooKeeper server seems unlikely, since the cluster is managed through Ambari - all ZooKeeper servers should have exactly the same configurations. We have restarted ZooKeeper on this machine multiple times with no improvement. Even restarting the physical host did not cause any improvement. We are having authentication issues on that machine which may contribute to the issue. However the zookeeper user is accessible, and all commands through Ambari are successful, with no permission denied errors. Some possibly relevant information: We are running HDP-2.6.0.3, with ZooKeeper 3.4.6 Our Quorum size is three Does anybody have any suggestions for how this can be improved or resolved? ... View more

We are having an issue with running out of disk space on HDFS. A little investigation has shown that the largest directory, by far, is /apps/hbase/data/archive. As I understand it, this directory keeps HFiles that need to be kept, typically because of snapshots. I know that a large archive directory is the first culprit of having too many snapshots. However snapshots do not seem to be the issue here: /apps/hbase/data/archive is a little larger than110TB, while the sum of all of our snapshots is <50TB. We have not set hbase.master.hfilecleaner.ttl, however I have read that the default is 5 minutes - this is definitely not the culprit for many of the HFiles we have, which frequently date back many months. What steps can I follow to try to reduce this usage? ... View more

I am currently in the process of setting up replication on HBase. Normal HBase replication (based on the WALs) is set up and working well. However I am struggling to set up replication for bulk loading as specified in [HBASE-13153] I am running HDP 2.6. I initially thought the version of HBase was the issue, but according to release notes this was patched in in HDP 2.4.3 so HBase version is no the issue here. I have followed the instructions provided in the link as well as I can understand them, but bulk loaded rows are not being replicated. I have set the following configurations: On the source cluster: hbase.replication.bulkload.enabled=true On the source cluster: hbase.replication.cluster.id=source On the peer cluster: hbase.replication.conf.dir=/tmp/fs_conf On the peer cluster: hbase.replication.source.fs.conf.provider=org.apache.hadoop.hbase.replication.regionserver.DefaultSourceFSConfigurationProvider I have copied the core-site.xml, hdfs-site.xml, yarn-site.xml, hbase-site.xml from the source cluster to the peer cluster, to all of the region servers under /tmp/fs_conf/source These are the only changes specified in the JIRA (at least as far as I understand it) however the rows that are bulk loaded are not being replicated. I suspect I have missed/misinterpreted part of the instructions. Any help will be appreciated. ... View more

I am using spark-phoenix integration to load data from a dataframe into a Phoenix table. Unfortunately this is ridiculously slow - pushing 23 rows of 25 columns each takes 7-8 seconds. This is with two executors, meaning it's actually twice as slow. This makes it unusable in my case, since it is planned to use in a streaming application - the number of records dropped in a 15 second window take up to a minute to load. When I look at the Spark History Server, I see two really strange things: The slowest part, by far, is 'saveAsNewAPIHadoopFile at DataFrameFunctions.scala:55', so the problem is not in my code. For 23 rows, 200 tasks are launched. This seems excessive. Does anybody have experience with how I could improve these loading speeds? Ideally I want to keep using Phoenix in some way, since we have secondary indexes on the table. ... View more

I'm experimenting with broadcast variables in PySpark at the moment, and I've noticed that whenever I create an explicit Java object using sc._jvm, I get errors when I try to broadcast these variables. Looking at the stack trace, the problem seems to be related to pickling. Does anybody know how I can broadcast such variables? ... View more

I have a Spark ETL process which reads from a CSV file into an RDD, performs some transformations and data quality checks, converts this into a DataFrame and pushes the DataFrame into Phoenix using Spark-Phoenix integration. Unfortunately, actually pushing the data to Phoenix is ridiculously slow - the saveAsNewAPIHadoopFile at DataFrameFunctions.scala:55 portion kicks off 120 tasks (which seems quite high for a file of ~3.5GB). Each task then runs between 15 minutes and 1 hour. I have 16 executors with 4GB of RAM each, which should be more than sufficient for the job. The job has currently been running for over two hours and will probably run for another hour or more, which is very long to push only 5.5 million rows. Does anybody have any insight into a) why this is so slow and b) how to speed it up? Thanks in advance. Edit 1: The job has completed in 4 hours. ... View more

I am currently developing a spark streaming application in Python to watch a directory and load new files into a Phoenix table. This part is fairly straightforward using Spark-Phoenix integration. However, for auditing purposes I need to keep track of what files are being loaded, and how many records come from each of these files. The textFileStream does not record the filename in elements of the RDD, and the only way I have found so far to get the filename is by getting the RDD's debug string. While this does give me the filename, it in no way indicates which elements belong to which files. What would be ideal is something like SparkContext's wholeTextFiles(...), which gives back and RDD of the form (filename, contents), while textFileStream acts more like SparkContext's textFiles(...) Is there any reasonable way to determine which elements belong to which files? ... View more

Today I encountered an error with NiFi that I have never seen before. We have been successfully running NiFi, managed by Ambari, for a number of months now. This morning while working in NiFi, the UI suddenly died. When refreshing the UI, I get ERR_CONNECTION_REFUSED. I have tried this in both Chrome and Explorer, so it is not a browser specific issue. I have now spent the morning trying to solve this issue. I can give the following info: NiFi is managed by Ambari. According to Ambari NiFi is up NiFi is SSL secured. This is not a security issue, since when switching off SSL Authentication and restarting Ambari, the non-secure UI gives the same issue Our network team has confirmed that this is not a firewall issue - port 9091 is not blocked, but is refusing all connections. selinux is disabled ps -ef | grep nifi shows NiFi is up and running on the machine bin/nifi.sh status shows 2017-06-06 11:34:47,657 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 51190, PID=13513 nifi-app.log stops writing anything shortly after startup. The last line is 2017-06-06 11:16:28,561 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded a total of 96 properties. Including precedence overrides effective accessible registry key size is 96 (That was nearly half an hour ago. Nothing since) I have restart NiFi a number of times trying to solve this, but this has no effect. Any help would be greatly appreciated. ... View more

Yesterday I performed a number of region mergers on one of our larger tables. This table had ~1400 regions when starting, but many of these were small regions - we wanted to get the average region size closer to our region limit size of 15GB. The region mergers went well, leaving us with just over 700 regions. However since then we have had a huge number of offline regions - nearly 600 regions are currently listed as offline. Does anybody know the cause of these offline regions, and how to fix them? I understand regions do sometimes go offline legally, but this many seems to me to indicate a problem. ... View more

We recently had a failure of all of the region servers in our cluster, although the active master and standby master stayed up. When the region servers were brought back up regions were assigned relatively quickly. However two regions have not come up, and according to the UI they are stuck in the OFFLINE state. I have tried running hbase hbck -repair a number of times, as well as various other options that I hoped would help (-fixAssignments, -fixSplitParents). None of these successfully brought the regions online. I check the logs of the region servers for the regions and there was no reference to them after they were closed prior to region server failure. When I checked the master logs however I found the following: master.AssignmentManager: Skip assigning table_name,13153,1485460927890.3d68e485cb6294345fe1469097fa5aca., it is on a dead but not processed yet server: server05,16020,1494493877392 The server listed as a dead server is alive and well, with over 200 regions already assigned to it. This error message led me to HBASE-13605, HBASE-13330 and HBASE-12440 which all describe pretty much the same issue. Unfortunately none of these JIRAs describe any way to fix the issue once it occurs. Does anybody have any advice for resolving this? This is a production system and so shutting down the master is a last resort. ... View more

We have a cluster running HDP 2.4 with 8 worker nodes. Recently two of our datanodes go down frequently - usually they both go down at least once a day, frequently more often than that. While they can be started up without any difficulty, they will usually fail again within 12 hours. There is nothing out of the ordinary in the logs except very long GC wait times before failure. For example, shortly before failing this morning, I saw the following in the logs: 2017-04-25 03:49:27,529 WARN util.JvmPauseMonitor (JvmPauseMonitor.java:run(192)) - Detected pause in JVM or host machine (eg GC): pause of approximately 23681ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=23769ms I checked the free memory on that node, and it was slightly more than the free memory on a similar node that isn't shutting down. Since it is the same two nodes repeatedly, I assume the problem is something to do with the nodes themselves. Does anybody have any advice for this problem? ... View more

I am currently having a serious problem with the output file for zookeeper. I recently noticed that /var/log had 15G full, but du -h /var/log only reported less than 1G. I checked my deleted files (lsof | grep deleted | grep /var/log) I noticed that there are a number of log files that are deleted, but still open. The most concerning of these is /var/log/zookeeper/zookeeper-zookeeper-server-xxxx.out, which is over 13G in size. In spite of being deleted the file is still open. Our system admin suggested restarting Zookeeper to clear the file lock - unfortunately this is a production cluster and so restarting Zookeeper is an absolute last resort. I have come up with some possible solutions: Since we have a quorum of 3 servers, would it be realistic to restart only the Zookeeper server on the machine that is giving the problem? If that is not realistic, can I truncate the file (as per this article) to clear the space issue without causing problems to Zookeeper? If neither of these is possible, what are my other options? Thanks in advance. ... View more

I am using HBase snapshots for the purpose of backups in my cluster. I have weekly snapshots to facilitate recovery from HBase failure. However something concerns me. I was under the impression that HBase snapshots stored only metadata without replicating any data, making them ideal for low footprint backups. However, after a short time (3+ weeks) a snapshot will often be exactly the same size as the source table, sharing 0% of the data with the source table. This is a problem since it means that keeping even a few weeks of backups can consume 25+ TB of space. Can anybody explain to me why this happens, and if there is any way to avoid it? ... View more

I have set up my NiFi instance (NiFi 1.0) to communicate over SSL and authenticate the admin user with a certificate, based on this article. This has worked correctly. Now I would like to add new users to NiFi, and have those users authenticate using username/password. I know this is possible with LDAP, as well as with Ranger and Kerberos, however I would prefer to manage my users directly through NiFi. Adding new users is straightforward, but there is no way I can see to set a password for a user. Is there any way to achieve this? I know it is possible to create my own LoginIdentityProvider. Does anybody have an example of code that could do what I want? ... View more

I am currently having a timeout on a Hive query over and HBase table, similar to this article. As per the instructions in that article, I intend to increase the HBase timeout, including the RPC timeout. However before going ahead, I would like to understand the potential consequences of this change on HBase and external services that access HBase, since this is a production cluster. If someone could give me an idea of what negative consequences this might have it would help a lot. ... View more