Member since
04-05-2016
188
Posts
19
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
461 | 10-30-2017 07:05 AM | |
694 | 10-12-2017 07:03 AM | |
2050 | 10-12-2017 06:59 AM | |
3858 | 03-01-2017 09:56 AM | |
14481 | 01-26-2017 11:52 AM |
11-19-2018
01:04 PM
@Geoffrey Shelton Okot Thanks for your response. My cluster has 11 nodes (3 master and 8 worker nodes). yes, i ran the balancer with a threshold of 5. I see it's still running from Friday morning... My Datanode:
/dev/sdb 5.4T 5.1T 17M 100% /grid/1
/dev/sdc 5.4T 5.1T 263M 100% /grid/2
/dev/sdd 5.4T 5.1T 912M 100% /grid/3
/dev/sde 5.4T 5.1T 283M 100% /grid/4
/dev/sdf 5.4T 5.1T 95M 100% /grid/5
/dev/sdg 5.4T 5.1T 388M 100% /grid/6
/dev/sdh 5.4T 5.1T 22G 100% /grid/7
/dev/sdi 5.4T 5.1T 694M 100% /grid/8
/dev/sdj 5.4T 5.1T 843M 100% /grid/9
/dev/sdk 5.4T 5.1T 36M 100% /grid/10
/dev/sdl 5.4T 5.1T 120M 100% /grid/11
/dev/sda 5.4T 5.1T 802M 100% /grid/0
tail of balancer output log:
18/11/19 12:12:02 INFO balancer.Dispatcher: Successfully moved blk_1107025919_33285238 with size=134217728 from nodeg:50010:DISK to nodeh:50010:DISK through nodeg:50010
18/11/19 12:12:02 INFO balancer.Dispatcher: Start moving blk_1107022998_33282317 with size=134217728 from nodeg:50010:DISK to nodeh:50010:DISK through nodeg:50010
18/11/19 12:12:07 INFO balancer.Dispatcher: Successfully moved blk_1107025997_33285316 with size=134217728 from nodeg:50010:DISK to nodeh:50010:DISK through nodeg:50010
18/11/19 12:12:07 INFO balancer.Dispatcher: Start moving blk_1107022634_33281953 with size=134217728 from nodeg:50010:DISK to nodeh:50010:DISK through nodej:50010
... View more
11-16-2018
11:30 AM
Hi, I am having some issues with rebalancing my HDF cluster (runs 2.6). There's a node whose data directory is 100% full. I used the hdfs balancer in Ambari and also ran the balancer command <hdfs balancer>. I have not seen any changes to the server node ... Please what's the way forward?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
11-12-2018
12:20 PM
Thanks @Isaac Arnault. I checked the documentation. However since the document says NiFi, Storm and Kafka cannot co-exist in the same node and the example given was a 19-node cluster, I wanted to know how i can achieve this with a 3-node cluster. General guidelines for production guidelines for service distribution:
NiFi, Storm, and Kafka should not be located on the same node or virtual machine.
NiFi, Storm, and Kafka must have a dedicated ZooKeeper cluster with at least three nodes.
If the HDF SAM is being used in an HDP cluster, the SAM should not be installed on the same node as the Storm
worker node.
... View more
11-12-2018
12:14 PM
thank you @Geoffrey Shelton Okot I am trying to upgrade to HDF 3.2.
... View more
11-12-2018
11:58 AM
I need to map YARN queues to analytic phoenix queries. That is, when user1 runs
"select col1, col2 from table1"
user1 runs it in yarn-queue1 and that way we can make sure adequate resources is given to the queue to avoid contention with lower priority jobs. However, I have tested with the yarn.scheduler.capacity.queue-mappings-override.enable=true yarn.scheduler.capacity.queue-mappings=u:user1:queue1,g:group1:queue2 parameters but i am not able to see the queries that are executed in YARN. How do i make sure these queries are catered for? N.B: We use squirrel to connect to Phoenix.
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache YARN
11-07-2018
05:01 AM
Hi @Bryan Bende @Matt Clarke I am currently running HDF 2.1.4 on a node as a cluster(256GB RAM) in production. We now have two more nodes to make three and want to install HDF 3.2. Would you please guide on how to make this work? I am only limited to 3 nodes...
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)
09-12-2018
01:13 PM
@rabbit s Reducing the memory specs for the spark executors will reduce the total memory consumed which should eventually allow for more jobs (new threads) to be spun...
... View more
09-12-2018
09:56 AM
@Jay Kumar SenSharma Yes, i see "hs_err_pid*" file. Please find attached the log. Also, the 2.22TB is the un-utilised RAM. hs-err-pid931.txt .
... View more
09-12-2018
09:15 AM
Hi, I am unable to launch more spark jobs on my cluster due to the error message below. I still have 2.22TB free according to YARN UI. I run HDP 2.6. #
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources. What's the way forward? @Jay Kumar SenSharma
... View more
Labels:
08-12-2018
09:42 PM
I find that ListFile processor fails to pick new files for a particular feed. Indeed, that feed has a lot of small files and totals about 500k per day. When i delete the old processor and recreate, it starts working again. What could be the problem? BTW i use HDF 2.1.4... Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
08-08-2018
07:29 AM
Thanks @Bryan Bende. Will try and revert back...
... View more
08-07-2018
06:02 AM
We currently use HDF-2.1.4.0-5 and will like for more AD users to be able to login to NiFi and create data flows themselves. Can you guide me on how to implement a user/password policy for each user. Currently, the AD users have been synced like we have in HDP 2.6 but i'm not sure how we'll get them to login to on NiFi with their AD accounts @Bryan Bende @Matt Clarke
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
07-23-2018
12:54 PM
Hi, I created an external table on HAR files but i'm not able to see meaningful records. Is this supported in Hive and what SerDe should be used? I currently run HDP 2.6
... View more
Labels:
- Labels:
-
Apache Hive
06-13-2018
02:51 PM
@Denise O Regan Did you manage to resolve this?
... View more
06-08-2018
06:42 AM
Has anyone successfully deployed HDP on isilon? I am having some errors...I'm using HDP 2.6 07 Jun 2018 14:45:22,450 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert ams_metrics_collector_process for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert yarn_resourcemanager_webui for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert oozie_server_status for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert yarn_app_timeline_server_webui for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert ams_metrics_collector_autostart for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert ambari_agent_disk_usage for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert hive_webhcat_server_status for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert nodemanager_health_summary for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert smartsense_gateway_status for an invalid cluster named cluster
07 Jun 2018 14:45:22,451 ERROR [alert-event-bus-2] AlertReceivedListener:480 - Unable to process alert ams_metrics_collector_hbase_master_process for an invalid cluster named cluster
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
06-05-2018
09:41 AM
This worked for me. Thanks @Anish Gupta
... View more
06-01-2018
05:42 AM
Hi, Is there any way to reduce the polling interval for ListSFTP on NiFi? I use 2.1.4
... View more
Labels:
- Labels:
-
Apache NiFi
05-07-2018
08:02 AM
I implemented log rotation on my HDP cluster (2.6.0) with ambari 2.5.0.3. I could not restart any any service with Ambari after changing the log4j properties. I got the error message below... Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 40, in <module>
BeforeStartHook().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py", line 33, in hook
setup_hadoop()
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py", line 93, in setup_hadoop
content=InlineTemplate(params.log4j_props)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
content = self._get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
return content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 52, in __call__
return self.get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 144, in get_content
rendered = self.template.render(self.context)
File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
return self.environment.handle_exception(exc_info, True)
File "<template>", line 101, in top-level template code
File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 371, in getattr
return getattr(obj, attribute)
ambari_jinja2.exceptions.UndefinedError: 'hadoop_log' is undefined
2018-04-19 07:15:51,479 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-04-19 07:15:51,488 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
Command failed after 1 tries
... View more
Labels:
03-29-2018
11:14 AM
I am encountering an issue and my NiFi cluster won't start. The error message is "Cluster is still in the process of voting on the appropriate Data Flow."; I have tried using the last flow.xml.gz file in the archive directory but it still gives the same error... 2018-03-29 13:03:48,739 INFO [NiFi Web Server-46] o.a.n.w.a.c.IllegalClusterStateExceptionMapper org.apache.nifi.cluster.manager.exception.IllegalClusterStateException: Cluster is still in the process of voting on the appropriate Data Flow.. Returning Conflict response.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
03-23-2018
09:09 AM
thanks @russ stevenson From the logs, the issue was due to incompatibility. I see EMC's compatibility chart talks about the Ambari version, it must be that they mean the Hortonworks version.
... View more
03-22-2018
02:11 PM
I am trying to add an isilon storage (using add host) to our HDP cluster. However, i encounter error "Registering with the server...Registration with the server failed." Has anyone come across this? I'm using HDP 2.6 with Ambari 2.5.3.
... View more
Labels:
02-27-2018
02:31 PM
@Prakash Punj Did you get the access tab to work in ranger? Will be glad if you post the solution to this issue...Thank you.
... View more
01-29-2018
10:21 AM
Thanks @Aditya Sirna I have checked using the sharelib command and error persists... I also ran the sharelibupdate command before relaunching... oozie admin -oozie http://{oozie-server}:11000/oozie -shareliblist
[Available ShareLib]
hive
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
hbase
pig
oozie admin -oozie http://{oozie-server}:11000/oozie -sharelibupdate
[ShareLib update status]
sharelibDirOld = hdfs://{name-node}:8020/user/oozie/share/lib/lib_20160610105311
host = http://{oozie-server}:11000/oozie
sharelibDirNew = hdfs://{name-node}:8020/user/oozie/share/lib/lib_20160610105311
status = Successful
... View more
01-29-2018
07:29 AM
I am unable to import into HBase using Oozie Sqoop. The command line works fine. Any one come across this? 16378 [main] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /usr/hdp/2.6.0.3-8/hadoop-mapreduce
2018-01-28 19:01:09,255 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is /usr/hdp/2.6.0.3-8/hadoop-mapreduce
18603 [main] INFO org.apache.sqoop.orm.CompilationManager - Writing jar file: /tmp/sqoop-yarn/compile/77d346afaacc732eba2256e5fcbbd275/QueryResult.jar
2018-01-28 19:01:11,480 INFO [main] orm.CompilationManager (CompilationManager.java:jar(330)) - Writing jar file: /tmp/sqoop-yarn/compile/77d346afaacc732eba2256e5fcbbd275/QueryResult.jar
18617 [main] ERROR org.apache.sqoop.tool.ImportTool - Error during import: HBase jars are not present in classpath, cannot import to HBase!
2018-01-28 19:01:11,494 ERROR [main] tool.ImportTool (ImportTool.java:run(621)) - Error during import: HBase jars are not present in classpath, cannot import to HBase!
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
Oozie Launcher failed, finishing Hadoop job gracefully
... View more
Labels:
- Labels:
-
Apache Oozie
11-10-2017
07:18 AM
I am trying to load a csv into spark but having difficulty with some newline characters in quotes. e.g "The csv
file is about
to be loaded into
Phoenix"
How i want it:
"The csv file is about to be loaded into Phoenix"
How do i get around this?
... View more
Labels:
- Labels:
-
Apache Spark
10-30-2017
07:05 AM
was able to create external tables using the OSCH. Issue was that the hadoop distribution used was downloaded from apache. This was resolved by shipping the hdp distribution in "/usr" to the oracle node and issue was resolved.
... View more
10-13-2017
10:20 AM
I am able to run the -createTable command successfully using the Oracle SQL connector for HDFS (OSCH). However, the external table created in Oracle is empty as against the over 38M rows that exist in Hive. Not sure why this is happening and i've checked the few log files but there's really nothing meaningful in the logs. Has anyone successfully used the OSCH to create external tables?
... View more
Labels:
- Labels:
-
Apache Hadoop