Member since
02-26-2017
25
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
17932 | 03-22-2017 07:11 PM |
05-31-2017
06:26 PM
Hi mbigelow, Thanks for your reply. Appreciate it.
... View more
05-28-2017
07:50 PM
Hi , I got the error as below. It seems we need to set memory for physical memory , but i can't find the configation about it. Where can i set physical memory? But the way , i just run some simple insert query in HIVE, and this error not happened on every time , i want to get the reason. ERROR: Diagnostic Messages for this Task: Container [pid=51661,containerID=container_e50_1493005386967_25486_01_000243] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_e50_1493005386967_25486_01_000243 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 51661 51659 51661 51661 (bash) 0 0 108662784 305 /bin/bash -c /usr/java/jdk1.8.0_101/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN.............
... View more
Labels:
- Labels:
-
Cloudera Manager
04-10-2017
07:05 PM
Sorry for late reply. I need to test it after business time. I set it to 2046->1024->512-> 256 and found NetWork IO take a low value with the number reducing. I'm sure the parameter is working for it. For someone who will refer this ticket. The speed of data transformation will take a very low speed for NetworkIO if you set a low value to this parameter. so please adjust schedul of jobs.
... View more
04-09-2017
06:35 PM
Yes, I set it from 4096 to 2048. The Network IO at the same level when it was 4096. Maybe i need to set less number than 2048?
... View more
04-06-2017
06:17 PM
It looks it( Maximum Number of Transfer Threads ) is not working. I set it from 4096 to 2048 , but NetworkIO still take a high value.
... View more
04-03-2017
02:02 AM
Hi , I want to limit speed of data transformation with CDM because other process * (Not bigdata , not CDH) is using same internet resource with it(CDM). and the process* can't be run as no resource. How can i set it in CHM? I tried to set 'DataNode Balancing Bandwidth' in HDFS configuration to 1Mb. But it looks not working.
... View more
Labels:
- Labels:
-
Cloudera Manager
-
HDFS
03-22-2017
07:11 PM
I installed the host again , and the problem has been fixed. I'm not sure the reason.
... View more
03-20-2017
10:58 PM
I created a new ticket for this problem as a summury, but i can't find it , why? I move it to here. I want to add a host to existing cluster as a datanode. But the host don't work fine as expected. 1, I confirmed status of agent and it is working fine. cloudera-scm-agent (pid 162575) is running... 2, I restarted the cluster and CDM , the problem can't be resolve. 3, I stopped agent and add the host and restart agent again , the problem can't be resolve also. Looking forward to your response. And please let me know if you want more info. Thanks, Wenjie
... View more
03-20-2017
10:57 PM
NO , The problem has not be resolved.
... View more
03-20-2017
08:19 PM
Hi , I want to add a host to existing cluster as a datanode. But the host don't work fine as expected. And i found the message in detail of host. and the log 1, I confirmed status of agent and it is working fine. cloudera-scm-agent (pid 162575) is running... 2, I restarted the cluster and CDM , the problem can't be resolve. 3, I stopped agent and add the host and restart agent again , the problem can't be resolve also. Looking forward to your response. And please let me know if you want more info. Thanks, Wenjie
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Manual Installation
03-17-2017
12:59 AM
I restarted the CDM and HOST MONITOR, it still can't be resloved. And i can't find CDH version in host list also for new host. By the way , HOST MONITOR service and other datanode (existing) is working fine. so i guess some setting in new host is not good.
... View more
03-17-2017
12:21 AM
I restarted the agent in host. But can't reslove it . And i found i can't get the CDH version ("None") in the host list. But existing datanode shows "CDH 5"
... View more
03-17-2017
12:11 AM
Hi csguna , Working fine. I want to add a new datanode to existing system. and existing system is working fine . The added host not work well only.
... View more
03-17-2017
12:08 AM
I restarted host monitor , but it still not working.
... View more
03-16-2017
11:55 PM
And i confirmed the log of host and found it. Caught unexpected exception in main loop. Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/agent.py", line 710, in __issue_heartbeat self._init_after_first_heartbeat_response(resp_data) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/agent.py", line 881, in _init_after_first_heartbeat_response assert response_data.get("cm_guid") is not None AssertionError
... View more
03-16-2017
11:47 PM
i'm sure i started agent in the host(datanode). /etc/init.d/cloudera-scm-agent status cloudera-scm-agent (pid 112359) is running...
... View more
03-16-2017
11:39 PM
Hi , I want to add a host to cluster as a datanode. Now i can find the host in host list and adding the New Hosts to existing Cluster. it looks it takes time because i began it 1hours ago but it is still running. And i can find it in the cluster with new windows even the processing is running. Bus Health test is bad. "This host is in contact with the Cloudera Manager Server. This host is not in contact with the Host Monitor." How can i investigate it ?
... View more
Labels:
- Labels:
-
Manual Installation
03-13-2017
06:14 PM
Got it! Thanks for your reply.
... View more
03-13-2017
12:17 AM
Hi, I want to add a data node to existing system , where can i find the manual or other? I tried to find it in this site, but i couldn't find content related it. Regards, wenjie
... View more
Labels:
- Labels:
-
Cloudera Manager
03-02-2017
07:57 PM
Hi, I got error as below. ...... Launching Job 2 out of 10 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> java.lang.RuntimeException: Error caching reduce.xml: java.nio.channels.ClosedByInterruptException at org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:737) at org.apache.hadoop.hive.ql.exec.Utilities.setReduceWork(Utilities.java:676) at org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:667) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:374) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1782) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1539) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1318) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1127) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:720) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:751) Job Submission failed with exception 'java.lang.RuntimeException(Error caching reduce.xml: java.nio.channels.Clos edByInterruptException)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: ...... I investigate about it and found it is related with calltime option of HIVE. My question is , Where can i find this option in cloudera manager ? HiveOptions.callTimeout?
... View more
Labels:
- Labels:
-
Cloudera Manager
02-26-2017
06:47 PM
@saranvisa Yes , you are right . I can't connect the host and i will apply it. Thanks a lot.
... View more
02-26-2017
06:05 PM
1 Kudo
When i tried to open WEBUI of IMPARA in cloudera Manager , it's not working. It looks the page can't be found as i got a blank page with error message . and i executed reflesh command with command mode in Linux also , it's working fine. Could you please give me some idea about it?
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager