Hadoop 2.2.0 (1) – Installation #3: TroubleShootings (editing…)

By | Y2014Y2014-9M-D

*Previous:

Hadoop 2.2.0 (1) – Installation #1: Prerequisites

Hadoop 2.2.0 (1) – Installation #2: Installing hadoop


 

  • Troubleshoots
  1. Namenode Daemon died
    1. 로그확인

먼저 로그를 확인해보자. 하둡 로그 디렉토리($HADOOP_HOME/logs) 의hadoop-${user.name}-namenode-NameNode_host.out.nn 파일이다.

    • 로그메시지

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

It’s highly recommended that you fix the library with ‘execstack -c <libfile>’, or link it with ‘-z noexecstack’.

ed

open files                  (-n) 1024

pipe size        (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority          (-r) 0

stack size          (kbytes, -s) 10240

cpu time           (seconds, -t) unlimited

max user processes          (-u) 1024

virtual memory      (kbytes, -v) unlimited

file locks                  (-x) unlimited

    1. Solution:: set the lib path to the all servers.
      (bash profile or custom.sh 수정)

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS=”-Djava.library.path=$HADOOP_PREFIX/lib”

  1. namenode startup failure:
    – The log request to set the parameter dfs.namenode.servicerpc-address or dfs.namenode.rpc-address but before doing this, the dfs.namenode.hosts parameter value in hdfs-site.xml set from bach to beethoven

Solution: 단순 설정 문제. 반드시 설정 변경 후, 각 서버에 scp로 설정 복사!

  1. YARN Startup Failure
    1. yarn.resourcemanager.scheduler.class 오류
    2. Port Binding 오류

Solution:

    1. yarn-site.xml 의 해당 값을 “CapacityScheduler” 가 아닌 “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler” 로 설정(기본값이므로 차라리 이 설정을 빼는 것이 낫다)
    2. start-yarn.sh ($HADOOP_YARN_HOME/sbin/yarn-daemon.sh –config $HADOOP_CONF_DIR start resourcemanager) 명령을 해당 서버에서 직접 실행한다.
  1. Runtime Library 32-64bit 호환 관련 Warning
    1. 오류 메시지
      WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    2. Solution
      http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos

I assume you’re running Hadoop on 64bit CentOS. The reason you saw that warning is the native Hadoop library $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 was actually compiled on 32 bit.

Anyway, it’s just a warning, and won’t impact Hadoop’s functionalities.

Here is the way if you do want to eliminate this warning, download the source code of Hadoop and recompile libhadoop.so.1.0.0 on 64bit system, then replace the 32bit one.

Steps on how to recompile source code are included here for Ubuntu:

  1. Job  실행 후,  datanode 문제로 IO 오류발생, ipc 통신불가

Solution: hdfs-site.xml 수정 및 bach(datanode):/${hadoop.tmp.dir}/dfs/data 폴더 생성

  1. MapReduce Application 실행 도중 오류
    1. 오류메시지
      14/04/13 14:29:11 INFO mapreduce.Job: Task Id : attempt_1397332325392_0014_r_000000_0, Status : FAILED

Container [pid=39020,containerID=container_1397332325392_0014_01_000003] is running beyond virtual memory limits. Current usage: 30.5 MB of 1 GB physical memory used; 3.2 GB of 2.1 GB virtual memory used. Killing container.

    1. Solution

사용되는 메모리 총량을 잘 계산하여야 한다.

http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

For our example cluster, we have the minimum RAM for a Container (yarn.scheduler.minimum-allocation-mb) = 2 GB. We’ll thus assign 4 GB for Map task Containers, and 8 GB for Reduce tasks Containers.

In mapred-site.xml:

mapreduce.map.memory.mb: 4096

mapreduce.reduce.memory.mb: 8192

Each Container will run JVMs for the Map and Reduce tasks. The JVM heap size should be set to lower than the Map and Reduce memory defined above, so that they are within the bounds of the Container memory allocated by YARN.

In mapred-site.xml:

mapreduce.map.java.opts: -Xmx3072m

mapreduce.reduce.java.opts: -Xmx6144m

The above settings configure the upper limit of the physical RAM that Map and Reduce tasks will use.

To sum it up:

In YARN, you should use the mapreduce configs, not the mapred ones. EDIT: This comment is not applicable anymore now that you’ve edited your question.

What you are configuring is actually how much you want to request, not what is the max to allocate.

The max limits are configured with the java.opts settings listed above.

Finally, you may want to check this otherSO question that describes a similar problem (and solution).

** 하둡 설정 문제보다는 일단 System Virtual Memeory 부족이므로, 4GB Virtual Memory 용 Swap File을 yarn machine(mozart) 에 추가

http://faq.hostway.co.kr/index.php?mid=Linux_ETC&page=6&document_srl=1443&sort_index=readed_count&order_type=desc


*Previous:

Hadoop 2.2.0 (1) – Installation #1: Prerequisites

Hadoop 2.2.0 (1) – Installation #2: Installing hadoop

4,422 total views, 1 views today

2 thoughts on “Hadoop 2.2.0 (1) – Installation #3: TroubleShootings (editing…)

  1. Pingback: Hadoop 2.2.0 (1) – Installation #1: Prerequisites | bluecore's...

  2. Pingback: Hadoop 2.2.0 (1) – Installation #2: Installing hadoop (editing..) | bluecore's...

댓글 남기기