Category Archives: Hadoop

Hive – Execution Error

reference: https://cwiki.apache.org/confluence/display/Hive/GettingStarted $ hive [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.TerminalFactory.create(TerminalFactory.java:101) at jline.TerminalFactory.get(TerminalFactory.java:158) Troubleshoot: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected Hive has upgraded to Jline2 but jline 0.94 exists in the Hadoop lib. Delete… Read More »

262 total views, 4 views today

Designating metadata store as Maria DB to Hive

Installing Maria DB Installation: http://zetawiki.com/wiki/CentOS_MariaDB_%EC%84%A4%EC%B9%98 Check installed  existing Maria DB by “rpm -qa” command Configuring repository: Add Mariadb.repo http://zetawiki.com/wiki/YUM_MariaDB_%EC%A0%80%EC%9E%A5%EC%86%8C_%EC%B6%94%EA%B0%80 Installing by “yum install” command DB Start / Change Password Check Connection / chkconfig on for service control automation Configuring Permission (External Connection): https://mariadb.com/kb/en/mariadb/grant/ Installing Driver: https://downloads.mariadb.org/connector-java/ Getting jdbc URL: http://tibang.tistory.com/454 Hive Configuration hive-site.xml: http://bigmark.tistory.com/35 2,061 total views, no views today

2,061 total views, no views today

Apache Giraph – From installation to Example execution

In this article, let me look into Apache Giraph and let’s run the example “SimpleShortestPathsComputation” First, look at the introduction of Giraph (http://giraph.apache.org/) Welcome to Apache Giraph! Apache Giraph is an iterative graph processing system built for high scalability. For example, it is currently used at Facebook to analyze the social graph formed by users and their… Read More »

2,461 total views, no views today

SSH / Key Setup for Seamless Authentication among AWS EC2 Servers

Prerequisite – In this post, it is supposed the EC servers to be exists the topic is related to the SSHD/SSH & Key concept Conditions – It is supposed that There are two EC servers The account of each server is default account, “ec2-user” URL of each server is ec1.compute.amazonaws.com, ec2.compute.amazonaws.com The default pem provided from AWS to… Read More »

2,713 total views, 2 views today

[HIVE] The Outline

Apache Hive (https://cwiki.apache.org/confluence/display/Hive/Home) Hive defines a simple SQL-like query language, called QL, that enables users familiar with SQL to query the data. At the same time, this language also allows programmers who are familiar with the MapReduce framework to be able to plug in their custom mappers and reducers to perform more sophisticated analysis that… Read More »

2,194 total views, no views today

[Hadoop] DBInputFormat – LIMIT & OFFSET Keyword

시스템의 학습 로그를 DB 에서 긁어와 하둡에 저장하기 위해서는 Job 을 DB 관련 형식으로 초기화하기 위한 설정 클래스인 DBConfiguration DB의  쿼리 실행을 제공하는 DBInputFormat 으로  job 의 in/out 에 대한 Format 설정 가져올 데이터 형식을 제공하는 DBWritable 인터페이스, Writable 클래스를 implementation, override 하는 클래스 선언 를 이용하여 DBConfiguration.configureDB>Job 생성>DBInputFormat.setInput(job,Class<? extends DBWritable> inputClass, Query, Record Count… Read More »

2,148 total views, no views today

Hadoop 2.2.0 (1) – Installation #3: TroubleShootings (editing…)

*Previous: Hadoop 2.2.0 (1) – Installation #1: Prerequisites Hadoop 2.2.0 (1) – Installation #2: Installing hadoop   Troubleshoots Namenode Daemon died 로그확인 먼저 로그를 확인해보자. 하둡 로그 디렉토리($HADOOP_HOME/logs) 의hadoop-${user.name}-namenode-NameNode_host.out.nn 파일이다. 로그메시지 Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the… Read More »

5,011 total views, 1 views today

Hadoop 2.2.0 (1) – Installation #2: Installing hadoop (editing..)

*Previous: Hadoop 2.2.0 (1) – Installation #1: Prerequisites *Next: Hadoop 2.2.0 (1) – Installation #3: TroubleShootings Download & Installation 설치폴더 복사:  /hdinstall/hadoop-2.2.0  폴더를  /usr/local/hadoop/ 에 복사 및 하위 폴더를 각 시스템의 위치에 해당하는 폴더로 복사 * 하둡 전체 폴더 아래에는 bin, etc, include, lib, libexec, logs, sbin, share, temp, tmp 폴더가 존재한다. 실제로 rpm으로 설치하면 이 폴더들은 /usr/sbin,… Read More »

3,733 total views, no views today

[Hadoop] Hadoop 2.2.0 (3) – Using Sequence File

내가 보고 있는 구닥다리 버전의 “톰 화이트” 선생이 쓴 [Hadoop 완벽 가이드] 를 보면 – 다른 책도 마찬가지겠지만 – 주어진 예제 데이터(기상, 항공노선운행정보 등)을 읽어서 간단히 처리해 보는 것으로 시작했을 것 같다. 하둡을 시작하면서 나 같은 초보자들은 당연히 데이터를 하둡의 파일시스템에 원하는대로 저장, 읽기를 바랄 것이고 그 결과를 출력하기를 바랄 것이다. 하둡의 파일 종류는, 현재까지… Read More »

2,811 total views, no views today

Hadoop 2.2.0 (1) – Installation #1: Prerequisites

* Next Hadoop 2.2.0 (1) – Installation #2: Installing hadoop Hadoop 2.2.0 (1) – Installation #3: TroubleShootings   칼을 들었으면 썩은 무라도 베어야 하고 공부를 했으면 실행을 해봐야..;;; 라지요. 일단 Cluster Mode 설치를 해봅니다. (* 완벽히 검증한 것은 아니라 아직 내용에 오류가 있을 수 있습니다. 과감한 지적 바랍니다. 구글닥에서 작성해서 옮겼더니 문서 포맷이 많이 깨지네요..ㅜㅜ) 하둡도… Read More »

5,195 total views, 1 views today