른록노트

[WebServer] Haproxy + keepalive + session Cluster 웹 이중화 (가용성) 본문

Web/[WebServer]

[WebServer] Haproxy + keepalive + session Cluster 웹 이중화 (가용성)

른록 2019. 8. 16. 16:21

1. 버전정보

haproxy : 1.8.10

keepalive : 2.0.4

tomcat : 7.0.42

 

2. 프로그램 설명

haproxy : 지정한 ip:port 로의 접속을 여러대의 서버로 로드밸런싱 해주는 기능

keepalive : 가상 IP를 사용하게해주고 서로의 서버상태를 확인하는 기능

tomcat : 웹서버를 구동할 수 있고 세션을 공유하는 기능 사용할것임 (세션 클러스터링)

3. 방법

  1. 서버 2대를 준비한다,  A서버:1.1.1.1 B서버:2.2.2.2
  2. keepalive를 설치한다 (두서버 모두 동일하게 설치)
...더보기

1. yum을 통해 설치

yum install keepalived

 

2. 소스코드 컴파일을 통해 설치

yum -y install wget kernel-headers kernel-devel

wget https://www.keepalived.org/software/keepalived-2.0.4.tar.gz

tar -xzvf keepalived-2.0.4.tar.gz

해당폴더에서

./configure

make && make install

 

cd /root/keepalived-2.0.4/keepalived 

cp -rfv etc/init.d/keepalived /etc/init.d/

cp -rfv etc/keepalived/ /etc/keepalived/

cp -rfv etc/sysconfig/keepalived /etc/sysconfig/

cp -rfv keepalived /usr/sbin/

 

vi /etc/keepalived/keepalived.conf

 

global_defs {
   router_id WEB_CLUSTER
}

vrrp_instance VI_1 {
    state MASTER
    interface eno1
    virtual_router_id 51 //두서버 모두 같아야함
    priority 100
    advert_int 1
    authentication {
        auth_type PASS  //인증값
        auth_pass 1111  //인증값
    }
    virtual_ipaddress {
        192.168.1.15  //가상아이피
    }
}

 

systemctl enable keepalived

systemctl start keepalived

 

 

 3. haproxy를 설치한다.

...더보기

1. yum을 통해 설치

sudo yum install haproxy
sudo vi /etc/haproxy/haproxy.cfg

systemctl enable haproxy
systemctl start haproxy

2. 소스코드 컴파일을 통해 설치
yum -y install gcc wget openssl-devel
wget https://www.haproxy.org/download/1.8/src/haproxy-1.8.20.tar.gz
tar -xzvf haproxy-1.8.20.tar.gz

리눅스 커널 버전확인
3.10.0-862.el7.x86_64
컴파일 할때 TARGET 변수에 알맞는 OS 정보를 넣어줘야한다(README 파일 확인)
linux2628 = for Linux 2.6.28, 3.x, and above
ssl 인증서 확성화, systemd 컨트롤 활성화
USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1

make TARGET=linux2628 USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1

컴파일이 완료됐다면 설치
make install
(/usr/local/sbin에 설치됌)

설치확인 haproxy -v

systemctl에 서비스 등록
cp ~/haproxy-1.8.20/examples/haproxy.init /etc/init.d/haproxy
chmod 755 /etc/init.d/haproxy

haproxy 명령어를 사용할 수 있도록 /usr/sbin 디렉토리에 링크를 설정
ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy

haproxy 관련 디렉토리 만듬
sudo mkdir -p /etc/haproxy
sudo mkdir -p /var/log/haproxy
sudo mkdir -p /etc/haproxy/certs
sudo mkdir -p /etc/haproxy/errors/

haproxy config 설정
vi /etc/haproxy/haproxy.cfg

global
log 127.0.0.1 local0
daemon
maxconn 1024
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
option forwardfor
option http-server-close

frontend https-in
bind [keepalive에서 지정한 가상IP]:443 ssl crt [인증서 경로]
reqadd X-Forwarded-Proto:\ https
default_backend app-backend
frontend http-in
bind [keepalive에서 지정한 가상IP]:80
redirect scheme https code 301 if !{ ssl_fc }
backend app-backend
balance source
server [톰캣 서버1 호스트명] [톰캣 서버1 IP]:80 check
server [톰캣 서버2 호스트명] [톰캣 서버2 IP]:80 check

로그설정 
vi /etc/rsyslog.d/haproxy.conf 

# Provides UDP syslog reception 
$ModLoad imudp 
$UDPServerRun 514 
$template Haproxy, "%msg%\n" 
#rsyslog 에는 rsyslog 가 메세지를 수신한 시각 및 데몬 이름같은 추가적인 정보가 prepend 되므로, message 만 출력하는 템플릿 지정 
# 이를 haproxy-info.log 에만 적용한다. 

# 모든 haproxy 를 남기려면 다음을 주석해재, 단 access log 가 기록되므로, 양이 많다. 
#local0.*   /var/log/haproxy/haproxy.log 

# local0.=info 는 haproxy 에서 에러로 처리된 이벤트들만 기록하게 됨 (포맷 적용) 
local0.=info    /var/log/haproxy/haproxy-info.log;Haproxy 

# local0.notice 는 haproxy 가 재시작되는 경우와 같은 시스템 메세지를 기록하게됨 (포맷 미적용) 
local0.notice   /var/log/haproxy/haproxy-allbutinfo.log


구동 방법
(keepalived에서 가상아이피를 등록해야 제대로 실행이되고,
백업상태일경우 다른서버의 keepalived를 재시작하여 마스터상태로 바꾸고 아래 명령어를 실행한다)
systemctl enable haproxy
systemctl start haproxy

 

(로그는 테스트 안해봤습니다)

  4. 톰캣클러스터 설정

  vi /톰켓폴더/conf/server.xml

 

* 여기에 443이 아닌 그냥 일반 80포트로 해줘야함

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

<GlobalNamingResources>
	<Resource name="UserDatabase" auth="Container"
	type="org.apache.catalina.UserDatabase"
	description="User database that can be updated and saved"
	factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
	pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>

<Service name="llnote">

	<Connector 
	port="80" protocol="HTTP/1.1"
	connectionTimeout="20000"
	redirectPort="443" 
	compression="on"    
	compressionMinSize="2048"
	compressableMimeType="text/html,text/xml, text/javascript, application/javascript, text/css"
	URIEncoding="UTF-8" 
	/>
	<Connector port="8009" protocol="AJP/1.3" redirectPort="443" URIEncoding="UTF-8"/>
	<Engine name="keris" defaultHost="ecsc" jvmRoute="app_1"> <!-- Haproxy에서 사용할 서버명 적어주기 -->

	<Realm className="org.apache.catalina.realm.LockOutRealm">
	<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
	</Realm>

	<Host name="test"  appBase="webapps" unpackWARs="true" autoDeploy="true">
		<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8" channelStartOptions="3">
			<Manager className="org.apache.catalina.ha.session.DeltaManager"
			expireSessionsOnShutdown="false"
			notifyListenersOnReplication="true"/>
			<Channel className="org.apache.catalina.tribes.group.GroupChannel">
				<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
				address="[자기서버IP]"
				port="4000"
				autoBind="100"
				selectorTimeout="5000"
				maxThreads="6"/>
				<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
					<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
				</Sender>
				<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor" staticOnly="true"/>
				<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
				<Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
					<Member 
					className="org.apache.catalina.tribes.membership.StaticMember"
					port="4000"
					host="[세션공유할서버IP]"
					uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2}" <!--고유 아이디 (임의지정)-->
					/>	
				</Interceptor>
				<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
			</Channel>
			<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
			<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
			<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
		</Cluster>
	</Host>
	</Engine>
</Service>
</Server>

적용 후 톰캣 재시작

 

 

*유의할점*

1. haproxy를 실행할때 keepalived 상태가 master인상태만 실행되니 실행할때 상태를 변경해줘야함

2. https 인증서를 사용할경우 haproxy.cfg에만 인증서 등록을 해주고 인증서는 cat 으로 cert와 key를 합쳐주면 사용가능함 (cat aaa.cert aaa.key > key.pem), 그리고 톰캣에서 인증서를 등록하면 안됌

3. haproxy로 들어오는 클라이언트의 ip 정보는 x-forwarded-for 헤더 정보에 들어있음

 

- 틀린부분이나 조언해주실 부분 댓글로 적어주시면 감사하겠습니다.

 

 

 

참고사이트

https://www.haproxy.org/download/1.8/src/ - haproxy 다운로드

https://findstar.pe.kr/2018/07/27/install-haproxy/ - haproxy 설치 방법

https://youl.me/9 - haproxy 설치 방법

http://dveamer.github.io/architecture/HAProxyAndKeepalived.html - haproxy, keepalive 설치 방법

https://www.keepalived.org/pdf/UserGuide.pdf - keepalive 설치 메뉴얼

반응형
'Web/[WebServer]' Related Articles more
Comments