Search Our Database

How to fix DRBD recovery from split brain

Last updated on |
by

DRBD® refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

Problem

DRBD does not connect between server

#/proc/drbd 
version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by gardner@, 2011-12-1
2 23:52:00
 0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:76

#log message on the node1
Mar  7 15:38:05 node1 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!

#/proc/drbd on the node2
version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by gardner@, 2011-12-1
2 23:52:00
 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
    ns:0 nr:0 dw:144 dr:4205 al:5 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:100

Solution:

1. Start drbd manually on both nodes (Go to both drbd primary and secondary servers and run the below command)

/etc/init.d/drbd start –version lower than CentOS7
systemctl start drbd –for CentOS7

2. Define one node as secondary and discard data on this

drbdadm secondary all
drbdadm disconnect all
drbdadm -- --discard-my-data connect all

3. Define another node as primary and connect

drbdadm primary all drbdadm disconnect all drbdadm connect all

Article posted on 20th October 2012 by Max