Balling’s Bits

LUKS Encrypted Hetzner VX6 vServer With Ubuntu 14.04 LTS

This is a guide to installing Ubuntu 14.04 LTS on an Hetzner VX6 vServer using LUKS encryption of the root partition. The guide is inspiried by blog posts of Martin Carpella and Oliver Feiler. On reboot you will be able to log in via ssh to provide the password for the LUKS partition (using dropbear and busybox).

As pointed out by Martin Carpella it may sound stupid to encrypt the disk of a virtual machine as the private key can be pulled from memory by the host (i.e. Hetzner). However, encrypting all data on the disk protects it in case you cancel the vServer, or the vServer gets moved to a new host, or a failed disk is returned to the manufacturer or discarded etc. So it is more a matter of protecting the data if the VM is cancelled/offline rather than online. If you do not want Hetzner to be able to pull the private key, go for something other than a virtual machine (e.g. a root server).

Anyway, here we go!

When signing up for the vServer select the Hetzner Ubuntu 14.04 LTS 64-bit minimal-install image. Once the server is setup log in via SSH and make backups of these files:

  • /etc/hostname
  • /etc/hosts
  • /etc/resolv.conf
  • /etc/network/interfaces

Log into the Hetzner robot interface and select 64-bit Linux rescue system, take note of the password for rescue system ssh login displayed on the page. Reboot your vServer in rescue mode and log in.

Setup disk partitions using fdisk:

1
fdisk /dev/vda

Create a boot partition of 256 MB (/dev/vda1) and a root partition of the remaining free space (/dev/vda2). Mark the boot partition as bootable. My partition table looks like this:

1
2
3
4
5
6
7
8
9
10
Disk /dev/vda: 26.8 GB, 26843545600 bytes
255 heads, 63 sectors/track, 3263 cylinders, total 52428800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00044f0a

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048      526335      262144   83  Linux
/dev/vda2          526336    52428799    25951232   83  Linux

Create the encrypted partition (enter passphrase when prompted):

1
cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/vda2

Decrypt the partition and create a volume group and two logical volumes (root and swap):

1
2
3
4
cryptsetup luksOpen /dev/vda2 vda2_decrypt
vgcreate vg-encrypted /dev/mapper/vda2_decrypt
lvcreate -L 2G -n swap vg-encrypted
lvcreate -L 22G -n root vg-encrypted

Create file systems:

1
2
3
mkfs.ext2 /dev/vda1
mkswap /dev/vg-encrypted/swap
mkfs.ext4 /dev/vg-encrypted/root

Record the UUIDs of the partitions:

1
2
3
4
5
6
blkid /dev/vda1 /dev/vda2 /dev/vg-encrypted/root /dev/vg-encrypted/swap
# These are my UUIDs
/dev/vda1: UUID="409938b1-7244-4c34-a665-8f086dd1c5f9" TYPE="ext2"
/dev/vda2: UUID="b360acf3-b392-417f-bc08-90f378a5d26b" TYPE="crypto_LUKS"
/dev/vg-encrypted/root: UUID="efc68b96-de3d-4cb9-a884-127dca8c97d5" TYPE="ext4"
/dev/vg-encrypted/swap: UUID="6556fbea-0e34-4f4e-92cc-c18b3ccb0ece" TYPE="swap"

MAke a chroot directory (/mnt/ubuntu) and mount volumes:

1
2
3
4
mkdir -p /mnt/ubuntu && \
mount /dev/vg-encrypted/root /mnt/ubuntu && \
mkdir /mnt/ubuntu/boot && \
mount /dev/vda1 /mnt/ubuntu/boot

Download and install the latest debootstrap, then debootstrap 64-bit Ubuntu 14.04 LTS to the chroot directory:

1
2
3
4
cd
wget "http://archive.ubuntu.com/ubuntu/pool/main/d/debootstrap/debootstrap_1.0.67_all.deb"
dpkg --install debootstrap_1.0.67_all.deb
debootstrap --arch amd64 trusty /mnt/ubuntu http://archive.ubuntu.com/ubuntu/

Mount proc, dev and sys and chroot into the bootstraped Ubuntu system:

1
2
3
4
5
mount -t proc none /mnt/ubuntu/proc
mount -o bind /dev /mnt/ubuntu/dev
mount -o bind /sys /mnt/ubuntu/sys
cp /etc/resolv.conf /mnt/ubuntu/etc/
LANG=C chroot /mnt/ubuntu /bin/bash

Create /etc/fstab (use your own UUIDs from above):

1
2
3
4
5
6
7
echo "
UUID=efc68b96-de3d-4cb9-a884-127dca8c97d5 / ext4 defaults,noatime 0 0
UUID=409938b1-7244-4c34-a665-8f086dd1c5f9 /boot ext2 defaults,relatime 0 1
UUID=6556fbea-0e34-4f4e-92cc-c18b3ccb0ece none swap sw 0 0
proc /proc proc defaults 0 0
sys /sys sysfs defaults 0 0
" > /etc/fstab

Copy hostname to /etc/hostname (use the hostname you recorded above):

1
echo "Ubuntu-1404-trusty-64-minimal" > /etc/hostname

Create /etc/hosts:

1
2
3
4
5
6
7
8
echo "127.0.0.1 localhost
127.0.1.1 Ubuntu-1404-trusty-64-minimal
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
" > /etc/hosts

Create /etc/network/interfaces (using your own IP and gateway information as recorded above):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
echo "
auto lo
iface lo inet loopback

# device: eth0
auto  eth0
iface eth0 inet static
  address   78.47.247.195
  broadcast 78.47.247.207
  netmask   255.255.255.240
  gateway   78.47.247.193
  # default route to access subnet
  up route add -net 78.47.247.192 netmask 255.255.255.240 gw 78.47.247.193 eth0

iface eth0 inet6 static
  address 2a01:4f8:d16:a8f::2
  netmask 64
  gateway 2a01:4f8:d16:a8f::1
" > /etc/network/interfaces

Setup apt to use Hetzner mirrors:

1
2
3
4
5
6
7
8
9
10
11
12
echo "
# Packages and Updates from the Hetzner Ubuntu Mirror
deb ftp://mirror.hetzner.de/ubuntu/packages trusty main restricted universe multiverse
deb ftp://mirror.hetzner.de/ubuntu/packages trusty-updates main restricted universe multiverse
deb ftp://mirror.hetzner.de/ubuntu/security trusty-security main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu trusty main
deb-src http://archive.ubuntu.com/ubuntu trusty main

deb http://security.ubuntu.com/ubuntu trusty-security main
deb-src http://security.ubuntu.com/ubuntu trusty-security main
" > /etc/apt/sources.list

Create /etc/crypttab (use your own UUIDs from above):

1
2
3
echo "# <target name> <source device> <key file> <options>
vda2_decrypt UUID=b360acf3-b392-417f-bc08-90f378a5d26b none luks
" > /etc/crypttab

Update packages and select time zone:

1
2
apt-get update
dpkg-reconfigure tzdata

Install essential packages (install grub to /dev/vda when prompted):

1
apt-get install aptitude openssh-server linux-image-generic cryptsetup lvm2 busybox dropbear

Extract the dropbear public key and add it to authorized_keys file:

1
2
3
dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | \
        grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub
cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys

Save the dropbear private key to your own computer as ~/.ssh/id_rsa.hetzner (remember to chmod 600 the file):

1
cat /etc/initramfs-tools/root/.ssh/id_rsa

Add modules to system and initramfs:

1
2
3
4
5
6
7
8
9
10
11
12
13
echo "dm-crypt" >> /etc/modules
echo "aes" >> /etc/initramfs-tools/modules
echo "aes_i586" >> /etc/initramfs-tools/modules
echo "aes_x86_64" >> /etc/initramfs-tools/modules
echo "aes_generic" >> /etc/initramfs-tools/modules
echo "dm-crypt" >> /etc/initramfs-tools/modules
echo "dm-mod" >> /etc/initramfs-tools/modules
echo "sha256" >> /etc/initramfs-tools/modules
echo "sha256_generic" >> /etc/initramfs-tools/modules
echo "lrw" >> /etc/initramfs-tools/modules
echo "xts" >> /etc/initramfs-tools/modules
echo "crypto_blkcipher" >> /etc/initramfs-tools/modules
echo "gf128mul" >> /etc/initramfs-tools/modules

IMPORTANT: remove or comment out (#) the grub hidden settings and the default cmd-line, and set the time out to 1 sec (in /etc/defaults/grub):

1
2
3
4
5
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

Update initramfs and grub:

1
2
3
update-initramfs -c -k all
update-grub
grub-install /dev/vda

Check that crytpsetup is in the initramfs sbin:

1
2
3
cd
zcat /initrd.img | cpio -i -d
ls sbin

Set root password:

1
passwd

Add a user:

1
adduser akwb

Make added user an admin:

1
2
addgroup --system admin
adduser akwb admin

If you experience problems with IPv6 you may need to add the following to /etc/rc.local (before exit 0):

1
2
/sbin/ifdown --force eth0
/sbin/ifup --force eth0

Exit the chroot, unmount and reboot:

1
2
3
4
5
6
7
8
exit
umount /mnt/ubuntu/boot
umount /mnt/ubuntu/proc
umount /mnt/ubuntu/sys
umount /mnt/ubuntu/dev
umount /mnt/ubuntu
sync
reboot

To enter the LUKS passphrase upon boot, first save the following script as unlock-cryptroot on your computer (not the vServer) and make it executable (chmod +x unlock-cryptroot):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
#!/bin/sh

usage() {
    cat <<EOF
${0##*/}:  Remotely unlock a LUKS-encrypted Ubuntu root filesystem

Works around bug #595648 <https://bugs.launchpad.net/bugs/595648>

Usage:  $0 [options] [--] <host>

Arguments:

  -h, --help
    Display this usage message and exit

  -i <identity_file>, --identity <identity_file>, --identity=<identity_file>
    Path to the SSH private key.
    Default: ${idbase}<host>

  -k <knownhosts>, --known-hosts <knownhosts>, --known-hosts=<knownhosts>
    Path to the 'known_hosts' file.
    Default: ${knownhosts}

  -l, --login
    Just log in, don't try to unlock the system.

  --
    End of options; treat the next argument as the hostname even if it
    begins with '-'.

  <host>
    The name of the remote system to unlock
EOF
}

# handy logging and error handling functions
log() { printf '%s\n' "$*"; }
error() { log "ERROR: $@" >&2; }
fatal() { error "$@"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }
usage_fatal() { usage >&2; printf '\n' >&2; fatal "$@"; }

# quote special characters so that:
#    eval "set -- $(shell_quote "$@")"
# is always a no-op no matter what values are in the positional
# parameters.  note that it is run in a subshell to protect the
# caller's environment.
shell_quote() (
    sep=
    for i in "$@"; do
       iesc=$(printf %s\\n "${i}eoi" | sed -e "s/'/'\\\\''/g")
       iesc=\'${iesc%eoi}\'
       printf %s "${sep}${iesc}"
       sep=" "
    done
)

# parse arguments
knownhosts=~/.ssh/known_hosts.initramfs
idbase=~/.ssh/id_rsa.initramfs_
unset id
runscript=true
while [ "$#" -gt 0 ]; do
    arg=$1
    case $1 in
        # convert "--opt=the value" to --opt "the value".
        --*'='*) shift; set -- "${arg%%=*}" "${arg#*=}" "$@"; continue;;
        -h|--help) usage; exit 0;;
        -i|--identity) shift; id=$1;;
        -k|--known-hosts) shift; knownhosts=$1;;
        -l|--login) runscript=false;;
        --) shift; break;;
        -*) usage_fatal "unknown option: '$1'";;
        *) break;;
    esac
    shift || usage_fatal "option '${arg}' requires a value"
done
[ "$#" -ge 1 ] || usage_fatal "no hostname specified"
host=$1; shift
[ "$#" -eq 0 ] || fatal "unknown argument: '$1'"

[ -n "${id+set}" ] || id=${idbase}${host%%.*}
[ -r "${id}" ] || fatal "can't read ssh key ${id}"


script='#!/bin/sh
PATH=/sbin:${PATH}

p() { printf %s\\n "$*"; }
log() { p "$@"; }
warn() { log "WARNING: $@" >&2; }
error() { log "ERROR: $@" >&2; }
fatal() { error "$@"; exit 1; }
try() { "$@" || fatal "'\''$@'\'' failed"; }

getpid() {
    psout=$(try ps) || exit 1
    psout=$(p "${psout}" | grep "$1") || return 0
    pswc=$(p "${psout}" | try wc -l) || exit 1
    [ "${pswc}" -eq 1 ] || fatal "more than one instance of $1:
${psout}"
    p "${psout}" | try awk '\''{print$1}'\'' || exit 1
}

# if cryptroot is not running, then there is no password prompt so
# there is nothing to do
log "checking if /scripts/local-top/cryptroot is running..."
cr_pid=$(getpid "/scripts/local-top/[c]ryptroot") || exit 1
[ -n "${cr_pid}" ] || fatal "/scripts/local-top/cryptroot is not running"

unset pw

# keep prompting for a password over and over until cryptroot has
# finished running
while true; do
    cs_pid=$(getpid "/sbin/[c]ryptsetup") || exit 1
    [ -n "${cs_pid}" ] || {
        log "waiting to see if there will be another passphrase prompt..."
        # the next commands are all on one line so that they are still
        # in busybox memory if the root filesystem is mounted during
        # the sleep (which would cause this script to disappear during
        # execution)
        sleep 1; [ -d /proc/"${cr_pid}" ] || { log "done"; exit 0; }
        continue
    }

    log "getting /sbin/cryptsetup command-line arguments..."
    cs_args=$(try tr "\\0" "\\n" </proc/${cs_pid}/cmdline) || exit 1
    set --
    while IFS= read -r line; do
        set -- "$@" "${line}"
    done <<EOF
${cs_args}
EOF
    log "command: $@"

    [ -n "${pw+set}" ] && {
        log "trying previously entered passphrase..."
        printf %s "${pw}" | try "$@"
    } || {
        pw=$(try /lib/cryptsetup/askpass "Enter passphrase: " && echo x) \
            || exit 1
        pw=${pw%x}
        printf %s "${pw}" | try "$@" || exit 1
    }

    log "passphrase accepted; killing passphrase prompt..."
    for ap in \
        "/lib/cryptsetup/[a]skpass" \
        "[a]sk-for-password" \
        ;
    do
        log "  checking for ${ap}..."
        ap_pid=$(getpid "${ap}") || exit 1
        [ -n "${ap_pid}" ] || continue
        log "    killing PID ${ap_pid}..."
        try kill "${ap_pid}"
    done
done
'

run_ssh() {
    unset forcetty
    case $1 in -t) forcetty=$1; shift;; esac
    ssh -o UserKnownHostsFile="${knownhosts}" \
        -i "${id}" \
        ${forcetty} \
        root@"${host}" "$@"
}

# $1 is the shell command to run; must be < 1024 characters (busybox
# limitation?)
run_ssh_cmd() {
    unset forcetty
    case $1 in -t) forcetty=$1; shift;; esac
    sshcmd='sh -c '$(shell_quote "$1")' -'
    run_ssh ${forcetty} "${sshcmd}"
}

"${runscript}" || {
    while IFS= read -r line; do
        log "${line}"
    done <<\EOF
After you are logged in:
  1. use 'ps -l' to get cryptsetup's command-line arguments
  2. run:
         /lib/cryptsetup/askpass "Enter passphrase: " \
             | /sbin/cryptsetup <args go here>
  3. kill 'plymouth ask-for-password' or 'askpass' as appropriate
  4. log out

EOF
    run_ssh
    exit $?
}

log "sending script to ${host}..."
printf %s\\n "${script}" |
run_ssh_cmd 'cat >tmp.sh && chmod +x tmp.sh' \
    || fatal "unable to create script"

log "running script on ${host}..."
run_ssh_cmd -t './tmp.sh'

Use the script like this (enter passphrase when prompted, use the IP address of your own vServer)

1
./unlock-cryptroot -k ~/.ssh/known_hosts.initramfs -i ~/.ssh/id_rsa.hetzner 78.47.247.195

Should something go wrong you can always boot the Hetzner rescue system and chroot to your vServer:

1
2
3
4
5
6
7
8
9
cryptsetup luksOpen /dev/vda2 vda2_decrypt
/etc/init.d/lvm2 start
mkdir -p /mnt/ubuntu && \
mount /dev/vg-encrypted/root /mnt/ubuntu
mount /dev/vda1 /mnt/ubuntu/boot
mount -t proc none /mnt/ubuntu/proc
mount -o bind /dev /mnt/ubuntu/dev
mount -o bind /sys /mnt/ubuntu/sys
LANG=C chroot /mnt/ubuntu /bin/bash

Comments