ESXi on Arm, Ubuntu, WireGuard and some Veeam

Like many others in the community I was very excited when I heard that ESXi on Arm is now general available. I heard about it first a few years back at VMworld, where a demo was shown. So, like many others I jumped to Amazon to get myself a Raspberry Pi 4 Kit. I opted to get the 8GB version, just in case.

Truth be told I saw a blog Post by Florian Grehl on how to play Doom on ESXi on Arm and I thought that is just so cool, I want to do that too! Check out his Blog here!

The CanaKit I purchased (I’ve used them before and was always happy with their sets) included a case, fan, memory card and power supply. You can check it out at Amazon, they have many to choose from.

There are already many great blog posts out there on how to install ESXi on a Pi, so I’m not going to go through all the steps again. Instead I’m going to write about issues (and me doing this for the first time, so definitely lack of knowledge) I ran into. Many thanks to William Lam for all his work to document this. Please head over to his blog as well, it is well written with great howtos.

There are a few modifications I did to my setup . First, I changed the fan. I bought this one: Noctua NF-A4x10 5V, Premium Quiet Fan. It is a perfect fit with the help of sanding paper or even better our familiar Dremel. It runs quite and is a better build than the kit fan. I also soldered some wires to make it fit and be able to route the cable they way I wanted it.

In the mean time I actually changed the case out and I’m using the TouchPi Touchscreen with a SmartiPi Touch 2 Case. Looks pretty cool. The fan still fits inside the case.

Installation of ESXi on Arm was pretty straight forward. I used the sd card as an installation source and target, which gave me no issues. The set came with a large sd card and I didn’t bother to resize it, as I was not going to use it for storage anyway. I first set up my old Qnap as an ISCI target, which did work but it was very slow. And by slow I mean like 10MB. I already had an unused case for a M2 SSD, so I bought a cheap 64GB m2 from Amazon. I also had an external SSD laying around, adding this gave me another 128GB.

Here are a few things I ran into:

/etc/init.d/usbarbitrator stop

chkconfig usbarbitrator off

Let’s check on the disk:

ls /dev/disks/

Find the one that is giving you issues. You can unplug and re plug without reboot, the disk will show up. My disk in question was: mpx.vmhba34:C0:T0:L0:1

Write new label:

partedUtil mklabel /dev/disks/ mpx.vmhba34:C0:T0:L0:1 gpt

Rescan and you should see your disk now.

• The M2 was working great, but health showed degraded. After some research I learned that is because there is only one path to storage. True and I appreciate the alarm, but I want green lights in my dashboard. I found this on VMware’s community pages.

esxcli storage core path list (will give us the diskid)
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL --device diskid --option="enable_local"     (replace the diskid with the disk identifier)
esxcli storage core claiming reclaim -d diskid   (replace the diskid with the disk identifier)
esxcli storage core claimrule load
esxcli storage core claimrule run

Let’s check now…

After all this we were in business. Step 1 was completed, we have ESXi running on the Pi.

The next step would be getting Ubuntu going. First, let’s head over and grab the proper ISO (https://ubuntu.com/download/server/armc). Next upload the ISO to our data storage. Finally, create a new VM. Select Ubuntu Linux (64-bit) for the Guest OS version. I used these settings below. Since I have an 8GB Pi I assigned 4GB to Ubuntu. Depending on your setup you might only want to assign 2GB.

Make sure you mount the cd drive with your downloaded Ubuntu iso and boot up. The installation process is pretty straight forward. Don’t forget to unmount the iso before the final reboot during installation.

After this is all done, it’s time to get Doom going! Head over to Virten.net and follow the instructions. It’s pretty sweet and I almost enjoyed running Doom in ssh more! Kind of mesmerizing!

So now that we have the basics down I was ready to install WireGuard! I’ve been wanting to run a better VPN solution at home for a while and having the Pi was just the piece I needed. Currently, I run OpenVPN on a modified Netgear router, running Tomato (Toastman mod) + a Windows box, which is great, but a bit cumbersome. Especially the certificate management was driving me crazy at times, plus WireGuard being a much leaner protocol was intriguing.

Ok, let’s do it.

First make sure our Ubuntu Sever is fully updated.

sudo apt update
and
sudo apt upgrade

With version 20.04 we don’t need to add repos anymore it’s all ready for us. Install WireGuard:

sudo apt install wireguard wireguard-tools

Generate the public and private key for the server (will be saved under /etc/wireguard)

wg genkey | sudo tee /etc/wireguard/server_private.key
wg pubkey | sudo tee /etc/wireguard/server_public.key

Create the main configuration file that will tell our WireGuard server everything it needs to know

sudo nano /etc/wireguard/wg0.conf

This is my config:

I want to point out a few things I ran into and took me a while to figure out. Be gentle, this was my first WireGuard set up.

SaveConfig = false really took me for a spin. Originally, I had set it to True, thinking I’ll modify wg0.conf to add clients. There are 2 ways of how you can add clients. Manually, by adding peers into the file and bringing the interface down and back up to take the change (downside is your tunnel goes down for all peers). Or you add it via wg (wg set wg0 peer “$CLIENT_PUBKEY” allowed-ips “$CLIENT_ADDRESS/32). When I had set the SaveConfig = True I misunderstood how it works, because I edited the config file manually and having this set to true it would kick out my modification after a restart or a tunnel up/down. This took me a while to figure out.

Also I had my iptables configured wrong at first. Specifically I didn’t set the correct interface, which lead to some wicked tunnel behavior. For example, I could ping certain hosts in my network via the tunnel, but not all. I could not always RDP into my boxes, but some would work for a while. You get the picture, not good.

You can use my config, but make sure you change the interface! In my config the ethernet interface is ens192. Run “ip a” and find the name of your interface.

doom@ubuntu:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:ea:ec:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.1.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feea:ec10/64 scope link
       valid_lft forever preferred_lft forever
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.10.10.1/24 scope global wg0
       valid_lft forever preferred_lft forever

and make sure you have the correct interface (here ens192) in your iptables.

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t n>
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens192 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t>

Ok, let’s continue….

For security let’s make sure only root can read.

sudo chmod 600 /etc/wireguard/{privatekey,wg0.conf}

Next enable port forwarding on the server.

sudo nano /etc/sysctl.conf

and uncomment this line in the file:

net.ipv4.ip_forward=1

Run

sudo sysctl -p

For the change to take effect.

Cool, not so hard. Next we need to the Ubuntu’s firewall to let traffic in.

sudo ufw allow 51820/udp
and while we are at it, let’s do port 22 for ssh as well
sudo ufw allow 22/tcp

Enable the firewall.

sudo ufw enable

Quick check, all good? Run

sudo ufw status verbose

You should get something like this:

Time to bring up our Wireguad interface.

sudo wg-quick up wg0

Success! On my box it shows:

[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.10.10.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ens192 -j MASQUERADE

Now, let’s make sure it comes up automatically after reboot.

sudo systemctl enable wg-quick@wg0

To control our WireGuard Server we can use (start/stop service)

sudo systemctl stop wg-quick@wg0.service
sudo systemctl start wg-quick@wg0.service

Bring tunnel interface up/down

wg-quick down wg0
wg-quick up wg0

Show status of our Tunnel(s):

sudo wg show wg0 <> show status

That is all for our server. Next step we need to get a client going.
I set up my Windows 10 Laptop. First install WireGuard for Windows and create an empty tunnel.

Here is my example configuration (not real keys):

[Interface]
PrivateKey = qIlIb5Ns/346532432115773234534242U99GXGnc=
Address = 10.10.10.4/24

[Peer]
PublicKey = ad2YoMs54564563454354354321118bLdH+07u7rSVWnE=
AllowedIPs = 0.0.0.0/0
Endpoint = XXX:51820
PersistentKeepalive = 25

Interface is the client’s key. Address is the address you already set up in the server config for this peer. Here is the snippet from my wg0.conf for that particular client. The allowed ips on the server equal address on the client.

[Peer]
#Thinkpad X1 FT
PublicKey = 0snrseqwewrfw2312134tgfsdd8sa0d80sadhVtamY=
AllowedIPs = 10.10.10.4/32

On the client, Allowed ips here is a full tunnel. You can do split tunnels as well by only putting in your home’s network(s). Endpoint is the public reachable ip or hostname of your Ubuntu Server. Keep alive is pretty clear. (If you want to change your DNS as well, add DNS = X.X.X.X under Address = 10.10.10.4/24

That’s all there is. Now bring up the tunnel and see if you can ping. First, test if you can ping the tunnel endpoint on the other side, in my case 10.10.10.1. Next, try if you can ping a machine in your home network. Lastly ping 8.8.8.8 and use your browser to make sure DNS is working

What if it doesn’t work…

Did you do port forwarding?

I didn’t mentioned it, but when you set your public address for your Ubuntu server I implied you need to do port forwarding on your router. This can be your ISP issued router or your own, but you have to forward all request for port 51820 (UDP) to the address of your Ubuntu Server.

Network with more than one router or subnet?

If you bring this network up in a normal, (flat) home network you probably don’t need any routes to make this work. But if the network is more complicated (cascading routers), or different vlans, subnets, and so on your pings to a local resource will probably fail.

But that’s ok, we can fix this!

If we take a look back at the config for my server, the WireGuard network I’m giving out is 10.10.10.0/24. My home network has 192.168.1.0/24 and 192.168.10.0/24, where my Ubuntu’s Ethernet interface is 192.168.1.20. Its gateway (Router) is 192.168.1.1

Let’s say I’m trying to ping my NAS @ 192.168.10.4 via the WireGuard tunnel. Remember those ip tables? My WireGuard takes my VPN client ip (eg 10.10.10.4), translates (nat’ed) it to 192.168.1.20 and sends it to 192.168.10.4.

Cool, but my QNAP has no idea where the original request from 10.10.10.4 comes from, so it will send it to the default gateway, in this case 192.168.10.1 This router does not know it either, so it drops the packet, sad panda, no ping.

We need to tell the router (192.168.10.1) in the 192.168.1.0 network what do to with 10.10.10.0. Every routers is different, but it really comes down to a statement like this:

Destination/Network Mask via next hop or 10.10.10.0 255.255.255.0 192.168.1.20

If everything is pinging, congratulations your WireGuard Tunnel is up and working.

I quickly want to touch on 2 things I ran into while setting this up.

Even if your client shows connected, it doesn’t mean it is. I had this on Windows clients, where my Pi was actually unplugged from the network and WireGuard client said the tunnel is active. Might be a Windows bug.

RDP sessions freeze. I’ve installed Ubuntu WireGuard on several places by now, one in a more complex network. Everything checked out, but RDP was unusable and would freeze all the time. If you run into this, it’s most likely your MTU setting.

To determine the proper MTU size, ping the system you want to connect to while the tunnel is established with changing MTU sizes and forcing the ping to not defragment.

ping myremotetarget -f -l 1500  (-f set the do not fragment bit, -l 1500 is the MTU size

Reduce MTU size until you get a successful ping. Once you hit the number, add 28 bytes. 20 bytes are added for the IP header and 8 bytes are allocated for the ICMP Echo Request Header.
In my case it was a MTU of 1410

Let’s set our magic number via:

sudo ip link set dev wg0 mtu 1410

and restart tunnel. This fixed my RDP issue.

I installed WireGuard Clients on all of my devices. My main Laptop has two profiles, one for split tunnel and one for full tunnel. I also have my Pixel and iPad set up for full tunnels. After using this setup for a little while now, I’m very happy with it. It’s fast and reliable. I actually had a Windows OpenVPN server as well, which I retired.

What else is missing from you ESXi on ARM setup…? Some Veeam Magic!

I’ve been using Veeam at work for a while now and Veeam offers are great community edition free of charge. Link to prev article

There are some limitations in terms of how many systems you can back up, but for our setup it’s a perfect fit.

Head over to https://www.veeam.com/virtual-machine-backup-solution-free.html and get a copy.

I installed it on an old Thinkpad X230, with an SMB connection to my trusty QNAP. Quickly added some jobs and was in business.

Little fun fact, while I was compiling VMware Tools on my Ubuntu server I completely fat-fingered some commands and “demolished” that machine. Instead of trying to fix it, I quickly restored from a recent backup job and was back in business. Thanks again to Veeam for offering this!

I’ve been playing around with my ESXi on Arm for a bit and have 3 VMs running on it right now. Ubuntu Server 20.x, Photon and Raspberry Pi OS (running virtualized on a Pi). You just gotta love virtualization!

I hope this post was helpful for some. I found William Lam and Florian Grehl’s posts very helpful and I’m thankful for all the work the community and ESXi on Arm fling have put into this.

’til next time!