So this post has some clues, and reading back over it I have a vague recollection of seeing something similar before. The tl;dr is the kernel headers need to be processed to make them usable by userspace, and something about how that works has gone wrong.
Let's see if the instructions here work...
osmc@osmc:~/src$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.55.tar.xz osmc@osmc:~/src$ tar xf linux-4.19.55.tar.xz osmc@osmc:~/src$ sudo rm -r /usr/src/rbp2-headers-4.19.55-6-osmc/include/linux/* osmc@osmc:~/src$ sudo cp -ar linux-4.19.55/include/linux/* /usr/src/rbp2-headers-4.19.55-6-osmc/include/linux/ osmc@osmc:~/src$ cd rtl8812au/ osmc@osmc:~/src/rtl8812au$ make
That took a good 20 minutes and spat out plenty of warnings but seems to have actually built something! Memo to self: next time, add -j4 to make it use all four cores.
osmc@osmc:~/src/rtl8812au$ sudo make install install -p -m 644 88XXau.ko /lib/modules/4.19.55-6-osmc/kernel/drivers/net/wireless/ /sbin/depmod -a 4.19.55-6-osmc osmc@osmc:~/src/rtl8812au$
Well, it did something... but I don't think it actually loaded the driver. Certainly it's not in lsmod or ifconfig -a. Hmm...
osmc@osmc:~/src/rtl8812au$ sudo modprobe 88XXau osmc@osmc:~/src/rtl8812au$ dmesg | tail [ 12.641865] Console: switching to colour dummy device 80x30 [ 27.421305] random: crng init done [ 27.421335] random: 7 urandom warning(s) missed due to ratelimiting [ 41.099403] logitech-hidpp-device 0003:046D:404D.0004: HID++ 4.1 device connected. [ 2543.899806] cfg80211: Loading compiled-in X.509 certificates for regulatory database [ 2543.936878] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' [ 2543.958637] 88XXau: loading out-of-tree module taints kernel. [ 2543.978195] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 [ 2543.978226] cfg80211: failed to load regulatory.db [ 2544.237245] usbcore: registered new interface driver rtl88xxau osmc@osmc:~/src/rtl8812au$ ifconfig -a eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC> mtu 1500 inet 192.168.137.240 netmask 255.255.255.0 broadcast 192.168.137.255 ether b8:27:eb:4a:e3:88 txqueuelen 1000 (Ethernet) RX packets 83625 bytes 108947299 (103.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 42020 bytes 9260750 (8.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 wlan0: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether d0:37:45:35:d6:5d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Now to configure it, I think the route to take through the UI is My OSMC -> Network... and there it is! Actually connecting is rather anti-climatic after all that: select the wifi network, enter the passphrase, and it happily connects and finds an IP address. It doesn't seem to have detected a default gateway though - possibly that's because it's still got an ethernet connection via Nyx. So let's try disabling that... and it now updates with the router as the default gateway.
So, I now have OSMC running on a Raspberry Pi 2 with USB 802.11ac wifi. It'll probably all fall apart as and when I update and the kernel changes but I can deal with that when it happens.
Next challenge is getting the TV Hat to work!