If you suspect this is happening to you look for lots of the following messages in the VirtualBox VM Log:
18:18:05.277 TM: Giving up catch-up attempt at a 60 009 852 153 ns lag; new total: 53 766 119 265 038 ns
18:19:10.901 TM: Giving up catch-up attempt at a 60 004 297 268 ns lag; new total: 53 826 123 562 306 ns
To fix this all you need to do is log into your Olive as root (log into the shell not the CLI) and edit (vi) the /boot/loader.conf file (You will need to force it to save using the :w! vi command).
Add "kern.hz=100" to the file at the end. Your loader.conf should look like this:
root@olive% more /boot/loader.conf
kernel="/kernel"
autoboot_delay="2"
console="comconsole"
platform_load="YES"
early_boot_load="YES"
kern.hz=100
Reload the olive and it should clear it up.