Trustworthy Systems

Download Linux On Linux Virtual Machine Monitor Documentation

Itanium Linux-on-Linux

Table of Contents

Licence and Copyright

Linux-on-Linux is Copyright 2005-2007 to National ICT Australia, the University of NSW, Matthew Chapman and Peter Chubb. It is licensed under the GNU Public Licence, version 2.0.

Building and Installation

  1. Get the virtual machine monitor (VMM) and patches from this directory.
  2. Install the prerequisites:
    • libatomic-ops-dev
    • gcc (version >= 4.1)
    • make
    • quilt
    • automake, and
    • autoconf
  3. Unpack the virtual machine monitor source:

    $ tar xjvf LinuxOnLinux-1.0.tar.bz2

  4. Create two copies of Linux kernel source. The patches from the CVS repository are usually against a recent kernel. One tree will be for the host, and one for the guest. Call them, e.g., linux-2.6-host and linux-2.6-guest.

    I usually do:
    $ cg-clone git:// linux-2.6-import
    $ cg-clone -l linux-2.6-import linux-2.6-guest
    $ cg-clone -l linux-2.6-import linux-2.6-host

  5. Run .../LinuxOnLinux/scripts/ on each source tree to patch it and to create the symbolic links.

    $ sh LinuxOnLinux/scripts/ linux-2.6-host host
    $ sh LinuxOnLinux/scripts/ linux-2.6-guest guest carries out the following steps:

    1. Applies appropriate patches to each tree using quilt. Patches are in .../vmm/patches/patches-host and .../vmm/patches/patches-guest
    2. Creates symbolic links from .../LinuxOnLinux/userspace/vmm-offsets.h into the guest Linux include/asm-ia64/vmm-offsets.h, and from .../LinuxOnLinux/afterburner/burn.S to arch/ia64/kernel/burn.S.
  6. Configure the host OS for your machine, build it, install it, and reboot. Include CONFIG_TUN and CONFIG_VMMHOST in the configuration, and exclude CONFIG_HUGETLBFS.
    There are sample configurations in the LinuxOnLinux source for HP and SGI systems, but you will have to adjust these for the filesystems and hardware you actually have
    The sample configuration also enables ethernet bridging. If you want to use it, install bridge-utils otherwise install iptables to set up NAT if you want your virtual machines to talk the the rest of the internet.
  7. Build the virtual machine monitor. The build process needs to be able to get at the output directory of the patched host kernel.

    $ cd LinuxOnLinux/userspace
    $ KERNEL=/lib/modules/`uname -r`/build sh

    (to rebuild, just type make).

  8. Configure the guest OS as if for the SKI simulator, but also select CONFIG_USRVMM and CONFIG_AFTERBURNER, and exclude CONFIG_HUGETLBFS (again, there are sample configurations in the LinuxOnLinux source tree).
    I prefer not to configure modules, because it's painful to get the modules into an appropriate place for the virtual machine to use.
    Build with a command line that looks like this:

    $ make EXTRA=-Bpath_to_LinuxOnLinux/afterburner EXTRA_AFLAGS=-Bpath_to_LinuxOnLinux/afterburner boot

    (note that you cannot use the tilde for home directory shortcut in path_to_LinuxOnLinux)
  9. Get or build yourself a root image. See SkiDiskImage on the Itanium WiKi for details). Call the result path_to_disksda.
  10. Run the resulting system with
    $ path_to_LinuxOnLinux/userspace/vmm vmlinux maxcpus=2 simscsi=path_to_disk/sd simeth=eth0

    If you prefer, there is a script runvmm in the source tree that you can start with to create and configure virtual machines. To use it, set the environment variable VMMPATH to the base of the LinuxOnLinux source tree. Or read the manual page and create a .vmmrc file in your home directory.

The VMM uses /dev/tap? as an ethernet interface; you can bridge these with brctl, or assign an IP address and use IP forwarding.

Current and Future Work

We're currently enhancing LinuxOnLinux to make it:

More information

More information is available on the Gelato@UNSW wiki and in the Virtualisation research pages.

Change Log

Release 1.0

Release 0.9

First release.


Troubleshooting is harder than it should be at present; this is one of the things we're working on.

Getting more information

The first thing to do when troubleshooting is to run with -k. If you're using vmm directly, it'll print stuff to standard error; if you're using runvmm the trace will be in /tmp/xNNNNN where NNNNN is the process number of runvmm.

You should get an error message of some sort in that trace. If it's not enough, edit userspace/debug.h, change the definition of the DEBUG symbol to (~0UL), and rebuild.

You'll now get a very verbose trace from the virtual machine monitor: every page fault, every privileged instruction, etc., will be logged.

Common Problems

The commonest problems are because the guest kernel hasn't been built correctly. Check that all the patches applied cleanly, that CONFIG_USRVMM is defined and that the afterburner was used when building. I've had problems using distcc where the afterburner wasn't installed in the same place on every machine; also you should disable ccache as it cannot see the afterburner. gcc-4.0 sometimes miscompiles, as well. I use gcc-4.1 without problems.

If you see a message like:
Cannot map from 0x8000000100000000,0x8000000103ffffff to 0x4000000--- mmap: Invalid argument PANIC in VMM: tlbop23 failed
it's because you've configured CONFIG_HUGETLBFS on in the host kernel. Note that the address is in region 4, which on a standard kernel is used for hugetlb pages, but for the virtual machine is used for the guest kernel's virtual memory area.