Bhyve setup for tcp testing

Posted: January 9th, 2017 | Author: | Filed under: FreeBSD, tcp, virtualization | Tags: | Comments Off on Bhyve setup for tcp testing

Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I’ve already wrote down how I do dummynet so I’ll focus on bhyve part.

Caution: Handbook entry on bhyve is the true source. Please refer to it for exact information. This post is super quick and may contain not-entierly-correct things. Also, I am lazy and all this config is what I am using, you may need to tweak a bit here and there.

Setup:
I’ll create 3 bhyve guests: client, router and server:

client            router            server
192.168.1.227     192.168.1.228     192.168.1.229 
10.10.10.10   10.10.10.11
                  10.10.11.11   10.10.11.10

Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.

First, create tap interfaces needed for all bhyve guests:

client has tap0 (ssh), tap1
router has tap2 (ssh), tap3, tap4
server has tap5 (ssh), tap6

ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig tap3 create
ifconfig tap4 create
ifconfig tap5 create
ifconfig tap6 create

Now create bridge interfaces for the communication.

bridge0 contains re0, tap0, tap2, tap5
bridge1 contains tap1, tap3
bridge2 contains tap4, tap6

re0 is host interface here.

ifconfig bridge0 create
ifconfig bridge0 addm re0 addm tap0 addm tap2 addm tap5
ifconfig bridge0 up
ifconfig bridge1 create
ifconfig bridge1 addm tap1 addm tap3
ifconfig bridge1 up
ifconfig bridge2 create
ifconfig bridge2 addm tap4 addm tap6
ifconfig bridge2 up

bridge0 would help connect all guests mgmt interfaces to re0 (host interface) so they all can reach out and for us to be able to ssh into them.

bridge1 connects client to router and bridge2 connects router to server.

Now, let’s create VMs.

truncate -s 10G client.img
truncate -s 10G router.img
truncate -s 10G server.img

Setup/install VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img -i -I iso client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img -i -I iso router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img -i -I iso server

Here, ‘iso’ is the path to iso image that you want to install with and last arguments – client, router,server – are VM names.

Start the VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img server

Stop a VM:

bhyvectl --force-poweroff --vm=

To setup networking, you’d need following in rc.conf files:

client:
ifconfig_vtnet0="inet 192.168.1.227 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.11.10 10.10.10.11"

router:
ifconfig_vtnet0="inet 192.168.1.228 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.11 netmask 255.255.255.0"
ifconfig_vtnet2="inet 10.10.11.11 netmask 255.255.255.0"

server:
ifconfig_vtnet0="inet 192.168.1.229 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.11.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.10.10 10.10.11.11"

static route entries make sure routes are setup correctly for client and server to communicate with each other.

router would also need following in /etc/sysctl.conf to be able to pass traffic between client and server.

net.inet.ip.forwarding=1

Try pinging client from server or the other way around to make sure networking is working:

root@server:~ # ping 10.10.10.10
PING 10.10.10.10 (10.10.10.10): 56 data bytes
64 bytes from 10.10.10.10: icmp_seq=0 ttl=63 time=0.718 ms
64 bytes from 10.10.10.10: icmp_seq=1 ttl=63 time=0.999 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=63 time=0.553 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=63 time=0.553 ms
^C
--- 10.10.10.10 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.553/0.706/0.999/0.182 ms

Working networking setup on the guest looks something like this:

root@server:~ # ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.1.229 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 10.10.11.10 netmask 0xffffff00 broadcast 10.10.11.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Working networking setup on the host looks something like this:

tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
tap6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
inet 192.168.1.224 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 20000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000000

Freebsd on Freebsd using QEMU

Posted: August 7th, 2012 | Author: | Filed under: FreeBSD, virtualization | Tags: , , | Comments Off on Freebsd on Freebsd using QEMU

My system looks like this:


$ uname -a
FreeBSD xxxx.xxx.net 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

1) Install qemu from /usr/ports/emulators/qemu

2) Load following 2 modules:

$ sudo kldload aio
$ sudo kldload kqemu

3) Get iso of the guest OS, in my case its FreeBSD so I got one from:
ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.1/

4) Create qemu image: (‘man 1 qemu-img’ for more information)

$ qemu-img create -f qcow2 ~/qemu/fbsd9_1beta.img 8G


$ ls -l ~/qemu/
total 652448
-rw-r--r-- 1 hpanchasara hpanchasara 667693056 Aug 7 12:43
FreeBSD-9.1-BETA1-amd64-disc1.iso
-rw-r--r-- 1 hpanchasara hpanchasara 262144 Aug 7 12:44 fbsd9_1beta.img

Then I tried to launch it with: (‘man 1 qemu’ for more information)

$ qemu -boot d -hda ~/qemu/fbsd9_1beta.img -m 512M -cdrom ~/qemu/FreeBSD-9.1-BETA1-amd64-disc1.iso -localtime

But got following errors:

Listing a few to help the web-crawler which intern helps some lost soul like me:
“CPU doesn’t support long mode”
“Can’t work out which disk we are booting from…”
“panic: free: guard1 fail @ …”

Asking “the Internet” helped me narrow down the root causes to following:

1) 32 bit host cannot have 64 bit guest – which was not the case for me.
2) Machine should be virtualization capable – it is.
from /var/run/dmesg.boot, it should have VMX or SVM under “Features”.

FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (3392.37-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7
Features=0xbfebfbff
Features2=0x179ae3bf
AMD Features=0x28100800
AMD Features2=0x1
TSC: P-state invariant, performance statistics

3) Virtualization support should be enabled in bios – it was enabled.

$ sudo dmidecode | grep VME
VME (Virtual mode extension)

4) Use of incorrect qemu binary – that was the culprit.

“qemu” has different binary versions depending on the platform/systems:

$ qemu
qemu qemu-system-mips qemu-system-ppcemb
qemu-img qemu-system-mips64 qemu-system-sh4
qemu-system-arm qemu-system-mips64el qemu-system-sh4eb
qemu-system-cris qemu-system-mipsel qemu-system-sparc
qemu-system-m68k qemu-system-ppc qemu-system-sparc64
qemu-system-microblaze qemu-system-ppc64 qemu-system-x86_64

Using following I could successfully launch the guest FreeBSD instance:

$ qemu-system-x86_64 -boot d -hda ~/qemu/fbsd9_1beta.img -m 512 -cdrom ~/qemu/FreeBSD-9.1-BETA1-amd64-disc1.iso -localtime

On this first launch, it does FreeBSD install from the specified iso to the .img file. This is a typical FreeBSD installation process.

One the installation is done, you I could launch the VM via:

$ qemu-system-x86_64 -boot c -hda ~/qemu/fbsd9_1beta.img