Linux kernel TCP smoothed-RTT estimation

Posted: February 18th, 2018 | Author: | Filed under: Linux, networking, tcp | Tags: , , , , | Comments Off on Linux kernel TCP smoothed-RTT estimation

Recently I decided to look under the hood to see how exactly srtt is calculated in Linux. Actual (Exponentially Weighted Moving Average) srtt calculation is a rather straight-forward part but what goes in as input to that calculation under various scenarios is interesting and very important in getting correct rtt estimate.

Also useful to note the difference between Linux and FreeBSD in this regard. Linux doesn’t trust tcp packet Timestamps option provided value whenever possible as middle-boxes can meddle with it.

Basic algorithm is:
For non-retransmitted packets, use saved packet send timestamp and ack arrival time.
For retransmitted packets, use timestamp option and if that’s not enabled, rtt is not calculated for such packets.

Let’s look at the code. I am using net-next.
When a TCP sender sends packets, it has to wait for acks for those packets before throwing them away. It stores them in a queue called ‘retransmission queue’.
When sent packets get acked, tcp_clean_rtx_queue() gets called to clear those packets from the retransmission queue.

A few useful variables in that function are:
seq_rtt_us – uses first packet from ackd range
ca_rtt_us – uses last packet from ackd range (mainly used for congestion control)
sack_rtt_us – uses sacked ack
tcp_mstamp is a tcp_sock member which represents timestamp of most recent packet received/sent. It gets updated by tcp_mstamp_refresh().

For a clean ack (not sack), seq_rtt_us = ca_rtt_us (as there is no range)

If such a clean is also for a non-retransmitted packet,
[sourcecode language=”c”]seq_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, first_ackt);[/sourcecode]

and for a sack which is again for a non-retransmitted packet,
[sourcecode language=”c”]sack_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, sack->first_sackt);[/sourcecode]

Code that updates sack→first_sackt is in tcp_sacktag_one() where it gets populated when the sack is for a non-retransmitted packet.

tcp_stamp_us_delta() gets the difference with timestamp that the stack maintains.

Now tcp_ack_update_rtt() gets called which starts out with:
[sourcecode language=”c”]
/* Prefer RTT measured from ACK’s timing to TS-ECR. This is because
* broken middle-boxes or peers may corrupt TS-ECR fields. But
* Karn’s algorithm forbids taking RTT if some retransmitted data
* is acked (RFC6298).
*/
if (seq_rtt_us < 0)
seq_rtt_us = sack_rtt_us;
[/sourcecode]

For acks acking retransmitted packets, seq_rtt_us would be -ve.
But if there is a SACK timestamp from a non-retransmitted packet, it would use that as it carries valid and useful timestamps.

Then it takes TS-opt provided timestamps only if seq_rtt_us is -ve.
[sourcecode language=”c”]
if (seq_rtt_us < 0 && tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
flag & FLAG_ACKED) {
u32 delta = tcp_time_stamp(tp) – tp->rx_opt.rcv_tsecr;
u32 delta_us = delta * (USEC_PER_SEC / TCP_TS_HZ);

seq_rtt_us = ca_rtt_us = delta_us;
}
[/sourcecode]

By this point, there is seq_rtt_us that can be fed into tcp_rtt_estimator() that’d generate smoothed-RTT (which is more or less based on SIGCOMM 88 paper by Van Jacobson).


Bhyve setup for tcp testing

Posted: January 9th, 2017 | Author: | Filed under: FreeBSD, tcp, virtualization | Tags: | Comments Off on Bhyve setup for tcp testing

Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I’ve already wrote down how I do dummynet so I’ll focus on bhyve part.

Caution: Handbook entry on bhyve is the true source. Please refer to it for exact information. This post is super quick and may contain not-entierly-correct things. Also, I am lazy and all this config is what I am using, you may need to tweak a bit here and there.

Setup:
I’ll create 3 bhyve guests: client, router and server:

client            router            server
192.168.1.227     192.168.1.228     192.168.1.229 
10.10.10.10   10.10.10.11
                  10.10.11.11   10.10.11.10

Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.

First, create tap interfaces needed for all bhyve guests:

client has tap0 (ssh), tap1
router has tap2 (ssh), tap3, tap4
server has tap5 (ssh), tap6

ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig tap3 create
ifconfig tap4 create
ifconfig tap5 create
ifconfig tap6 create

Now create bridge interfaces for the communication.

bridge0 contains re0, tap0, tap2, tap5
bridge1 contains tap1, tap3
bridge2 contains tap4, tap6

re0 is host interface here.

ifconfig bridge0 create
ifconfig bridge0 addm re0 addm tap0 addm tap2 addm tap5
ifconfig bridge0 up
ifconfig bridge1 create
ifconfig bridge1 addm tap1 addm tap3
ifconfig bridge1 up
ifconfig bridge2 create
ifconfig bridge2 addm tap4 addm tap6
ifconfig bridge2 up

bridge0 would help connect all guests mgmt interfaces to re0 (host interface) so they all can reach out and for us to be able to ssh into them.

bridge1 connects client to router and bridge2 connects router to server.

Now, let’s create VMs.

truncate -s 10G client.img
truncate -s 10G router.img
truncate -s 10G server.img

Setup/install VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img -i -I iso client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img -i -I iso router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img -i -I iso server

Here, ‘iso’ is the path to iso image that you want to install with and last arguments – client, router,server – are VM names.

Start the VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img server

Stop a VM:

bhyvectl --force-poweroff --vm=

To setup networking, you’d need following in rc.conf files:

client:
ifconfig_vtnet0="inet 192.168.1.227 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.11.10 10.10.10.11"

router:
ifconfig_vtnet0="inet 192.168.1.228 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.11 netmask 255.255.255.0"
ifconfig_vtnet2="inet 10.10.11.11 netmask 255.255.255.0"

server:
ifconfig_vtnet0="inet 192.168.1.229 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.11.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.10.10 10.10.11.11"

static route entries make sure routes are setup correctly for client and server to communicate with each other.

router would also need following in /etc/sysctl.conf to be able to pass traffic between client and server.

net.inet.ip.forwarding=1

Try pinging client from server or the other way around to make sure networking is working:

root@server:~ # ping 10.10.10.10
PING 10.10.10.10 (10.10.10.10): 56 data bytes
64 bytes from 10.10.10.10: icmp_seq=0 ttl=63 time=0.718 ms
64 bytes from 10.10.10.10: icmp_seq=1 ttl=63 time=0.999 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=63 time=0.553 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=63 time=0.553 ms
^C
--- 10.10.10.10 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.553/0.706/0.999/0.182 ms

Working networking setup on the guest looks something like this:

root@server:~ # ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.1.229 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 10.10.11.10 netmask 0xffffff00 broadcast 10.10.11.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Working networking setup on the host looks something like this:

tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
tap6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
inet 192.168.1.224 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 20000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000000

Drop a packet

Posted: October 14th, 2015 | Author: | Filed under: FreeBSD, networking, tcp | Tags: , , | Comments Off on Drop a packet

A few months back when I started looking into improving FreeBSD TCP’s response to packet loss, I looked around for traffic simulators which can do deterministic packet drop for me.

I had used dummynet(4) before so I thought of using it but the problem is that it only provided probabilistic drops. You can specify dropping 10% of the total packets, for example. I came across dpd work from CAIA, Swinburne University but it was written for FreeBSD7 and I couldn’t port it forward to FreeBSD11 with reasonable time/efforts as ipfw/dummynet has changed quite a bit.

So I decided to hack dummynet to provide me deterministic drops. Here is the patch: drop.patch
(Yes, it’s a hack and it needs polishing.)

Here is how I use it:
Setup:

client              dummynet          server
10.10.10.10  <--->  10.10.10.11
                    10.10.11.11 <---> 10.10.11.12

Both client and server need their routing tables setup correctly so that they can reach each other.

Dummynet node is the traffic shaping node here. We need to enable forwarding between interfaces:

sysctl net.inet.ip.forwarding=1

We need to setup links (called ‘pipes’) and their parameters on dummynet node like this:

# ipfw add pipe 100 ip from 10.10.11.12 to 10.10.10.10 out 
# ipfw add pipe 101 ip from 10.10.10.10 to 10.10.11.12 out
# ipfw pipe 100 config mask proto TCP src-ip 10.10.11.12 dst-ip 10.10.10.10 pls 3,4,5 plsr 7
# ipfw pipe 101 config mask proto TCP src-ip 10.10.10.10 dst-ip 10.10.11.12

‘pls 3,4,5 plsr 7’ – is the new configuration that the patch provides here.
pls : packet loss sequence
plsr : repeat frequency for the loss pattern

In the example above, it configures the pipe 100 to drop 3rd, 4th and 5th packet and repeat this pattern at every 7 packets going from server to client. So it’d also drop 10th, 11th and 12th packets and so on and so forth.

Side note: delay, bw and queue depth are other very useful parameters that you can set for the link to simulate however you want the link to behave. For example: ‘delay 5ms bw 40Mbps queue 50Kbytes’ would create a link with 10ms RTT, 40Mbps bandwidth with 50Kbytes worth of queue depth/capacity. Queue depth is usually decided based on BDP (bandwidth delay product) of the link. Dummynet drops packets once the limit is reached.

For simulations, I run a lighttpd web-server on the server which serves different sized objects and I request them via curl or wget from the client. I have tcpdump running on any/all of four interfaces involved to observe traffic and I can see specified packets getting dropped by dummynet.
sysctl net.inet.ip.dummynet.io_pkt_drop is incremented with each packet that dummynet drops.

Future work:
* Work on getting this patch committed into FreeBSD-head.
* sysctl net.inet.ip.dummynet.io_pkt_drop increments on any type of loss (which includes queue overflow and any other random error) so I am planning to add a more specific counter to show explicitly dropped packets only.
* I’ve (unsuccessfully) tried adding deterministic delay to dummynet so that we can delay specific packet(s) which can be useful in simulating link delays and also in debugging any delay-based congestion control algorithms. Turns out it’s trickier that I thought. I’d like to resume working on it as time permits.


Improving FreeBSD’s transport layer

Posted: October 9th, 2015 | Author: | Filed under: FreeBSD, networking, tcp | Tags: , , | Comments Off on Improving FreeBSD’s transport layer

FreeBSD network stack is quite stable but lacks some of the improvements/features available in other OSes.

A bunch of us have started an effort to try and identify current problems to improve transport layer (TCP, UDP, SCTP and others) for FreeBSD:

Transport Protocols wiki

Traditionally, freebsd-net has been the mailing list where networking problems get discussed but some have complained it to be too spammy and too focused on NIC drivers related issues. So a new mailing list has been created to specifically talk about transport level protocols: transport@

We’ve also started creating a list of TCP related RFCs and their support for FreeBSD to have a single point of reference.

Plan is to have a coordinated effort to improve TCP, UDP, etc.. so if you are interested in any of those protocols, please join the mailing list and help FreeBSD. :)


Build just one FreeBSD port

Posted: January 23rd, 2015 | Author: | Filed under: FreeBSD | Tags: , | 1 Comment »

I usually build my own packages with poudriere but it’s not fun to do on tiny boxes so I just do ‘pkg install ‘ on them and use upstream packages. One downside is, that package is build with default options. I recently ran into a situation where I wanted to change some options for just a single port.

Now, what is the minimal set of things in /usr/ports/ that I need to checkout to be able to config/build just one port?
Turns out to be:


/usr/ports/Mk
/usr/ports/Keywords
/usr/ports/Templates
/usr/ports/ports-mgmt/dialog4ports

And the port I want to actually build. Now I can ‘make config’, change the options and build/install that port without checking out entire ports tree.


FreeBSD on PicoStation M2HP

Posted: September 13th, 2013 | Author: | Filed under: embedded, FreeBSD, mips, wifi, wireless | Tags: , , , , , , | 2 Comments »

After tplink, it was time to play with picostation m2hp.
image

image

I’ve learned a bunch of things from this experience and this entire bring up would have been impossible without the help from loos@ and adrian@

First of all, a usb-ttl serial adapter is needed to connect to the board:
image

Here,
black – GND
white – TX
green – RX

When you power it on, the boot up sequence looks like this:

U-Boot 1.1.4.2-s564 (Jul 19 2012 - 10:41:56)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0

Here is where you should top the autoboot and it will drop into uboot>.

ar7240> printenv
bootdelay=1
baudrate=115200
ethaddr=00:15:6d:0d:00:00
mtdids=nor0=ar7240-nor0
partition=nor0,0
mtddevnum=0
mtddevname=u-boot
filesize=10000
fileaddr=81000000
serverip=192.168.1.254
ethact=eth0
mtdparts=mtdparts=ar7240-nor0:256k(u-boot),64k(u-boot-env),1024k(kernel),5760k(rootfs),256k(cfg),64k(EEPROM)
bootcmd=bootm 0x9f050000
bootargs=console=tty0 root=31:03 rootfstype=squashfs init=/init
ipaddr=192.168.1.20
stdin=serial
stdout=serial
stderr=serial

Environment size: 452/65532 bytes
ar7240>

Again, mtdparts an important piece here to see how uboot expects the image layout:

mtdparts=ar7240-nor0:256k(u-boot),64k(u-boot-env),1024k(kernel),5760k(rootfs),256k(cfg),64k(EEPROM)

I picked up a working openwrt image and tried to load it.
My setup looks like this:
image

Black power adapter has 2 cables going to it:
o POE (yellow)- power over ethernet – which connects to the board
o LAN (green)- which connects to my working router

Laptop act as a tftp server here which is also connected to the router (via gray cable). This way laptop and board are both in the same network.
laptop has a tftp server running and I’ve assigned 192.168.1.254 to that network interface (em0 in my case) with “ifconfig alias”. This is because as you can see in uboot’s printenv, uboot expects the tftp server to be running at that address.
The board will obviously act as a tftp client.

Now, to transfer the image, after generating the image (which we will look into in a bit), copy that into /tftpboot on the server.

This is how you put board into “urescue” mode:

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
ar7240> ping 192.168.1.254
Using eth0 device
host 192.168.1.254 is alive
ar7240> urescue
Setting default IP 192.168.1.20
Starting TFTP server...
Using eth0 (192.168.1.20), address: 0x81000000
Waiting for connection: /

Now, board is waiting for image. Send it from the tftp server:

% tftp 192.168.1.20
tftp> bin
tftp> put PICOSTATION_M2HP.initial.img
Sent 5994346 bytes during 15.2 seconds in 11709 blocks
tftp>

board receiving the image:

Receiving file from 192.168.1.18:18401
Received 6040751 bytes
Firmware Version: XM.ar7240.FreeBSD

Alright this was the basics of how to upload a valid image onto the board. Fun part is to generate a *valid* image that board will accept.
I tried a bunch of different kenrconf’s available in the freebsd-wifi-build project but board was not accepting the generated images. After accepting the image, it used to fail like this:

Firmware check failed! (-2)

This error was coming from uboot which was proprietary and I could not find this error in any opensourced version of uboot.

General consensus about the reasons for this error was:
– bad layout of the firmware image
– wrong header (something that this uboot is not liking)

loos@ showed me a trick to look at images with hexdum. After looking at openwrt’s working image by “hexdump -c image”, I found out that uboot was expecting something along the lines of “XS2.ar7240.FreeBSD” as version string in its header. If it does not see “ar7240” in the header, it would fail the check.

So, after all that and with ray@ suggestion of using lzma’ed kernel, I could boot up the board which later failed at mounting the rootfs:

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
ar7240> rescu
Unknown command 'rescu' - try 'help'
ar7240> uresc

Setting default IP 192.168.1.20
Starting TFTP server...
Using eth0 (192.168.1.20), address: 0x81000000
Waiting for connection: \
Receiving file from 192.168.1.254:57971
Received 5917237 bytes

Firmware Version: XS2.ar7240.FreeBSD
Setting U-Boot environment variables
Un-Protected 1 sectors
Erasing Flash.... done
Erased 1 sectors
Writing to Flash... done
Protected 1 sectors
Copying partition 'kernel' to flash memory:
erasing range 0x9F050000..0x9F12FFFF: .............. done
Erased 14 sectors
writing to address 0x9f050000, length 0x000e0000 ...
Copying partition 'rootfs' to flash memory:
erasing range 0x9F130000..0x9F5FFFFF: ............................................................................. done
Erased 77 sectors
writing to address 0x9f130000, length 0x004d0000 ...

Firmware update complete.

Resetting...

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
## Booting image at 9f050000 ...
Image Name: FreeBSD
Created: 2013-08-26 14:51:52 UTC

Image Type: MIPS Linux Kernel Image (lzma compressed)
Data Size: 893021 Bytes = 872.1 kB

Load Address: 80050000
Entry Point: 80050100
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK

Starting kernel ...

CPU platform: Atheros AR7241 rev 1
CPU Frequency=390 MHz
CPU DDR Frequency=390 MHz
CPU AHB Frequency=195 MHz
platform frequency: 390000000
CPU reference clock: 5 MHz
arguments:
a0 = 00000006
a1 = a1f4bfb0
a2 = a1f4c450
a3 = 00000000

Cmd line:argv is invalid
Environment:
envp is invalid
Cache info:
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r254676: Thu Aug 22 18:05:52 UTC 2013
[email protected]:/usr/home/hirenp/work/freebsd/head/obj/mipseb/mips.mips/usr/home/hirenp/work/freebsd/head/src/sys/AP91 mips

gcc version 4.2.1 20070831 patched [FreeBSD]
real memory = 16777216 (16384K bytes)
avail memory = 12087296 (11MB)

random device not loaded; using insecure entropy
nexus0:
nexus0: failed to add child: arge0
clock0: on nexus0
Timecounter "MIPS32" frequency 195000000 Hz quality 800
Event timer "MIPS32" frequency 195000000 Hz quality 800
argemdio0: at mem 0x19000000-0x19000fff on nexus0

mdio0: on argemdio0
mdioproxy0: on mdio0
arswitch0: on mdio0
arswitch0: attaching PHY 0 failed
arswitch0: attaching PHY 1 failed
arswitch0: attaching PHY 2 failed
arswitch0: attaching PHY 3 failed
device_attach: arswitch0 attach returned 6
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
pcib0 at irq 0 on nexus0
pcib0: found EEPROM at 0x1fff1000 on 0.0.0
pcib0: EEPROM firmware: 0x1fff1000 @ 4096 bytes
pcib0: device EEPROM 'pcib.0.bus.0.0.0.eeprom_firmware' registered
pci0: on pcib0
pci0: at device 0.0 (no driver attached)
arge0: at mem 0x19000000-0x19000fff irq 2 on nexus0
miiproxy0: on arge0
arge0: can't attach proxy
arge0: finishing attachment, phymask 0010, proxy null
arge0: unable to attach PHY 4: 6
device_attach: arge0 attach returned 6
arge1: at mem 0x1a000000-0x1a000fff irq 3 on nexus0
arge1: finishing attachment, phymask 0000, proxy null
arge1: Ethernet address: 62:73:64:dd:03:41

spi0: at mem 0x1f000000-0x1f00000f on nexus0
spibus0: on spi0
mx25l0: at cs 0 on spibus0
mx25l0: w25q64, sector 65536 bytes, 128 sectors
ar71xx_wdog0: on nexus0
ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/map/rootfs.uncompress []...
mountroot: waiting for device /dev/map/rootfs.uncompress ...
Mounting from ufs:/dev/map/rootfs.uncompress failed with error 19.

Loader variables:

Manual root filesystem specification:
: [options]
Mount using filesystem
and with the specified (optional) option list.

eg. ufs:/dev/da0s1a
zfs:tank
cd9660:/dev/acd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

? List valid disk boot devices
. Yield 1 second (for background tasks)
Abort manual input

mountroot>

jmg@ suspected that /dev/map/rootfs.uncompress doesn’t exist for some reason as error: 19 is ENODEV and asked me to enable debugging in g_uncompress.c
I did not find anything useful after enabling debugs:

flash/spi0.uncompress: media sectorsize 512, mediasize 8388608
flash/spi0.uncompress: no CLOOP magic
map/u-boot.uncompress: media sectorsize 512, mediasize 262144
map/u-boot.uncompress: no CLOOP magic
map/u-boot-env.uncompress: media sectorsize 512, mediasize 65536
map/u-boot-env.uncompress: no CLOOP magic
map/kernel.uncompress: media sectorsize 512, mediasize 1048576
map/kernel.uncompress: no CLOOP magic
map/rootfs.uncompress: media sectorsize 512, mediasize 6684672
map/rootfs.uncompress: no CLOOP magic
map/cfg.uncompress: media sectorsize 512, mediasize 262144
map/cfg.uncompress: no CLOOP magic
map/eeprom.uncompress: media sectorsize 512, mediasize 65536
map/eeprom.uncompress: no CLOOP magic

Trying to mount root from ufs:/dev/map/rootfs.uncompress []...
mountroot: waiting for device /dev/map/rootfs.uncompress ...
Mounting from ufs:/dev/map/rootfs.uncompress failed with error 19.

Loader variables:

Manual root filesystem specification:
: [options]
Mount using filesystem
and with the specified (optional) option list.

eg. ufs:/dev/da0s1a
zfs:tank
cd9660:/dev/acd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

? List valid disk boot devices
. Yield 1 second (for background tasks)
Abort manual input

mountroot> ?

List of GEOM managed disk devices:
map/eeprom map/cfg map/rootfs map/kernel map/u-boot-env map/u-boot flash/spi0

mountroot>

Later loos@ caught that the problem was the rootfs start point in the hints file.

Specially we had to specify:
hint.map.3.start=0x130000

Because mtdparts specify 0x130000 as rootfs offset.

After this bring up part was done:

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
ar7240> ures

Setting default IP 192.168.1.20
Starting TFTP server...
Using eth0 (192.168.1.20), address: 0x81000000
Waiting for connection: |
Receiving file from 192.168.1.254:57971
Received 5966201 bytes
Firmware Version: XM.ar7240.FreeBSD

Setting U-Boot environment variables
Un-Protected 1 sectors
Erasing Flash.... done
Erased 1 sectors
Writing to Flash... done

Protected 1 sectors
Copying partition 'kernel' to flash memory:
erasing range 0x9F050000..0x9F12FFFF: .............. done
Erased 14 sectors
writing to address 0x9f050000, length 0x000e0000 ...
Copying partition 'rootfs' to flash memory:
erasing range 0x9F130000..0x9F5FFFFF: ............................................................................. done
Erased 77 sectors
writing to address 0x9f130000, length 0x004d0000 ...
Copying partition 'cfg' to flash memory:
erasing range 0x9F6F0000..0x9F6FFFFF: . done
Erased 1 sectors
writing to address 0x9f6f0000, length 0x00010000 ...

Firmware update complete.

Resetting...

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
## Booting image at 9f050000 ...
Image Name: FreeBSD
Created: 2013-08-27 11:12:34 UTC

Image Type: MIPS Linux Kernel Image (lzma compressed)
Data Size: 892769 Bytes = 871.8 kB
FreeBSD 10.0-CURRENT #6 r254676M: Tue Aug 27 11:11:27 UTC 2013

[email protected]:/usr/home/hirenp/work/freebsd/head/obj/mipseb/mips.mips/usr/home/hirenp/work/freebsd/head/src/sys/AP91 mips
gcc version 4.2.1 20070831 patched [FreeBSD]
real memory = 16777216 (16384K bytes)
avail memory = 12087296 (11MB)
random device not loaded; using insecure entropy
nexus0:
clock0: on nexus0
Timecounter "MIPS32" frequency 195000000 Hz quality 800
Event timer "MIPS32" frequency 195000000 Hz quality 800
argemdio0: at mem 0x1a000000-0x1a000fff on nexus0

mdio0: on argemdio0
mdioproxy0: on mdio0
arswitch0: on mdio0
miibus0: on arswitch0
ukphy0: PHY 0 on miibus0
ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus1: on arswitch0
ukphy1: PHY 1 on miibus1
ukphy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus2: on arswitch0
ukphy2: PHY 2 on miibus2
ukphy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus3: on arswitch0
ukphy3: PHY 3 on miibus3
ukphy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
etherswitch0: on arswitch0
mdio1: on arswitch0
mdioproxy1: on mdio1

apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
pcib0 at irq 0 on nexus0
pcib0: found EEPROM at 0x1fff1000 on 0.0.0
pcib0: EEPROM firmware: 0x1fff1000 @ 4096 bytes
pcib0: device EEPROM 'pcib.0.bus.0.0.0.eeprom_firmware' registered
pci0: on pcib0
pci0: at device 0.0 (no driver attached)
arge0: at mem 0x19000000-0x19000fff irq 2 on nexus0
arge0: Overriding MAC from EEPROM
miiproxy0: on arge0
miiproxy0: attached to target mdio1
arge0: finishing attachment, phymask 0010, proxy set
miibus4: on miiproxy0
ukphy4: PHY 4 on miibus4
ukphy4: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
arge0: Ethernet address: 12:00:00:18:3c:02

arge1: at mem 0x1a000000-0x1a000fff irq 3 on nexus0
arge1: finishing attachment, phymask 0000, proxy null
arge1: Ethernet address: 12:00:00:18:3c:03

spi0: at mem 0x1f000000-0x1f00000f on nexus0
spibus0: on spi0
mx25l0: at cs 0 on spibus0
mx25l0: w25q64, sector 65536 bytes, 128 sectors
ar71xx_wdog0: on nexus0
ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
arswitch0port1: link state changed to DOWN
arswitch0port2: link state changed to DOWN
arswitch0port3: link state changed to DOWN
arswitch0port4: link state changed to DOWN
map/rootfs.uncompress: GEOM_ULZMA image found
map/rootfs.uncompress: 173 x 131072 blocks

Trying to mount root from ufs:/dev/map/rootfs.uncompress []...
warning: no time-of-day clock registered, system time will not be set accurately
Aug 27 11:11:45 init: login_getclass: unknown class 'daemon'
*** Populating /var ..
*** Loading configuration files ..
*** Restoring from /dev/map/cfg ..
gunzip: unknown compression format
0 blocks
*** Completed.
*** setting up hostname
*** Load kernel modules
random: initialized
*** bringing up loopback ..
*** Starting networking via /etc/rc.d/base/net
sysctl: unknown oid 'dev.ath.0.txq_mcastq_maxdepth': No such file or directory
sysctl: unknown oid 'dev.ath.1.txq_mcastq_maxdepth': No such file or directory
*** Interface: arge0: start
arge0: link state changed to UP
*** Interface: arge0: done
*** Interface: bridge0: start
bridge0: Ethernet address: d2:c4:a8:63:0e:57
arge0: promiscuous mode enabled
bridge0: link state changed to UP
*** Interface: bridge0: done
*** Default password/login databases ..
*** inetd
*** Done!

FreeBSD/mips (freebsd-wifi-build) (ttyu0)

login: root
No home directory.
Logging in with home = "/".
# uname -a
FreeBSD freebsd-wifi-build 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r254676M: Tue Aug 27 11:11:27 UTC 2013 [email protected]:/usr/home/hirenp/work/freebsd/head/obj/mipseb/mips.mips/usr/home/hirenp/work/freebsd/head/src/sys/AP91 mips
# df -k
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/map/rootfs.uncompress 21851 20633 -529 103% /
devfs 1 1 0 100% /dev
/dev/md0 828 8 756 1% /tmp
/dev/md1 828 56 708 7% /var
/dev/md2 828 436 328 57% /etc
# pciconf -lv
none0@pci0:0:0:0: class=0x028000 card=0xe3020777 chip=0x002a168c rev=0x01 hdr=0x00
vendor = 'Atheros Communications Inc.'
device = 'AR928X Wireless Network Adapter (PCI-Express)'
class = network
# ifconfig
arge0: flags=8943 metric 0 mtu 1500
options=80000
ether 12:00:00:18:3c:02
media: Ethernet autoselect (100baseTX )
status: active
arge1: flags=8802 metric 0 mtu 1500
ether 12:00:00:18:3c:03
media: Ethernet 1000baseT
status: active
lo0: flags=8049 metric 0 mtu 16384
options=600003
inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843 metric 0 mtu 1500
ether d2:c4:a8:63:0e:57
inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: arge0 flags=143
ifmaxaddr 0 port 5 priority 128 path cost 200000
#

Next up was figuring out networking.

I loaded all needed modules:

# kldstat
Id Refs Address Size Name
1 22 0x80050000 2d8f10 kernel
2 1 0xc0081000 1078 if_ath_pci.ko
3 1 0xc0083000 11e4bc if_ath.ko
4 4 0xc01a2000 606fc wlan.ko
5 2 0xc0203000 4924 bridgestp.ko
6 1 0xc0208000 6d90 if_bridge.ko
7 1 0xc020f000 a5e0 random.ko
8 1 0xc021a000 3ac wlan_xauth.ko
9 1 0xc021b000 27e4 wlan_tkip.ko
#

But if loading if_ath_pci failed in:

ath0: at device 0.0 on pci0
ath0: ath_pci_attach: EEPROM firmware @ 0x8046c000
[ath]: default pwr offset: -5 dBm != EEPROM pwr offset: 0 dBm; curves will be adjusted.
ath0: ath_getchannels: unable to collect channel list from hal, status 12
device_attach: ath0 attach returned 22

Turning up debugging level:

# sysctl hw.ath.hal.debug=25600000000
hw.ath.hal.debug: 0 -> 2147483647

# kldload if_ath_pci
ath0: mem 0x10000000-0x1000ffff irq 0 at device 0.0 on pci0

ath0: ath_pci_attach: EEPROM firmware @ 0x8046c000
ar9280Attach: sc 0xc090d000 st 0x803006b4 sh 0xb0000000
ar5416SetReset Applying descriptor swap
ar5416SetPowerMode: AWAKE -> AWAKE (set chip )
ar9280Attach: ID 0x802ff VERSION 0x2 TYPE 0x0 REVISION 0x2
ath_hal_v14EepromAttach Eeprom Version 14.22
v14EepromReadCTLInfo Numctls = 11
ar5416SetPowerMode: AWAKE -> AWAKE (set chip )
ar9280RfAttach: attach AR9280 radio
[ath]: default pwr offset: -5 dBm != EEPROM pwr offset: 0 dBm; curves will be adjusted.
enableAniMIBCounters: Enable mib counters: OfdmPhyErrBase 0xbffe0c cckPhyErrBase 0xbfff38
ar9280Attach: return
getchannels: cc 0 regDmn 0xf0 mode 0xffffff ecm
isEepromValid: invalid regulatory domain/country code 0x2a
getregstate: invalid EEPROM contents
ath0: ath_getchannels: unable to collect channel list from hal, status 12
ar5416Detach:
ar5416SetPowerMode: AWAKE -> AWAKE (set chip )
Detaching Ani
Disable MIB counters
ar5416SetPowerMode: AWAKE -> AWAKE (set chip )
ar5416SetReset Applying descriptor swap
ar5416SetPowerMode: AWAKE -> FULL-SLEEP (set chip )

device_attach: ath0 attach returned 22

Error here was: isEepromValid: invalid regulatory domain/country code 0x2a
What it meant was that hal regdomain needed to be updated with country code 0x2a

Just to hack that up for time being, adrian@ suggested to set it to 0x0 which worked.


Index: dev/ath/ath_hal/ah_regdomain.c
===================================================================
--- dev/ath/ath_hal/ah_regdomain.c (revision 255320)
+++ dev/ath/ath_hal/ah_regdomain.c (working copy)
@@ -169,6 +169,10 @@
if (regDomainPairs[i].regDmnEnum == rd)
return AH_TRUE;
}
+ if (rd == 42) {
+ rd = 0;
+ return AH_TRUE;
+ }
HALDEBUG(ah, HAL_DEBUG_REGDOMAIN,
"%s: invalid regulatory domain/country code 0x%x\n", __func__, rd);
return AH_FALSE;

The kernconf and hints file are committed as r255086.

This is how you can generate the image:

$ cd ~
$ mkdir -p work/freebsd/head
$ cd work/freebsd/head
$ svn checkout svn://svn.freebsd.org/base/head src
$ svn checkout http://freebsd-wifi-build.googlecode.com/svn/trunk/ build
$ cd build/programs/ubnt-mkfwimage
$ make
$ su
# make install
# exit
$ cd ~/work/freebsd/head/src
$ ../build/build/bin/build picostation_m2hp buildworld buildkernel
$ su
# mkdir /tftpboot
# ../build/build/bin/build picostation_m2hp installworld installkernel distribution mfsroot fsimage uboot ubnt
# exit

On successful completion, resulting image PICOSTATION_M2HP.initial.img will be in /tftpboot.

My entire effort/rant has been also recorded at freebsd-embedded@


Making TL-WR1043ND do networking

Posted: July 23rd, 2013 | Author: | Filed under: embedded, FreeBSD, mips, wifi, wireless | No Comments »

This is a followup post to Installing FreeBSD on TP-Link TL-WR1043ND

I’ve learned a few very basic networking things while trying to make this work.

After having FreeBSD run on the board, next thing was to make it do networking. How Adrian’s scripts work is, it has configuration in /etc/cfg/


# pwd
/etc/cfg
# ls
hostapd.wlan0.conf manifest rc.conf
#

Here,
manifest file has list of files to be stored in flash on “cfg_save”. Basic workflow is that you make changes you want and do “cfg_save” which writes things to flash and then “reboot”. So router comes up with the saved settings.

To create a wifi network for let’s say wlan0, we need to create a hostapd file for it:

# cat hostapd.wlan0.conf
interface=wlan0
driver=bsd
ssid=STRCDR_11N
wpa=3
wpa_key_mgmt=WPA-PSK
wpa_passphrase=NotAdminORPassword
wpa_pairwise=CCMP TKIP
ctrl_interface=/var/run/hostapd

As this is a new file and we want to preserve it across reboots, we should add an entry for this file into manifest file. This is how manifest file looks:

# cat manifest
etc/cfg/manifest
etc/master.passwd
etc/group
etc/cfg/rc.conf
etc/cfg/hostapd.wlan0.conf

rc.conf is where you specify your networking configuration:

# cat rc.conf
# Set the default system hostname
system_hostname="freebsd-wifi-build"

# Modules to load
kernel_modules="bridgestp if_bridge random"

# These interfaces are configured in-order
network_interfaces="arge0 wlan0 bridge0"

# Create arge0, no interface address
netif_arge0_enable="YES"
netif_arge0_type="ether"
netif_arge0_addrtype="none"
netif_arge0_descr="default"
netif_arge0_name="arge0"

# Create a bridge, flip on an IPv4 static address
netif_bridge0_type="bridge"
netif_bridge0_addrtype="static"
netif_bridge0_descr="default"
netif_bridge0_name="bridge0"
# These are bridge members w/ STP enabled
netif_bridge0_members_stp="arge0"
# These are bridge members w/ STP disabled
netif_bridge0_members="arge0 wlan0"
netif_bridge0_ipv4_address="192.168.1.20"
netif_bridge0_ipv4_netmask="255.255.255.0"
netif_wlan0_enable="YES"
netif_wlan0_type="wifi"
netif_wlan0_wifi_mode="hostap"
netif_wlan0_descr="default"
netif_wlan0_addrtype="none"
netif_wlan0_name="wlan0"
netif_wlan0_wifi_parent="ath0"
netif_wlan0_wifi_createargs1="country US regdomain FCC3"
netif_wlan0_wifi_createargs2="channel 1:ht/20 up"
netif_wlan0_wifi_hostapd_enable="yes"
netif_wlan0_wifi_hostapd_conf="/etc/cfg/hostapd.wlan0.conf"

A few things to notice here:
* arge0 is the ethernet interface
* wlan0 is the wifi interface
* bridge0 is the bridge interface connecting all of it together. It does the bridging of ethernet and wifi interfaces to send and receive bits (and bytes).

After making changes, to save them do:

# cfg_save
*** Storing configuration files from /etc/cfg/manifest -> /dev/map/cfg..
etc/cfg/manifest
etc/master.passwd
etc/group
etc/cfg/rc.conf
etc/cfg/hostapd.wlan0.conf
8 blocks
dd: /dev/map/cfg: end of device
0+2 records in
1+0 records out
65536 bytes transferred in 0.479322 secs (136726 bytes/sec)

And “reboot” the router. When it comes back, it _should_ have everything ready and setup which was not the case with my setup.

This is how interfaces looked:

# ifconfig
arge0: flags=8943 metric 0 mtu 1500
ether 64:70:02:cd:94:56
inet6 fe80::6670:2ff:fecd:9456%arge0 prefixlen 64 scopeid 0x6
nd6 options=21
media: Ethernet 1000baseT
status: active
arge1: flags=8802 metric 0 mtu 1500
ether 64:70:02:cd:94:57
nd6 options=21
media: Ethernet 100baseTX
status: active
ath0: flags=8843 metric 0 mtu 2290
ether 00:19:e0:66:66:68
nd6 options=21
media: IEEE 802.11 Wireless Ethernet autoselect mode 11ng
status: running
lo0: flags=8049 metric 0 mtu 16384
options=600003
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
inet 127.0.0.1 netmask 0xff000000
nd6 options=21
wlan0: flags=8943 metric 0 mtu 1500
ether 00:19:e0:66:66:68
inet6 fe80::219:e0ff:fe66:6668%wlan0 prefixlen 64 scopeid 0xa
nd6 options=21
media: IEEE 802.11 Wireless Ethernet autoselect mode 11ng
status: running
ssid STRCDR_11N channel 1 (2412 MHz 11g ht/20) bssid 00:19:e0:66:66:68
regdomain FCC3 country US ecm authmode WPA1+WPA2/802.11i
privacy MIXED deftxkey 3 TKIP 2:128-bit TKIP 3:128-bit txpower 30
scanvalid 60 protmode CTS ampdulimit 64k ampdudensity 8 shortgi wme
burst dtimperiod 1 -dfs
bridge0: flags=8843 metric 0 mtu 1500
ether 96:ce:f3:c8:09:b4
inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: wlan0 flags=143
ifmaxaddr 0 port 10 priority 128 path cost 33333
member: arge0 flags=143
ifmaxaddr 0 port 6 priority 128 path cost 2000000
#

As you can see everything looked sane and I could see it advertizing wifi network but when a client tries to associate, it would get stuck on “Obtaining IP address” and even if you assign static IP, client does not receive any packets.

We then looked at “etherswitchcfg” which provides ethernet connectivity for the board:

# etherswitchcfg
port0:
vlangroup: 1
media: Ethernet autoselect (100baseTX )
status: active
port1:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port2:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port3:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port4:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port5:
vlangroup: 0
media: Ethernet 1000baseT
status: active
vlangroup0:
vlan: 1
members 1,2,3,4,5
vlangroup1:
vlan: 2
members 0,5t
#

Now, as you can see the ethernet cable in WAN (blue) port shows up as port0 and it is in vlangroup1. But vlangroup1 is in vlan 2 and it is tagged. Moving ethernet cable from WAN port to one of the LAN ports fixed the problem.


# etherswitchcfg
port0:
vlangroup: 1
media: Ethernet autoselect (none)
status: no carrier
port1:
vlangroup: 0
media: Ethernet autoselect (100baseTX )
status: active
port2:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port3:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port4:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port5:
vlangroup: 0
media: Ethernet 1000baseT
status: active
vlangroup0:
vlan: 1
members 1,2,3,4,5
vlangroup1:
vlan: 2
members 0,5t
#

Now a client could associate and get an IP via dhcp from the incoming ethernet connection on the router. (No dhcpd is needed/running on the router itself).

To look at connected clients:

# ifconfig wlan0 list sta
ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG
08:11:96:f9:b2:ec 1 1 144M 38.5 0 4012 27616 EPS AQEHTRS RSN HTCAP WME

To debug, we can look at individual ports with “tcpdump -ni “.

Verbose debugging from ath wifi stack can be obtained by:

# wlandebug -i wlan1 +assoc
net.wlan.1.debug: 0x0 => 0x800000

Just as a side-note, hostapd proc is using the configuration we provided and is running:

# ps awwux | grep hostapd
root 176 0.0 6.6 13104 2176 - Ss 6:56PM 0:00.13 hostapd -P /var/run/hostapd.wlan0.pid -B /etc/cfg/hostapd.wlan0.conf

Again a *HUGE* Thanks to Mr FreeBSD Wifi (Adrian Chadd) for all the build scripts and help with the bring-up.


Installing FreeBSD on TP-Link TL-WR1043ND

Posted: July 1st, 2013 | Author: | Filed under: FreeBSD, wifi, wireless | Tags: | 11 Comments »

I decided to play with some wireless router and put FreeBSD on it. Adrian suggested TL-WR1043ND so I bought it.

I’ve used Adrian Chadd’s scripts do this. All the commands for flashing is also from him. A *huge* thanks to him.

I’ve also referenced OpenWRT page a lot.

Serial connection was the first thing I had to determine. Figured out serial pinouts from the OpenWRT page. Did my first soldering job:
image

Got usb to ttl serial cable to connect to the tx, rx and gnd pins.
image

After having serial connection setup, next thing was to load FreeBSD on it. (It comes with preloaded linux on it.) I grabbed image building tool from Adrian Chadd’s scripts and generated an image out of freebsd-head suitable for this board: TP-WN1043ND.factory.bin

Now, what you do is, plug usb end of the serial cable into laptop which FreeBSD presents as a cuaUN device. (Check your dmesg o/p to know value of “N”)
Connect to it via:

#cu -s 115200 -l /dev/cuaU0

Now power on the router and interrupt the boot process by typing “tpl” to drop into “uboot” prompt.
uboot looks like this:

ar7100> printenv
bootargs=console=ttyS0,115200 root=31:02 rootfstype=jffs2 init=/sbin/init mtdparts=ar9100-nor0:128k(u-boot),1024k(kernel),4096k(rootfs
),64k(art)
bootcmd=bootm 0xbf020000
bootdelay=1
baudrate=115200
ethaddr=00:1D:0F:11:22:33
ipaddr=192.168.0.2
serverip=192.168.0.5
stdin=serial
stdout=serial
stderr=serial
ethact=eth0

Environment size: 317/131068 bytes

Now starts the fleshing of firmware part. The first step below is *VERY* important…

# This erases the flash between uboot and the firmware configuration area. Whatever you do, don't mistype this!
erase 0xbf020000 +7c0000

Now, we should setup the tftp part so that we can get image we prepared from tftpserver (laptop) to tftpclient (router board). uboot is very specific about the IP addresses of both server and client. As per “printenv”, server IP *must* be 192.168.0.5 so assign that IP to the ethernet interface (via ifconfig alias). Run a network cable from laptop to one if the LAN ports of the router. (Original instructions suggest using the WAN port which did not work for me (because of which I wasted 2 frustrating weeks debugging)). Start tftpserver on laptop and copy prepared image in respective location (/tftpboot by default)

Let’s transfer/copy the image to router via tftp into RAM:

ar7100> tftpboot 0x81000000 TP-WN1043ND.factory.bin
dup 1 speed 1000
Using eth0 device
TFTP from server 192.168.0.5; our IP address is 192.168.0.2
Filename 'TP-WN1043ND.factory.bin'.
Load address: 0x81000000
Loading: T T T T T T T T #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############################
done
Bytes transferred = 8126464 (7c0000 hex)

Next step is to copy firmware image from RAM to the flash.

cp.b 0x81000000 0xbf020000 0x7c0000

Let’s boot the damn thing :-)

bootm 0xbf020000
## Booting image at bf020000 ...
Uncompressing Kernel Image ... OK

Starting kernel ...

CPU platform: Atheros AR9132 rev 2
CPU Frequency=400 MHz
CPU DDR Frequency=400 MHz
CPU AHB Frequency=200 MHz
platform frequency: 400000000
arguments:
a0 = 00000007
a1 = a1f77fb0
a2 = a1f78440
a3 = 00000008
Cmd line:argv is invalid
Environment:
envp is invalid
Cache info:
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r248805M: Wed Dec 31 16:00:00 PST 1969
hirenp@dummy:/home/hirenp/work/freebsd/head/obj/mipseb/mips.mips/usr/home/hirenp/work/freebsd/head/src/sys/TP-WN1043ND mips
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
real memory = 33554432 (32768K bytes)
avail memory = 24944640 (23MB)
random device not loaded; using insecure entropy
nexus0:
clock0: on nexus0
Timecounter "MIPS32" frequency 200000000 Hz quality 800
Event timer "MIPS32" frequency 200000000 Hz quality 800
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
gpio0: on apb0
gpio0: [GIANT-LOCKED]
gpio0: function_set: 0x2000
gpio0: function_clear: 0x0
gpio0: gpio pinmask=0x1c02ae
gpioc0: on gpio0
gpiobus0: on gpio0
gpioled1: at pin(s) 2 on gpiobus0
gpioled2: at pin(s) 5 on gpiobus0
gpioled3: at pin(s) 9 on gpiobus0
gpioiic0: at pin(s) 18-19 on gpiobus0
iicbb0: on gpioiic0
iicbus0: on iicbb0 master-only
iic0: on iicbus0
rtl8366rb0: at addr 0xa8 on iicbus0
rtl8366rb0: rev. 3
miibus0: on rtl8366rb0
ukphy0: PHY 0 on miibus0
ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus1: on rtl8366rb0
ukphy1: PHY 1 on miibus1
ukphy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus2: on rtl8366rb0
ukphy2: PHY 2 on miibus2
ukphy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus3: on rtl8366rb0
ukphy3: PHY 3 on miibus3
ukphy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
miibus4: on rtl8366rb0
ukphy4: PHY 4 on miibus4
ukphy4: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
etherswitch0: on rtl8366rb0
ehci0: at mem 0x1b000100-0x1bffffff irq 1 on nexus0
usbus0: set host controller mode
usbus0: EHCI version 1.0
usbus0: set host controller mode
usbus0 on ehci0
arge0: at mem 0x19000000-0x19000fff irq 2 on nexus0
arge0: Overriding MAC from EEPROM
arge0: finishing attachment, phymask 0000, proxy null
arge0: Ethernet address: 64:70:02:cd:94:56
arge1: at mem 0x1a000000-0x1a000fff irq 3 on nexus0
arge1: finishing attachment, phymask 0000, proxy null
arge1: Ethernet address: 64:70:02:cd:94:57
ath0: at mem 0x180c0000-0x180effff irq 0 on nexus0
ath0: eeprom @ 0x1fff1000
ath0: eeprom data @ 0xbfff1000
ath0: [HT] enabling HT modes
ath0: [HT] 2 RX streams; 2 TX streams
ath0: AR9130 mac 20.1 RF2133 phy 10.2
ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00d0
spi0: at mem 0x1f000000-0x1f00000f on nexus0
spibus0: on spi0
mx25l0: at cs 0 on spibus0
mx25l0: s25sl064a, sector 65536 bytes, 128 sectors
ar71xx_wdog0: on nexus0
ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
rtl8366rb0port0: link state changed to DOWN
rtl8366rb0port1: link state changed to DOWN
rtl8366rb0port2: link state changed to DOWN
rtl8366rb0port3: link state changed to DOWN
rtl8366rb0port4: link state changed to DOWN
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: at usbus0
uhub0: on usbus0
map/rootfs.uzip: 1123 x 16384 blocks
hwpmc: SOFT/16/64/0x67 MIPS24K/2/32/0x1ff
WARNING: WITNESS option enabled, expect reduced performance.
Root mount waiting for: usbus0
uhub0: 1 port with 1 removable, self powered
Trying to mount root from ufs:map/rootfs.uzip []...
warning: no time-of-day clock registered, system time will not be set accurately
May 24 18:55:57 init: login_getclass: unknown class 'daemon'
rtl8366rb0port1: link state changed to UP
*** Populating /var ..
*** Loading configuration files ..
*** Restoring from /dev/map/cfg ..
gunzip: unknown compression format
1+0 records in
1+0 records out
65536 bytes transferred in 0.088297 secs (742223 bytes/sec)
0 blocks
*** Completed.
*** setting up hostname
*** Load kernel modules
kldload: can't load bridgestp: File exists
kldload: can't load if_bridge: No such file or directory
kldload: can't load random: File exists
*** bringing up loopback ..
*** Starting networking via /etc/rc.d/base/net
dev.ath.0.txq_mcastq_maxdepth: 512 -> 32
sysctl: unknown oid 'dev.ath.1.txq_mcastq_maxdepth': No such file or directory
*** Interface: arge0: start
*** Interface: arge0: done
*** Interface: bridge0: start
bridge0: Ethernet address: 9e:de:f8:cb:67:4f
arge0: promiscuous mode enabled
bridge0: link state changed to UP
*** Interface: bridge0: done
*** Default password/login databases ..
*** inetd
*** Done!

FreeBSD/mips (freebsd-wifi-build) (ttyu0)

# df -k
Filesystem 1024-blocks Used Avail Capacity Mounted on
map/rootfs.uzip 17695 16769 -489 103% /
devfs 1 1 0 100% /dev
/dev/md0 828 8 756 1% /tmp
/dev/md1 828 56 708 7% /var
/dev/md2 828 436 328 57% /etc
#

Next up is _actually_ using this thing: creating wireless network and other fun stuff.


Resize partitions and zpool on the fly

Posted: June 20th, 2013 | Author: | Filed under: FreeBSD, storage | Tags: | No Comments »

This is how my disk layout looked before the change:

# gpart show -l
=> 63 250069617 ada0 MBR (119G)
63 250069617 1 (null) [active] (119G)

=> 0 250069617 ada0s1 BSD (119G)
0 115343360 1 (null) (55G)
115343360 117440512 2 (null) (56G)
232783872 17285745 4 (null) (8.2G)

# bsdlabel ada0s1
# /dev/ada0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 115343360 0 ZFS
b: 117440512 115343360 ZFS
c: 250069617 0 unused 0 0 # "raw" part, don't edit
d: 17285745 232783872 swap

# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada0s1a ONLINE 0 0 0

errors: No known data errors

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 54.5G 39.1G 15.4G 71% 1.00x ONLINE -

Above you can see only ada0s1a (55G) was being used by my zpool: zroot.
I wanted to extend it to use the ada0s1b (56G) too.

1) Set autoexpand to “on” on zroot

# zpool get autoexpand zroot
NAME PROPERTY VALUE SOURCE
zroot autoexpand off default

# zpool set autoexpand=on zroot

# zpool get autoexpand zroot
NAME PROPERTY VALUE SOURCE
zroot autoexpand on local

2) Tell geom: “I know what I am doing”

# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16

3) Delete the unused partition

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart show
=> 63 250069617 ada0 MBR (119G)
63 250069617 1 freebsd [active] (119G)

=> 0 250069617 ada0s1 BSD (119G)
0 115343360 1 freebsd-zfs (55G)
115343360 117440512 - free - (56G)
232783872 17285745 4 freebsd-swap (8.2G)

4) Resize the partition in use to include the unused space

# gpart resize -s 232783872 -i 1 ada0s1
ada0s1a resized

# gpart show
=> 63 250069617 ada0 MBR (119G)
63 250069617 1 freebsd [active] (119G)

=> 0 250069617 ada0s1 BSD (119G)
0 232783872 1 freebsd-zfs (111G)
232783872 17285745 4 freebsd-swap (8.2G)

# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada0s1a ONLINE 0 0 0

errors: No known data errors

5) Make zfs realize the fact that partition has been changed and make zpool
use the new partition which is actually the same one (ada0s1a).

# zpool online -e zroot ada0s1a ada0s1a
# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada0s1a ONLINE 0 0 0

errors: No known data errors

6) Look at the success (SIZE is now 110G)

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 110G 39.1G 71.4G 35% 1.00x ONLINE -


sshfs on FreeBSD

Posted: April 2nd, 2013 | Author: | Filed under: Uncategorized | 1 Comment »

With sshfs you can mount remote filesystem locally via ssh. It uses FUSE underneath.
Very quick and convenient.

On server:

You need to enable sftp from /etc/ssh/sshd_config.
Uncomment following line if it’s commented:
Subsystem sftp /usr/libexec/sftp-server

And restart sshd.

On client:

Create a mountpoint:
mkdir mount_shared

sshfs user@server:/home/user/share mount_shared/

I am using 9.0 stable on server and 10.0 current on client.
“sysctl vfs.usermount=1” is not needed as suggested by others.

sshfs_debug is useful to get debugs.