Background
While I have been a pfSense user for about 2 years, I decided to see how the FreeBSD base OS would stack up against Vyatta, a Linux base free router software package. Without going into too much detail, main differences currently is a more CLI centric interface (close to JunOS “set” syntax) for Vyatta in comparison to a comprehensive WebGUI for pfSense. Beyond that there are many features betwene the two which I will not get into here, as my primary question here is speed.
Purpose
The goal here is to just test IP forwarding speed between the two platforms in completely stock configuration. I’m sure there are tweaks I could have made either way, but in order to be consistent, no changes were made from the base install on either platform. The hardware used is a low power Jetway mini PC. I am fully aware this is not server class hardware by any means, but until I have unlimited funds, this will have to do. Hardware specifications listed below.
Methodology
Case/Model: Jetway JC104-B-J7F4K1G5D
CPU: Via C-7D 1.5 GHz
North Bridge: VIA CN700
South Bridge: VIA VT8237RP
Memory: 1GB DDR2 533 Kingston
LAN NICs: Dual Realtek RTL8110SC 10/100/1000
Notes on the hardware used:
- The Realtek RTL8110SC series are frankly not good hardware. They are cheap, and do not have many of the features of a solid Intel or Broadcom NIC. Again, working with what I have.
- The Via C-7D was saught after because of its extremely low power rating and built in AES cryptographic acceleration for VPN features.
- Only one NIC will be used in these tests, since in reality this in a “router on a stick” configuration which ties together several VLANs.
Client Hardware
Case/Model: Lenovo T400 series
CPU: Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz
Memory: 3GB
NIC: Intel(R) 82567LF Gigabit Network Connection
OS: Windows 7 Ultimate 64bit
The test was conducted by simply plugging in a Cat5e cable from the client workstation to the router, putting the router into server mode with iperf and starting the iperf client. No fancy options were used on iperf either, and all was left on defaults. The only option used was the “-d” command to perform a bidrectional test, which can be seen in the second set of benchmarks for each platform. Results are below.
Data
[Pfsense 1.2.3 Stock Installation]
[One way test]
Server listening on TCP port 5001
TCP window size: 63.7 KByte (default)
————————————————————
[ 10] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49661
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 251 MBytes 210 Mbits/sec
[ 11] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49662
[ ID] Interval Transfer Bandwidth
[ 11] 0.0-10.0 sec 212 MBytes 177 Mbits/sec
[ 10] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49663
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 257 MBytes 216 Mbits/sec
[ 11] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49664
[ ID] Interval Transfer Bandwidth
[ 11] 0.0-10.0 sec 251 MBytes 211 Mbits/sec[Bidirectional -d tests]
————————————————————
Client connecting to 192.168.6.5, TCP port 5001
TCP window size: 88.6 KByte (default)
————————————————————
[ 12] local 192.168.6.1 port 2161 connected with 192.168.6.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 12] 0.0-10.0 sec 207 MBytes 174 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 75.2 MBytes 63.1 Mbits/sec[Vyatta 5.0 CE Stock Installation]
[One Way Test]
Server listening on TCP port 5001
TCP window size: 272 KByte (default)
————————————————————
[ 10] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49661
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 632 MBytes 530 Mbits/sec
[ 11] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49662
[ ID] Interval Transfer Bandwidth
[ 11] 0.0-10.0 sec 631 MBytes 529 Mbits/sec
[ 10] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49663
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 714 MBytes 598 Mbits/sec
[ 11] local 192.168.6.1 port 5001 connected with 192.168.6.5 port 49664
[ ID] Interval Transfer Bandwidth
[ 11] 0.0-10.0 sec 714 MBytes 598 Mbits/sec[Bidirectional -d tests]
————————————————————
Client connecting to 192.168.6.5, TCP port 5001
TCP window size: 272 KByte (default)
————————————————————
[ 12] local 192.168.6.1 port 2161 connected with 192.168.6.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 12] 0.0-10.0 sec 334 MBytes 280 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 338 MBytes 283 Mbits/sec
Results
As you can see, the Vyatta base install completely blew pfSense out of the water by a factor of around 250% on the host to host tests. Bidirectional tests were also much more consistent, and several more bi-directional tests were performed with very similar results. Without a larger variety of hardware to test on, it’s hard to make an overall conclusion, but for this hardware, the choice is clear for performance.
If anyone has hardware to donate or simply would like to send results of their own tests using mostly the same methodology, please let me know through the comment section and it would be greatly appreciated. Any insights on why this could be are also welcome. I suspect the Linux kernel has greater support for those NIC drivers, but perhaps the scheduling is more aggressive overall.
Chris Buechler | 27-Mar-10 at 3:36 pm | Permalink
I suspect you have a major discrepancy in what you’re comparing, and your testing methodology is flawed.
First, testing traffic initiated by the firewall/router in question, rather than its ability to route traffic, isn’t good even for a router on a stick deployment. If you’re VLAN routing, put one end if iperf on one VLAN and the other on another VLAN, the device you’re testing should not be one of the endpoints.
Second, make sure you’re actually testing something comparable – do you have filtering enabled on Vyatta? If you disable PF on pfSense, it’s up to 10 times faster strictly routing traffic with no filtering abilities. If you’re testing without any filtering on Vyatta, disable PF under System > Advanced to get a proper comparison. This is true even for traffic initiated/terminated by the firewall as it still has to go through the filter.
Third, yeah the Realtek hardware sucks. The Linux driver may suck less than the FreeBSD 7.2 driver, which may explain the entire difference if all else is equal. Testing on a FreeBSD 8.x release may show a difference, though the only pfSense releases we have on 8.x at the moment have kernel debugging enabled so that wouldn’t be a fair comparison.
The last I did a side by side comparison of Linux to FreeBSD packet filtering performance (which is what you’re attempting to do here) using decent hardware, the two performed comparably.
Andrew | 27-Mar-10 at 6:34 pm | Permalink
Chris,
Absolutely agree with you on almost all points there. I hope this didn’t sound like a Rah Rah! post for Vyatta because it’s not. Most of my experience is with pfSense and I’ve been very happy with it so far. You guys have done an amazing job with that project, to say the very least.
Now, all shameless flattery aside, yes. I do need to have a more exact testing methodology. Performing actual by-VLAN routing instead of traffic destined for the device itself would be a much better test, and one I will actually be performing soon to look into this. I’m totally aware that pfSense is not involved with the driver selection and maintenance process, so if this is due to NIC driver, this is not an accurate comparison. I did not have any active PF rules for any interfaces, as I said both installs were stock (minus setting up interfaces of course), but I should turn off PF completely as you stated.
This could be completely to do with the hardware this was running on, so anyone who looks at this and says to themselves “That’s it pfSense is crap, I’m switching today!” needs a reality check. To be honest I was trying to get most of this testing done and functional internet back before I went to bed, so things were rushed. I asked for some criticism and I rightfully got it. I’ll be re-running these tests with a more explained and thoroughly thought out methodology, and post an update when completed.
Thanks for calling me out Chris. And even though I mostly support Cisco and HP gear where I work, I know 99% of the situations could be solved with pfSense, and I make sure to always mention it.
Cheers,
Andrew
Pedro Enrique | 31-Mar-10 at 3:26 pm | Permalink
Also make all the test using the same TCP window size (you can do it with the -w option, eg. -w 85K) or the comparative will not be fair.
Cheers,
Pedro Enrique
Renchu Mathew | 06-Apr-10 at 7:13 am | Permalink
Hi Andrew,
I am much interested on Vyatta. Could you pls guide me how to configure the firewall rules on vyatta. I know the basic configuration of Vyatta like setting up interfaces etc..
thanks
Andrew | 06-Apr-10 at 8:25 am | Permalink
Renchu,
I would suggest going to Vyatta’s main site for that, their documentation is pretty straightforward.
http://www.vyatta.com/downloads/documentation.php
You need to register, but the account is free and not a big deal. The documentation doesn’t do a lot of hand-holding, but it at least gives a few examples. Hope that helps.