pfsense performance / Configs on ESXI (vmware)

[Visentinel 12]

Well, as you may or may not know pfsense has vmware optimisations, also there is going to be a hyperv optimised release in the near future.

 

i have taken my time before making a post about the performance and reliability of my setup, i simply wanted to be sure it was

 

A. Consistant

B. Stable

C. Learn the Do's and Dont's of pfSense on ESXI

 

Firstly heres the cool bits

 

I run two instances of pfSense, one runs my actual internet connection via an ADSL modem in bridge mode and pfSense connects using pppoe and the other runs the openVPN to AirVPN. Its a personal choice as i want to keep the VPN to its own sandbox so i can better secure the clearnet from machines that should use the VPN this way there can be ZERO leaks of any kind.

 

The hardware is a gigabyte G33M motherboard running an Intel Core2Quad Q6600 2.4GHZ cpu with 8GB of 800mhz DDR2 CAS 2.5

Only a Single Intel Pro 1000 GT PCI desktop adaptor (ESXI can virtualise all the NIC's you need)

 

This box boots up Vmware ESXI 5.5 and runs 2 instances of 64bit pfsense

Instance 1 is for internet, configured with 2 vCPU's and 1536mb of ram (squid uses it)

Instance 2 is for VPN, configured with 2vCPU's and 768mb of ram

 

2 other VM's running server 2012r2 also live on this ESXI host.

 

Perfomance analysis is easy, pfSense itself reports cpu usage poorly on PC CPU's becuase actual clock cycles used for networking related business arent showing in pfsense and neither is the openVPN crypto cpu usage showing But ESXI can report this usage perfectly. For a download over the VPN both instances are used, the PC uses the VPN pfsense as default gateway and the vpn connects to the airvpn server via the other pfsense and out the ADSL.

 

Here is the clock usage for a 1000kilobytes/sec download over bittorrent on the VPN

no AES-NI is been used (not supported)

 

pfsense internet uses 500-750 mhz

pfsense VPN uses      950-1100mhz

 

Combined cpu usage of the ESXI host (includes virtualisation overheads) is on avg 1600mhz - 2GHZ

this usage is offcourse spread out across 4 2.4ghz cores and ESXI does a fantastic job of spreading out the load across cores, you can also reserve cpu speed to individual VM's

 

The performance is Not fantastic in that the load is high if you compare what a cpu designed for networking would do.... but for me its fine coz i cant sustain more than 1400k/sec since despite having a 20mbit download i only have 1mbit upload and the VPN saturates the upload at arround 1400k/sec download.

It is my opinion that you need about 1ghz of a older core2 platform for every 8mbit of AirVPN download speed you need when only using 1 pfsense instance, in my case its 1.5x due to the other internet instance.

 

It is interesting that the VPN crypto only accounts for half of the overall usage of the VPN instance of pfSense, compare the 2 instances ! it takes on avg 600mhz for pfsense to "router" 1000k/sec on the internet instance.

Desktop CPU's suck at this =P

 

Now heres what you need to know before you go trying this yourself

 

when creating the VM select suse linux enterprise 10 64bit (the NIC should be the e1000)

 

NEVER NEVER select to "Insert my local MAC address " in pfSense nor enter a mac, BE SURE THIS FIELD IS EMPTY before you save your config.

What will happen is, and i have no idea why but pfSense if a mac address of any kind is in that field will cause ESXI's virtual NIC to crash, you will loose management connectivity to the ESXI host and the only solution is to reset button or initiate a reboot on the ESXI console.

This is especially important after setting up automatic startup in ESXI to autostart the pfsense instance.

 

Install the package named Open-VM-Tools, this allows for guest OS restarts and shutdowns and the vmware memory management.

[Ernst89 11]

I run virtual pfSense. There appears to be a significant system cpu overhead on the NICs vs non virtualised.

 

To the extent that 110Mb/s openvpn took apporximately 20% cpu on a non virtual  AMD Kabini 5350 (cheap cpu) and about 50% on a more powerful but virtualised haswell 4570s.

 

ESXi provides a passthrough "VT-d/IOMMU" capability which may reduce this virtual NIC overhead. Unfortunately I couldn't run ESXi on my new 2014 machines  due to hardware compatibility issues (SATA) and my old machine didn't support passthough  so I don't know how much it would help.

 

Finally as a note be careful of oracle virtualbox it doesn't support AES-NI hardware acceleration.

[Visentinel 12]

Friend of mine has ESXI 5.5 running on a intel haswell refresh 

 

Took us a few days but the 9 series chip-set was a major headache with the sata controller. The method is to get the driver and use the ESXI cd customiser to build the driver into the install cd.

 

Dont bother trying to get the onboard intel NIC working, we tried every way and all failed.Those Intel pro 1000 cards rock anyway 

[Ernst89 11]

Yep you've just described what I was doing with a Haswell gigabyte z97n motherboard, ESXi 5.5.

 

AIUI you don't need to install the SATA driver, you just have to add a reference in ESXi to say it is a valid device. I gave up when I realized i would need to modify the ESXi install usb drive. It seemed to me VMWare was making things difficult just for the sake of it.

 

I would also need at least one of the on board Nics  to work  (Intel or Atheros it has both) as I would use the only pci-e socket for a tv card (the only always on server I have is my HTPC).

 

I did think of running a separate  pfSense server but that would mean another 40 watts + extra space + extra hardware which didn't seem to offer enough advantage, electricity alone would be $80 per year .

 

I would be interested to know if ESXi with NIC passthrough did improve the cpu load over say VMWare Player or Workstation, if your friend knows.