29 Sep 2014

New ESXi Whitebox Servers for Home Lab

I’ve been thinking about retiring my old home lab server hardware for some time now. I’ve had two little HP ProLiant ML110 G5 servers for 5+ years. They’ve been good little machines and didn’t cost too much to run, but I can now tell that time has taken its toll on them. They each have a dual core Intel Xeon processor and maxed out at 8GB of RAM. With the management components of products such as vSphere, vCAC, vCD, etc. nowadays requiring at least 8GB per appliance, these machines have basically been made obsolete by the requirements of most enterprise applications today.

 

So I needed to find some hardware to replace these two servers. I needed to find a server, or “whitebox” that meets the following requirements:

  1. Needs to be affordable to purchase and to run 24x7;
  2. Needs to be quiet;
  3. At least a Quad-core processor;
  4. Needs to have 32GB of RAM per host.

I’ve been looking around for “small to medium business” class servers, but I’ve not quite been able to spec something up that met all my requirements, and when they did, I just couldn’t justify the expense. The alternative was to buy decommissioned DL380s from eBay, but then I’d be footing a pretty high energy bill, and not to mention having to live with the constant noise of server cooling fans, blowing out the heat that resembles the money I’d be spending in energy bills, from the back of the server.

So, I did what any other techie in my position on a budget would do. I built my own whitebox to run ESXi 5.5 on. When I first purchased my ML110 servers, it was in the ESX 3.0 days (remember those). ESX 3 was very fussy about hardware and literally wouldn’t install on most non-HCL hardware. Today, ESXi 5 installs on almost anything (although not supported when run on non HCL hardware).

After having a look at what’s available on the market for mid to high end desktop computer components, I realised that there is a lot of processing power being packed into today’s desktop class products, and that at a fraction of the cost compared to enterprise server components.

Intel v.s. AMD:

Considering what I’ll be using the servers for, to run vCloud Suite products mostly, I determined that AMD offers the best value for money. They provide processors that fit my requirements perfectly. Although Intel provides premium processors which in my opinion are far superior to AMD processors in terms of performance vs. energy consumption, AMD offers processors with lots of cores (perfect for virtualisation), high clock speeds (not that I’m all that fussed about maximising GHz per core) and all the processor extensions required for the workloads I’d be running, at a fraction of the cost of an equivalent Intel processor. Yes sure, if I was building a new gaming machine, or a flight simulator, or a vSphere environment for a customer production environment, then I’d push for Intel, but for my home lab, there’s not much sense in spending the additional $$$.

So, with the processor vendor decided, here’s what I came up with:

  1. AMD FX-8320E 8-Core FX Series CPU = £106.6 Incl. VAT*
  2. Asus M5A78L-M/USB3 Socket AM3+ Motherboard (Micro-ATX) = £49.98 Incl. VAT
  3. 2x CORSAIR CML16GX3M2A1600C9 (32GB Dual Channel DDR3) = £247.58 Incl. VAT
  4. 350W BeQuiet PURE L8 BN221 Power Supply = £35.22 Incl. VAT
  5. Gigabyte GZ-MA02 Case (Micro-ATX) = £24.00 Incl. VAT

Total = £463.38 Incl. VAT

*For you guys in the US, VAT in the UK is sales tax in the US.

You’ll notice that there are no HDDs or SSDs included in the hardware list above. That’s because I’m not playing with VSAN or any other local storage or caching at the moment. Storage is currently provided by my Synology DS1512+ NAS.

All these components arrived on Saturday, 27 September 2014, and I can report that with the exception of one small issue (which I’ll explain below), ESXi 5.5 Update 2 is running marvellously on the whitebox.

Issues

Well, there is one catch here that could be a serious issue for those who aren’t planning on installing an additional supported NIC. The on-board Realtek 8111E/F PCIe Gigabit LAN controller is not supported and not even detected by ESXi 5.5. This isn’t an issue for me as I had installed a quad port 1Gbps NIC card that was sitting on my desk for a while, which is supported, so I hadn’t even noticed the on-board NIC wasn’t showing up until I came to configuring the networking.

However, if you try and install ESXi on a system built with this motherboard or any other motherboard with the Realtek 8xxx series NIC without installing a supported NIC first, then I’m afraid ESXi won’t even install.

I did see on the community forums and some blog posts that suggest there are unsupported drivers floating around, that might enable the on-board NIC, but I’m not too fussed with it at the moment as I have a quad NIC card installed, which probably cost me about £20 off eBay.

 

Written by  12 comments
Last modified on Tuesday, 09 December 2014 15:57
Rate this item
(0 votes)

Comments (12)

  1. Gomez

Nice setup. I have basic knowledge of server and have a question for setting up a vSphere5.5 home lab. What is best practice to setup my Luns? I have have 4x146GB serial ssci 10k.
Do I setup ESXi on 1x146GB and create a large LUN with 3x146GB?
I...

Nice setup. I have basic knowledge of server and have a question for setting up a vSphere5.5 home lab. What is best practice to setup my Luns? I have have 4x146GB serial ssci 10k.
Do I setup ESXi on 1x146GB and create a large LUN with 3x146GB?
I will be having a nested ESXi.

studying for VCP5-DCV

My server
HP Proliant G6
2x Intel Xeon CPU E5540 2.53GHz
24GB RAM

Read More
  Attachments
 
  1. Rynardt Spies    Gomez

Gomez,

Thank you for the comment. Based on the server hardware you have specified, I assume that you would have a RAID controller present in that server. If that is the case, I would suggest 2 options:

Option 1. Create a single RAID-5 volume...

Gomez,

Thank you for the comment. Based on the server hardware you have specified, I assume that you would have a RAID controller present in that server. If that is the case, I would suggest 2 options:

Option 1. Create a single RAID-5 volume which includes all 4 disks. You can then install ESXi 5.5 on the RAID volume which will provide your ESXi installation with disk redundancy. The installation will use less than 5GB on the disk for it's partitions. The remaining space on the array can then be used to create a VMFS5 partition for your VM datastore.

Option 2. Install ESXi on a USB memory stick (8GB recommended). Create a RAID-5 array consisting of all 4 disks and format he entire array with VMFS5 for your VM datastore.

Regards,

Rynardt

Read More
  Attachments
 
  1. Gustavo    Rynardt Spies

Thanks for the advice. I will try and let you know my results.

  Attachments
 
  1. Carlos

Hi,

I'm thinking about building something similar with 32 GB to deploy 10x to 20x Cisco CRS1000v for my CCIE studies.

My other option is building with a Xeon Quad Core E-1225V3.

what I like of AMD is the number of Cores. However I'm not sure...

Hi,

I'm thinking about building something similar with 32 GB to deploy 10x to 20x Cisco CRS1000v for my CCIE studies.

My other option is building with a Xeon Quad Core E-1225V3.

what I like of AMD is the number of Cores. However I'm not sure if this setup will meet my expectations.

Can you provide me your experiences or thoughts about it? Appreciate your help.

Read More
  Attachments
 
  1. Rynardt Spies    Carlos

From my experiences with this AMD processor, I've got no reason to believe that it does not provide sufficient processing power. I've overcommitted and stressed the processor pretty heavily, with no issues.

However aside from performance, I...

From my experiences with this AMD processor, I've got no reason to believe that it does not provide sufficient processing power. I've overcommitted and stressed the processor pretty heavily, with no issues.

However aside from performance, I don't know how running multiple 1000v on one host would practically work?

Read More
  Attachments
 
  1. Aditya Sahu

HI

Thanks for the valuable information about setting up the lab. Can you also please elaborate on the nics you are using.
I have got exactly same setup like yours with onboard reeltech nic. I am planning to put some extra nic for added...

HI

Thanks for the valuable information about setting up the lab. Can you also please elaborate on the nics you are using.
I have got exactly same setup like yours with onboard reeltech nic. I am planning to put some extra nic for added functionalities.
It would be great if you can tell me the make and model of nic . Also i would appreciate if you can let me know what networking components are you using.


Thanks
Aditya

Read More
  Attachments
 
  1. Ben

Hi,

Does anyone know if this setup supports USB passthrough ?

Thanks,
Ben

  Attachments
 
  1. ptrac3

Very informative article, what's the name of your quad NIC that it's support by ESXI? Thank you

  Attachments
 
  1. Rynardt Spies    ptrac3

I think I'm currently using an either an Intel PT1000 or a GT1000. I think it's the GT. Will check tomorrow and get back to you.

  Attachments
 
There are no comments posted here yet

Leave your comments

Posting comment as a guest. Sign up or login to your account.
0 Characters
Attachments (0 / 3)
Share Your Location

Microsoft MVVM and Caliburn Micro: My head still hurts trying to wrap my mind around this, even after years of trying!
Follow Rynardt Spies on Twitter