Interesting that their previous server is just one or two models up from the server I recently grabbed foe $150 on eBay.
I'm just now getting serious about a homelab and it's shocking how much compute you can get for peanuts from enterprise surplus. I've had this machine (1U poweredge r620) for about a year and I'm already itching for an upgrade. Hopefully something with a lot of 3.5" drive bays for long-term data hoarding.
I have three R620s. They are shockingly quiet after being booted, even under pretty decent load. If you tune C-states for energy conservation in the BIOS, they idle at just under 100W each. With moderate load (like you'd have in a homelab), around 125W. According to iDRAC, the peak they've seen is about 350W.
For better apples-to-apples, each of them has 2x E5-2650v2, 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD (mounted in a PCIe adapter).
As a comparison, I also have a Supermicro X11SSH-F with a bunch (9) of spinning disks. It has a single E3-1220 v5, 1x NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the relative power consumption of HDDs, that's a pretty impressive difference IMO.
My entire rack (the aforementioned servers, and some Unifi networking gear) pulls around 500-600W in total. When the backup server (Supermicro X9DRi-LN4F+; dual-socket something-or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks on daily to ingest ZFS snapshots, the total consumption is around 700W, I think. Maybe 750W.
Tons of 25mm fans, dual socket, and a bunch of sticks of RAM add up.
I'm not trying to say they sip power, just that not every rack server is going to bankrupt you from power in most parts of CONUS, nor is a 1U guaranteed to be a screaming banshee. Assuming you had three of them, and nothing else, 300W over a month is still only $50/month with $0.25/kWh, which is about as high as I think it gets anywhere. The average is about half that. While not cheap, if you can afford a rack and three servers, you can probably afford the power.
If you're in the UK where it's somewhere around double that, then yeah maybe it's not really worthwhile.
It's also of course worth pointing out that for most employers, the useful skills you can get from a homelab can also be had from EC2s. Even if your job does run a datacenter, it's highly likely that they already have an infra team whose job it is to deal with the hardware.
Like most people on r/homelab, it started out with Plex. Rough timeline/services below:
0. Got a Synology DS413 with 4x WD Red 3TB drives. Use Playstation Media Server to stream videos from it. Eventually find some Busybox stuff to add various functionality to the NAS, but it had a habit of undoing them periodically, which was frustrating. I also experienced my first and (knock on wood) only drive failure during this time, which concluded without fanfare once the faulty drive was replaced, and the array repaired itself.
1. While teaching self Python as an Electrical Distribution Engineer at a utility, I befriended the IT head, who gave me an ancient (I think Nehalem? Quad-core Xeon) Dell T310. Promptly got more drives, totaling 7, and tried various OS / NAS platforms. I had OpenMediaVault for a while, but got tired of the UI fighting me when I knew how to do things in shell, so I switched to Debian (which it's based on anyway). Moved to MergerFS [0] + SnapRAID [1] for storage management, and Plex for media. I was also tinkering with various Linux stuff on it constantly.
1.1 Got tired of my tinkering breaking things and requiring troubleshooting/fixing (in retrospect, this provided excellent learning), so I installed Proxmox, reinstalled Debian, and made a golden image with everything set up as desired so I could easily revert.
1.2 A friend told me about Docker. I promptly moved Plex over to it, and probably around this time also got the *Arr Stack [2] going.
2. Got a Supermicro X9DRi-LN4F+ in a 2U chassis w/ 12x 3.5" bays. Got faster/bigger CPUs (E5-2680v2), more RAM, more drives, etc. Shifted container management to Docker Compose. Modded the BIOS to allow it to boot from a NVMe drive on a PCIe adapter.
2.1 Shifted to ZFS on Debian. Other than DKMS occasionally losing its mind during kernel upgrades, this worked well.
2.2 Forked [3] some [4] Packer/Ansible projects to suit my needs, made a VM for everything. NAS, Dev, Webserver, Docker host, etc. Other than outgrowing (IMO) MergerFS/SnapRAID, honestly at this point I could have easily stopped, and could to this day revert back to this setup. It was dead reliable and worked extremely well. IIRC I was also playing with Terraform at this time.
2.3 Successfully broke into tech (Associate SRE) as a mid-career shift, due largely (according to the hiring manager) to what I had done with my homelab. Hooray for hobbies paying off.
3. Got a single Dell R620. I think the idea was to install either pfSense or VyOS on it, but that never came to fruition. Networking was from a Unifi USG (their tiny router + firewall + switch) and 8-port switch, with some AC Pro APs.
4. Got two more R620s. Kubernetes all the things. Each one runs Proxmox in a 3-node cluster with two VMs - a control plane, and worker.
4.0.1 Perhaps worth noting here that I thoroughly tested my migration plan via spinning up some VMs in, IIRC, Digital Ocean that mimicked my home setup. I successfully ran it twice, which was good enough for me.
4.1 Played with Ceph via Rook, but a. disliked (and still to this day) running storage for everything out of K8s b. kept getting clock skew between nodes. Someone on Reddit mentioned it was my low-power C-state settings, but since that was saving me something like ~50 watts/node, I didn't want to deal with the higher power/heat. I landed on Longhorn [5] for cluster storage (i.e. anything that wasn't being handled by the ZFS pool), which was fine, but slow. SATA SSDs (used Intel enterprise drives with PLP, if you're wondering) over GBe aren't super fast, but they should be able to exceed 30 MBps.
4.1.1 Again, worth noting that I spent literally a week poring over every bit of Ceph documentation I could find, from the Red Hat stuff to random Wikis and blog posts. It's not something you just jump into, IMO, and most of the horror stories I read boiled down to "you didn't follow the recommended practices."
5. Got a newer Supermicro, an X11SSH-F, thinking that it would save power consumption over the older dual-socket I had for the NAS. It turned out to not make a big difference. For some reason I don't recall, I had a second X9DRi-LN4F+ mobo, so I sold the other one with the faster CPUs, bought some cheaper CPUs for the other one, and bought more drives for it. It's now a backup target that boots up daily to ingest ZFS snapshots. I have 100% on-site backups for everything. Important things (i.e. anything that I can't get from a torrent) are also off-site.
6. Got some Samsung PM863 NVMe SSDs mounted on PCIe adapters for the Dells, and set up Ceph, but this time handled by Proxmox. It's dead easy, and for whatever reason isn't troubled by the same clock skew issues as I had previously. Still in the process of shifting cluster storage from Longhorn, but I have been successfully using Ceph block storage as fast (1 GBe, anyway - a 10G switch is on the horizon) storage for databases.
So specifically, you asked what I do with the hardware. What I do, as far as my family is concerned, is block ads and serve media. On a more useful level, I try things out related to my job, most recently database-related (I moved from SRE to DBRE a year ago). I have MySQL and Postgres running, and am constantly playing with them. Can you actually do a live buffer pool resize in MySQL? (yes) Is XFS actually faster than ext4 for large DROP TABLE operations? (yes, but not by much) Is it faster to shut down a MySQL server and roll back to a previous ZFS snapshot than to rollback a big transaction? (often yes, although obviously a full shutdown has its own problems) Does Postgres suffer from the same write performance issue as MySQL with random PKs like UUIDv4, despite not clustering by default? (yes, but not to the same extent - still enough to matter, and you should use UUIDv7 if you absolutely need them)
I legitimately love this stuff. I could quite easily make do without a fancy enclosed rack and multiple servers, but I like them, so I have them. The fact that it tends to help my professional growth out at the same time is a bonus.
Those are pretty awful by current standards - you could buy a midrange minipc or laptop from a few cpu generations ago and get similar/better compute performance for much lower power draw.
If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either.
Although you said you were thinking of upgrading, so you're probably aware of this already.
>If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either.
It has nothing to do with the quality or reliability of the mains supply; ECC increases reliability via protecting data integrity from bitflips caused by cosmic rays etc.
Oh, fully agree. My base spec M1 Air destroys them in anything single-threaded. I got them because I wanted three nodes to play around with various things requiring HA, and I didn't want to just virtualize that. That said, they _are_ virtualized. Every server I have runs Proxmox, with templated VMs built from Packer + Ansible. I'm currently in the process of shifting K8s providers; right now they're running k3os but since Rancher has killed that, it's forever stuck at v1.21, so I'm shifting to Talos. I have the Terraform for them working fine, just need to set up a PXE server so they'll pick up the desired static IP when being created.
> you aren't really gaining anything from ECC ram either
Sibling comment to mine addressed this; to add on, I disagree. Cosmic rays aside, there's a ton of EMI inside a computer, and anything I can do to help prevent data corruption is a win in my book. Torvalds agrees, if that carries any weight with you. [0]
As far as cost, it costs me ~$30-40/month to run in electricity, and some small amount of A/C load (roughly equivalent to 5 extra people existing in the house) [1].
A friend of mine built a k8s system with Raspberry Pis, back when they were cheap and available, which burned very little electricity and produced no detectable heat.
Yep! I have an RPi 4, and at one point it was running Hypriot [0] on it, with some lighter containers being handled by it instead of the larger x86 nodes. I'm pretty sure I also had it running as part of a K8s cluster for a brief time.
The absolute maximum I've ever seen it hit is 400W. At my electricity rate, the theoretical maximum is something like $15/month. I'm perfectly fine with that, it's well within my hobby budget.
Really, that's what it is to me, a hobby. It doesn't have to be useful or profitable, just an interesting way to spend my free time tinkering
I'm just now getting serious about a homelab and it's shocking how much compute you can get for peanuts from enterprise surplus. I've had this machine (1U poweredge r620) for about a year and I'm already itching for an upgrade. Hopefully something with a lot of 3.5" drive bays for long-term data hoarding.