Lesson's Learned the hard (expensive) way

I've been running my own home lab since my parents upgraded their desktop at home back in 2008 and left me with a computer that just needed a hard drive to function. I used that machine as a catalyst for me to learn more about Linux (and computers without a graphical user interface) and how to run my own systems from home, including learning how to install a LAMP stack and learn more about the setup for early on development items, which I used to do things like develop PHP applications while I was hosting office hours as a programming TA and needing to be able to handle the students coming through the very packed office hours time.

Once I had my Linux server set up, I had set up a computer at home specifically to be my media server, in College it was connected to my cable subscription and I was using Windows Media Center (on Windows 7) to record live TV and then used XBOX 360's Media Center Extender to be able to watch those things live. I had hooked up a number of other software plugins to do things like automatic commercial detection and skipping for the playback (like TiVo) as well as have a way to take the recordings and commercials detected, and strip them for later archiving and playback. When Cable TV moved to Digital-first, I had to get a digital tuner to support that mess, and ended up being one of the first people in Iowa City on Mediacom that was able to simply get a CableCARD rental as I had purchased a CableCARD TV Tuner for my computer. Getting a CableCARD was an absolute blast when you don't have a "retail set top-box" but instead a computer.

Back when my primary server was Windows, I had a great strategy for creating system images (using the Windows system image creator as part of the Backup and Restore settings), but they've both made that more difficult (thanks to the pus for Windows to use the cloud for backups via OneDrive) as well as it only working with Windows. Once I moved down to Kansas City for my first job, I no longer had access to cable, and migrated all of my "server" things over to Linux full-time. The primary reason for moving to Linux was for better support for multiple drives. I had amassed a large amount of TV show recordings from close to 5 years of data, and had supplemented all of my media server items with my personal collection of Movies that I had from DVDs and Blu-rays that I had previously purchased and watched. This led me to leveraging Plex rather than Windows Media Server for future work. (Windows 8 dropped Media Server, so there was no way to continue my existing process) Back when I first did the migration, I had worked by installing every application on the bare Operating System and configuring things appropriately, and then working to leverage Apache (or even nginx later) to set up reverse proxying to those endpoints so that I could do more automation.

I've historically been pretty good about backing up the big data (movies, tv shows, etc.) that have been part of my collection because some of those things would be nearly impossible for me to get back, but I've always been pretty bad at OS-level items and have taken things like cloud syncing for granted. Recently, I had my primary OS drive crash on my primary server at home and that's been an interesting process. I've thought (for a long time) that I had more than enough systems in place to handle backing up my configurations, but had somewhat always relied on my primary OS drive being in place to do so. This blog, for instance (prior to this incident) was only present on the primary OS drive of my primary server at home, so when that OS drive went up in flames (so to speak) it meant that I lost all of my historical posts that I had made. While that wasn't the worst thing in the world (I don't think anyone REALLY reads these blogs) it did mean things like my traefik post (which powers my full reverse proxy configuration) was gone, so for me to recover that, I leveraged the internet archive of my blog posts. Most of this was because I always thought the critical data on my server(s) was the large media files, or large backup files from other areas, not the actual configuration for running the applications on the server (like Plex which holds mine and all of my friends' play/watch history). I have 3 servers at home, one powers this blog and most of the public-facing endpoints that I have, one powers a virtualization server running a few gaming servers and my home assistant instance, and a third which turns on every night to do a backup sync across a few different areas. This backup used to do a weekly sync of my larger file content, but now also does a weekly sync of my configuration and application data sync. I do a daily archive of my entire application folder structure and that syncs weekly to my "cold storage" backup server. One of these days, I might actually properly archive to tape, but man that's expensive and not sure it's worth it right now.

The reason why my backup strategy has changed is that when my primary OS drive failed (it had bad sectors) I had no backup of that data, and I REALLY wanted a good portion of it back. It ended up costing me $1600 and 4 weeks of time to recover that data so I could get things back to normal. I've been happy that I've been able to return to normal, but during that time, I had to deal with a lot of manual "IT helpdesk" like questions that I didn't enjoy having to do on top of my real job too. I had lost things like my mail server, which took a good amount of work to set up 6 years ago was great and I'm not sure I'd remember how to do it now. This blog had posts that I couldn't recover and I wanted to have that history. I have a personal "OSS" version of artifactory running for a local cache of things like maven artifacts that again I set up years ago and I'm not sure it'd be easy to set up myself again. Most of these things I've set up now only work "via cloud services" these days, and I'm just not interested in paying for someone else to host things where I'm just playing around.

In the end, I've been really happy with being able to do a bunch of things like setting up this blog, and a mail server, and all sorts of development tools historically. I've learned so much from this process that I think a lot of people take for granted when it comes to actually building up the skills to not only be a software engineer, but understand the ops side of things as well is an invaluable skill that I think has helped me historically. I think my next step in this all is trying to figure out how to make some of these things more automatable via something like terraform so I can configure all of the services that I generally rely on in an easier way, and find a way to externalize some of my data to the cloud for safekeeping. (that's a next step) Anyway, that's my crazy story about how I've had to spend way more money than I should've been willing to to simply recover data that I should've been backing up and handling properly this whole time.