My useless advice: Do it in phases as you learn.

• Start off with Yunohost. It is simple to get started and works pretty well. Try different apps to see what you like and what might be worth using for real. Just make sure that you keep in mind this is more of a “proof of concept” for testing things. Unless you plan to purchase another mini pc later.
• When you feel like you have out grown it and want to start learning more about things, you can move to something like Proxmox. This allows you to create virtual machines and play with containers (docker/lxc). If you plan well, you can back up your Yunohost data and configs to another drive, wipe Yunohost install and replace it with Proxmox. Then install a VM running Yunohost and restore your data and configs you previously backed up.
• Then you can start playing with lxc containers and docker containers.
• If you can get a second machine with multiple drives, install TrueNAS or OMV. Use that to store all of your data on NFS drive that you mount from your Proxmox VMs and containers.

Years ago I used to run a linux server with everything installed under Apache virtual directories and fought the constant upgrade cycle. Life got in the way and I gave up on it until the pandemic slowed life down enough for me to start playing again. So I went the Yunhost route on an old Mac Mini. I now have a 3 node Proxmox cluster with Yunhost in a VM (with a dozen apps running on it) and another 15-20 containers running under either lxc or docker. I eventually purchased a cheap NAS device for data storage so that I could make use of the Proxmox fail over capabilities.

If your mini pc has the capability for two drives, install the OS on one and store data on the other (unless/until you get a second pc/NAS).

I wouldn’t do Windows, Linux will give you freedom to use docker for most things that you might want to host. As for which distro use whatever you find nice, there’s not going to be much difference. Some of the things people are suggesting are great for extremely advanced use cases, for just spinning up some services whatever you feel more comfortable would be best.

Personally… Id spin up a Ubuntu desktop instance for Jellyfin and Sonarr. I went through a few different operating systems before I went back to Ubuntu and…it just works. Lots of documentation on installing those services that way. Jellyfin is very happy behind Sonarr… I’d stay away from Windows for any kind of selfhosting because I don’t trust them

I’d recommend Debian Linux. It’s free, stable, has all the software you’ll need with long term support, ton of online resources and communities to learn from. You can start with or without UI.

I recommend debian 12 works like charm

I highly recommend Ubuntu Desktop for homelab stuff.

I recently set up my self-hosted system and went down this path:
Ubuntu Server - decided against it because terminal only is tedious sometimes
Ubuntu Desktop - easy, popular (so more documentation), comes ready with Docker
Decided to use Yacht for my container manager and have been happy with it so far, super simple compared to other options out there but you still have the control you need

current containers:

• Home Assistant to unify all smart home gadgets
• Nginx Proxy Manager to create secure reverse proxies so some services are available to the internet for remote use
• Photoprism for all my photo / personal video management
• Plex to serve my media
• Dashy a dashboard for all your services? I still need to learn and play with this one
• Firefly III a personal finance tool, hard to configure but seems super useful once everything is set up and imported. still learning this one too
• Minecraft Server

I definitely want to add to this list. I want to get some stuff behind a VPN and sail the open seas to fill out my plex server, I just want to make sure I am being as safe as possible.

I have been on this path (10+ years in)

• windows server
• FreeNAS > TrueNAS
• Proxmox
• Bare metal Debian with all Docker containers

I learned the most with Proxmox (and ZFS), everything in LXCs.

But I enjoy the pure Debian install with Docker containers the most!

Ps: do not, I repeat, do not install a desktop OS. Pps: stay away from Ubuntu ;)

If you’re super new to all of this, check out Yunohost. You can either run it on Debian 11 or use their own OS image. Its a simple UI interface and makes apps deployable with one click. Doesn’t support everything but I have this running beside my docker just because it’s so dang easy to use and maintain.

Linux for sure. Whatever distro seems cool to you.