If you average out your knowledge in different fields of expertise you’ll realize you’re just as dumb as they are. This brings something that isn’t peace of mind, but does look very much like it. Though if your biggest problem is “people not understanding you”, I suggest working on your communication skills.
For management ports, I set up a firewall on the VPS to only respond to connections from known IPs.