Vibes, generally. If you see a server regularly having lots of downvotes or upvotes on things that otherwise would go the other way, it’s probably a bad sign and asking the other instance admin what’s going on should be a way to start figuring out how to handle it. As for activitypub, there isn’t a generic server to server message that could contain a VoteTotals field while updates to the standard are proposed for addition?
Ah okay, my bad. Sorry about this.