You could make the saame argument against commenting, say the wrong thing and you get harassed. But what do you do with someone harassing you? You block them. What do you do when someone is harassing you by downvoting you? You most likely suck it up because you don’t know who is downvoting you.
I would argue it’s more a tool against harassment than a tool for harassment, especially since anyone determined enough can already access that information.
It would be pointless to do. Anyone can view your votes without notifying you. Just set up your own instance, download the data (that you need to do anyway because of how activitypub works) and then just open up the database with a different software to access the data. No notification can be sent because the application doesn’t know the data was accessed.
This opens a door to vote manipulation. If you can’t verify users someone can send random hashes.
The votes still exist in the activitypub. They’re already publicly available, the question is how accessible they should be because right now if you want to track downvotes you need to put in some effort. Upvotes you can already easily check from any mbin instance