Dirty Harry movies had a pithy statement uttered by the protagonist to the antagonist, or over his dead body. It was a formula that worked so well it has been widely copied in action movies ever since. In the movie Magnum Force, Clint Eastwood says to Hal Holbrook, "A man's got to know his limitations."
English grammar is like a mighty torrent starting in the headwaters of Beowolf or something and flowing through the Norman Invasion and the British Empire's world-wide conquest to your mother's knee (or someone else) who taught you to speak.
When I got to school I was blessed because my mother spoke a grammatically correct, Midwestern American dialect of the language. Thus most of my English classes consisted of writing answers that "sounded right." This generally worked out fine except for those occasional instances of unusual usage where both "who" or "whom" sound right. Thus it wasn't until I learned Spanish and later German that I grew to appreciate the fact that English has a grammar associated with it.
writers doing it wrong. But "right" can sound stilted in some contexts. And "wrong" is obligatory in other contexts.
That's one reason why I wrote "The Aristotelian" when I did. It allowed me to jump into the deep water of proper English grammar as used by a Victorian gentleman, Mycroft Holmes. I relished the notion of using difficult parts of the language and working them out correctly like a puzzle. Normally, I have the good sense to portage around a rapids of difficult grammar, but in "The Aristotelian" and in "Steamship to Kashmir," when my fancy takes the prose there, I feel I can ride out the grammatical rapids, despite the fact that "correct" prose may sound wooden.
There are only two commandments to writing that I require without exception: Thou Shalt Be Clear, and Thou Shalt Be Interesting.
If you find your usage distracts from that, reconsider it.