In theater, TV and/or movies there's a term, "the fourth wall" referring to the space that separates the performer from the audience. An actor should address his or her remarks to the other players on stage and not address the audience directly.
This is one of those rules that was largely observed until sometime in the late '60s or early '70s when it was broken with great comedic effect. You just don't do that. But they did and that made the comedy work.
There's a good reason why you don't break down the fourth wall. And that's got to do with the fictive dream. When you start reading a story about a fella who leaps tall buildings in a single bound, is bullet-proof, and has x-ray vision, you go along with it even if he's wearing blue spandex. You suspend disbelief because you're in the story. You can believe for a while that all this is because he's from Krypton where there is a red sun and heavier gravity and stuff.
People go along and build a mental image of this fictional world even when nobody's wearing spandex and there's just a little old lady is sipping tea in Saint Mary Mead while people around her are dropping like flies.
The story has an internal consistency that defines the world of the writer's construction. The world's consistency keeps the reader in the story. In this world, the reader should just be a wiretap on a POV character's sensory inputs with just the POV character's interpretations to guide the reader's perception.
I can get peeved when something breaks that pattern and takes me out of the story. As a writer this is a very bad thing. Because when the reader gets out of the story, s/he may realize the hour is getting late and turn out the light. Or there's something better on TV.
The writer wants the reader to be oblivious to the world outside the story.
Can you break this rule? Sure. Look at the scenes with Fred Savage and Peter Falk in The Princess Bride. They work to bracket the story of Wesley and Princess Buttercup, and when the story might get bogged down, the kid and the grandfather are there to provide color commentary. Framing is a good thing, and these little asides are also a good thing to let the reader settle into a story and reflect upon it. Technically, this is not really breaking the fourth wall as much as a Greek Chorus,
There are subtler ways to take the reader out of the story that I regard as mere ineptitude. Things like, the "Little did I suspect" annoyance I've talked about before.
The worst way to take the reader out of the story is to make a character do something completely out of character. Characters have to act according to human nature, and they have to be consistent with the sort of person you've established already in the novel. Someone who's a nervous nellie thru the first two thirds of the story can't become laid back in the third act. If you've got a tough biker dude in a leather jacket, you can't put him on water skis and make him jump a shark.
If you write science fiction you can have aliens and ray guns, but you can't drop a tentacled body in the library of a manor in Saint Mary Mead with a hot gat in Miss Marple's fist and having her beat the truth out of a reluctant witness.
I'm cool with putting a sly reference to something the reader might recognize to another work, but it has to be done with a straight face so that anyone who doesn't recognize the allusion will notice nothing strange. But this is risky. You aren't as clever as you wish you were. Nobody is.
My advice is to be consistent. Think through how your story-world works. Write some scaffolding if you aren't sure whether the rest of the story-world has room in it for a hard-boiled spinster detective.