It sounds like you are writing a memoir. To be non-fiction, all facts should be verifiable, or Oprah will hunt you down (reference "A Thousand Little Pieces"). Once you cross the line into embellishment, or tiny fictional vignettes, it's fiction. Fiction is like pregnancy, there's no such thing as a little bit fictional. It is or it isn't.

There are genres devoted to mixing fictional characters into historically accurate settings, or actual people (Adolf Hitler) in alternate histories. Kind of the truth with a twist, but still fiction.

That doesn't mean creativity isn't possible within non-fiction. How vibrantly you describe things is free territory. And remember that much of memoir is how you perceive things and how you felt.

Good luck -Craven