But as the world now knows, that changed this week when he rescued the kidnapped Amanda Berry.
As Ramsey told CNN's Anderson Cooper on AC360 Tuesday night that he "heard that girl scream and saw [a neighbor] run across the street, and I went outside and wondered what he was doing, and ... Amanda said, 'I'm stuck in here, help get me out.' "
Ramsey continued: "So here I come with my, you know, half-eaten Big Mac, and I looked and I said, 'Well, what's up?' And she's like, I've been trapped in here, he won't let me out, me and my baby. I said, 'Well, we ain't going to talk no more, come on."
Because the door was jammed, "I did what I had to do," said Ramsey, "and kicked the bottom of the door, and she crawled out of it. She grabs her baby, which threw me off, all right, so fine. I got some girl and her kid."
Cooper asked about her appearance: "I mean, what was she wearing?" Answered Ramsey, "Jumpsuit. She had a white tank top on, rings on, mascara. You know, she was well groomed. She didn't look like she was kidnapped."
"And you'd never seen her before ... in the year that you had been there?" asked Cooper.
"Never," said Ramsey. "That woman didn't come out the house. The only kids that came out the house were two little girls. They played in the backyard. ... They did the same thing, play in the backyard for a couple of hours, go back in the house. Same thing every day."
As for being a hero, Ramsey played down the label, telling Cooper: "No, no, no. Bro, I'm a Christian, an American and just like you. We bleed same blood, put our pants on the same way. It's just that you got to put that – being a coward, and I don't want to get in nobody's business. You got to put that away for a minute."
For much more on the Cleveland kidnapping case, including details about the women's harrowing escape, pick up this week's PEOPLE, on newsstands Friday
Additional reporting by ANDREA BILLUPS