Ben Flajnik: Why I Drove Kris Jenner's Bentley (Not What You May Think)
11/26/2013 AT 08:00 AM EST
"I happened to be in L.A. and I didn't have a car, so she let me borrow hers. She wasn't even there. Her assistant opened up the garage and just said, 'Pick one,' " Flajnik tells PEOPLE about recently being photographed leaving Jenner's Calabasas home behind the wheel of her car.
The photos – in addition to those of Jenner and Flajnik together at Flajnik's winery and on a vacation in Mexico – have fueled another round of reports that the tech mogul and momager are secretly having an affair, but Flanik says his relationship with Jenner is strictly professional.
"We have some really great things in the works, and this is frustrating because it's not the light I wanted shed on our projects," says Flajnik, who adds that his time with Jenner in Mexico was spent with mutual friend Joe Francis. "It is like you can't do anything without people thinking you're hooking up or in a relationship."
It turns out the dating rumors are damaging to the currently single Flajnik's personal life, as well.
"It comes up a lot," Flajnik says of dates who bring up the rumors. "I can handle it one person at a time, but when people write articles it is more difficult."
Not that Flajnik thinks Jenner isn't a catch, he is quick to point out.
"She's a beautiful, smart woman," he says. "But we're friends. We talked this morning and she said, 'Hey, don't worry.' She understands why I'm upset and uncomfortable with [the reports.] She deals with this kind of thing on a daily basis, and I've worked really hard to stay away from this kind of image."
Reporting by MONICA RIZZO