Ben Flajnik and Kris Jenner
Kabik/Retna; David Livingston/Getty
11/25/2013 at 06:00 PM EST
's Ben Flajnik enjoys driving Kris Jenner's Bentley, but don't get the idea that they are anything more than just friends.
"I happened to be in L.A. and I didn't have a car, so she let me borrow hers. She wasn't even there. Her assistant opened up the garage and just said, 'Pick one,' " Flajnik tells PEOPLE about recently being photographed leaving Jenner's Calabasas home behind the wheel of her car.
The photos – in addition to those of Jenner and Flajnik together at Flajnik's winery
and on a vacation in Mexico – have fueled another round of reports that the tech mogul and momager are secretly having an affair, but Flanik says his relationship with Jenner is strictly professional.
"We have some really great things in the works, and this is frustrating because it's not the light I wanted shed on our projects," says Flajnik, who adds that his time with Jenner in Mexico was spent with mutual friend Joe Francis. "It is like you can't do anything without people thinking you're hooking up or in a relationship."
It turns out the dating rumors are damaging to the currently single Flajnik's personal life, as well.
"It comes up a lot," Flajnik says of dates who bring up the rumors. "I can handle it one person at a time, but when people write articles it is more difficult."
Not that Flajnik thinks Jenner isn't a catch, he is quick to point out.
"She's a beautiful, smart woman," he says. "But we're friends. We talked this morning and she said, 'Hey, don't worry.' She understands why I'm upset and uncomfortable with [the reports.] She deals with this kind of thing on a daily basis, and I've worked really hard to stay away from this kind of image
Reporting by MONICA RIZZO