Thursday night, the musician, 19, was seen with the 33-year-old Kimberly Stewart, along with her father, Rod Stewart, Rod's wife Penny Lancaster, Sean Stewart and Adrienne Maloof for a triple dinner date at Dan Tana's restaurant in West Hollywood.
"Age doesn't really matter in the Stewart family," a source tells PEOPLE, noting that Rod is 68 to Lancaster's 42 and Kimberly's 32-year-old brother Sean has been dating the 52-year-old Maloof.
"Kim and Harry looked like they were enjoying each other's company," the source also says.
While the source adds that the two are dating, another source says Styles is simply enjoying his time off.
"Harry's not dating Kim," the source says. "He was flirting with a different blonde at the SoHo House on Friday night."
Kimberly is mom to 20-month-old daughter Delilah with Benecio del Toro.
Most recently, Styles was linked to Taylor Swift. A rep for Styles had no comment.
With reporting by JENNIFER GARCIA, RAHA LEWIS, MARY MARGARET and KEVIN O'DONNELL