"My son is completely frustrated because he was the baby," Wahlberg, 37, said during an interview Thursday on the Tonight Show with Jay Leno. "He literally said, 'The baby has to go to Florida!' "
"That's the only full sentence he's said," Wahlberg added.
Big sister Ella, 5, had the opposite reaction. "My daughter's thrilled," Wahlberg said. "She thinks she has a little baby – a little plaything."
As if having a new baby isn't hectic enough, the Max Payne star told Leno that he and longtime girlfriend Rhea Durham, 30, are looking to move soon. Oh, and possibly tie the knot.
"In the middle of moving, [we're] hoping to get married all in the stretch of six or nine months," he said, joking with the talk show host about Durham. "We're getting to know each other."
Marriage MindedWahlberg has had marriage to Durham on his mind. In July, he told PEOPLE, "We continue to grow in our relationship. I think we have a much better chance at succeeding and staying together," and he echoed that sentiment on the Tonight Show.
"Both of our parents are divorced," Wahlberg said. "So we're trying to make it work. We decided we wanted to get married. Then the second baby was coming – then the third. We hardly had time."
Another thing the actor will have to consider: How to talk to his children about his risqué early career moves, which included those now-iconic Calvin Klein underwear ads and his role in Boogie Nights.
"I have a lot of explaining to do," he quipped. "I'm trying to figure out how to destroy the evidence."
How did your kids react to the new baby? Join the discussion at PEOPLE's Celebrity Baby Blog