Larry Birkhead Buys House in Kentucky Hometown

By Howard Breuer

06/07/2008 at 01:00 PM EDT

Larry Birkhead has purchased a 10,000-sq.-ft., six-bedroom house in his hometown of Louisville, Ky., for himself and 21-month-old daughter Dannielynn, he tells PEOPLE.

Birkhead, 35, says the house, which has 40-ft. ceilings and is modeled after a Sicilian church, was never completed, needs a lot of work, and right now "looks like a haunted house," but he loves it.

"When I walked in, I said to myself, 'This is our home!'" says Birkhead.

He adds he might sell the Los Angeles house where he's surrounded by memories of its former occupants – Dannielynn's mother, Anna Nicole Smith, and half-brother Daniel, who both died from drug-related complications.

"At first, I couldn't imagine parting with it," says Birkhead. "Then I realized I was probably holding on to it for the wrong reasons, thinking to myself that would ease the pain of losing Anna and Daniel. I have always thought that it would be great for Dannielynn to grow up in her mother's home; however, there is no yard as it is over a canyon."

He adds: "There also isn't any little friends for her to run around with – which makes the idea of Kentucky even more inviting."

Like this story? Click on the Yahoo Buzz! button below to Buzz it up!