Stern also had a warning for Smith's mom, Virgie Arthur, who flew to the Bahamas Friday to seek protective custody of the baby. "Anna despised that woman. As long as I have breath in my body, that woman will not see Dannielynn," Stern told ET's Mark Steines.
Stern reportedly picked up Dannielynn and hugged her, saying, "This is the only piece of Anna I have left." Prior to Stern's arrival, Dannielynn was at the Bahamas home Smith and Stern shared together, and was being cared for by Stern's mother and sister.
Following Smith's death on Thursday, the paternity battle over Dannielynn has heated up: California photographer Larry Birkhead, who has long claimed to be the father of Smith's baby, was denied custody of Dannielynn on Friday, while – in a strange twist – Zsa Zsa Gabor's husband also came forward Friday to suggest that he may be the father of the baby.
In Birkhead's case, a judge refused to order an emergency DNA test on Smith's corpse to help determine paternity, but asked that her body be preserved until Feb. 20, when another hearing will be held.
The Post reports that Birkhead has now teamed up with Smith's mother to try to gain custody of Dannielynn. "Everybody is in agreement that that child must be removed from Howard K. Stern's environment and everybody is going to do whatever they can to get that done," Birkhead's attorney Debra Opri told the paper.
Stern, however, is defiant: "She has no right. She can't take my baby. She can't take Anna's baby."
Visit PEOPLE.com for continued coverage of Anna Nicole Smith as the tragic story continues to develop.