ORLANDO, Fla. -- Con-man Lou Pearlman is headed to federal prison. A federal judge won't let him stay at the Orange County jail any longer.

---

VIDEO: Victims Hope To Get Money Back From Lou Pearlman
READ: Pearlman Case 47-Page Plea Agreement (PDF file)
ARCHIVE: Previous Reports Involving Lou Pearlman

---

Pearlman was in federal court Wednesday morning, because prosecutors were trying to convince the judge to force the former boy-band mogul into paying $125 million in interest to the investors he scammed.

Pearlman has had a year, since federal investigators brought him back to the United States from Bali, to lead them to millions of dollars he might have stashed somewhere so that his victims can get at least some of their money back. In federal court Wednesday, prosecutors told the judge Pearlman has yet to produce a penny of it.

"We're focused on tracing assets, money that has been stolen from the victim, and we're still working on that," said federal prosecutor Roger Handberg.

The federal judge put off ruling on whether he would order Pearlman to pay another $125 million in interest in addition to the $300 million he owes investors and banks. The judge's concern was that the total, $425 million, would be impossible for Pearlman to pay back and so he'd be less inclined to pay any of it back to the victims.

Investor James Taylor agrees.

"I think he done the right thing. If he had assessed all that interest, it would make it a lot harder for Pearlman to get out of this mess," Taylor said.

In court Wednesday, Pearlman's defense attorney called him "indigent," which sparked a couple of grunts from fraud victims.

"I don't believe that. They said he had $5 million when he left in January. Where's the $5 million?" questioned fraud victim Margaret Smith.

The bankruptcy trustee had some good news Wednesday that he's suing two men who did business with Pearlman, expecting to recoup about $5 million for the conman's victims.

Pearlman, who is 54 years old, is serving a 25-year federal prison sentence. Federal Judge G. Kendall Sharp said he'd cut that by one month for every million Pearlman pays back.