Toronto-based Brookfield Asset Management won the auction for $110 million but backed out of the deal when it couldn't resolve a dispute over debt from the casino's costly power plant.

Straub says in the filing that he believes the auction was unfair because his side wasn't fully notified of the other bidder. He said he wouldn't have bid more than his original $90 million "stalking horse" bid that set the floor price for the auction if he knew about what he described as "significant and numerous improprieties" in the bidding process.

He also wants a federal judge to deduct a $3 million breakup fee he was originally promised because Revel is keeping Brookfield's $11 million deposit.

Watch More: An exclusive look at Sydney's luxurious addition

The filing said Revel should be put up for auction again and he should get his deposit back if the judge finds that "the auction process was deficient to the extent the auction significantly and adversely affected" Revel's ability to receive the highest bid.

The judge is scheduled to consider Revel's plans to sell the casino to Straub on Jan. 5.

Straub has proposed several uses for the former casino resort. One of them was a so-called genius academy at which the world's best minds would look for solutions to society's problems.

The $2.4 billion Revel opened in April 2012 with high hopes of helping turn around Atlantic City's struggling casino market. But it never turned a profit, filed for bankruptcy twice and closed Sept. 2.

Read MoreIs Macau's gaming industry played out?

It was one of four Atlantic City casinos to shut down this year amid growing competition in the saturated Northeastern U.S. market. A fifth, the Trump Tal Mahal, averted being closed this year after investor Carl Icahn put up $20 million to keep it running through bankruptcy.