Early turnout in a Milford casino referendum has been brisk, with more than one-third of the town’s registered voters casting ballots by mid-afternoon, according to the town clerk’s office.