Work is beginning at the Victoria Park site of Brisbane's Olympics stadium, with the state's premier saying those opposed are free to express their views, but not between "excavators and bulldozers".