With the help of a computer algorithm, the state’s biggest health insurer says it will scrutinize and cut payments to doctors who frequently bill it for the most expensive patient visits.