Blocking or rate-limiting IPv6: what size prefix

The best algorithm is to start blocking separate addresses. Then when multiple addresses are blocked in the same /64 you block the whole /64. Repeat that for bigger aggregates.

Prefixes are usually given out on nibble boundaries (multiples of 4, or one hexadecimal digit). Do you might want to scale from /64 to /60, /56, /52 and /48. A /48 is usually the largest prefix given to a single site.

Depending how careful you want to be you can skip from /64 straight to /56 and /48.

Depending on which region you are in you also might use the regional internet registry's (RIR) public database. In the RIPE database internet providers document the aggregation size they give to customers there. In that case you know exactly what size to use.