"Police say there appear to be signs of a break-in." Why "appear" instead of "appears"?

No, linking verbs are different from modals in English grammar, though there may very well be an historical connection. The modals in current English have very special grammatical properties, one of which is inversion in questions: "May/Should/Will/.. I show you my portrait?" But you can't invert linking verbs.

Your example has been a puzzle for grammarians,

Police say there appear to be signs of a break-in.

because "appear" seems to agree in number with "signs", even though "signs" can't possibly count as subject of "appear". An artificial solution to this is to suppose that "there" can have plural number, even though it has no plural ending, and in

There are signs of a break-in.

the "there" is the subject and is plural, with "are" agreeing with plural "there". "Appear" takes a sentence complement as subject, and it undergoes a transformation Subject-Raising-to-Subject so that the plural "there" becomes the subject of "appear" which then agrees in number with this plural subject. That's why "appear" shows plural agreement. (I don't think anyone really likes this analysis.)