Can you say "accurate to"?

You can definitely say "accurate to" in some cases: for instance, "The stopwatch is accurate to a thousandth of a second" would be entirely correct, and many dictionaries specifically list this use of the word accurate.

The question, then, is whether something can be accurate not only to an amount or measure (be those seconds, meters, or grams) or to n significant digits or n decimal places, but also to a thing.

Personally, I think your example sentence is correct. Other phrases I've found in the corpus include:

"a painting that seems accurate to life"

"in settings accurate to the period and locale"

"the movie was accurate to the Bible or attempted to be"

Other ways of saying your sentence would include:

"The building was built according to the schematics"

"The building adheres to/conforms to the schematics"

...but in both cases, I'd add "precisely" to make it obvious just how accurate the link between the building and the schematics is.