Is it incorrect to use "hard" when I mean "difficult"?

There is nothing incorrect about hard being used that way. It's just more colloquial than difficult, but other than that, they are synonyms, and have been for a long time. The Oxford English Dictionary has citations going back all the way to 1340:

• a1340 HAMPOLE Psalter vi. 4 Ful hard it is to be turnyd enterly til þe bryghthed and þe pees of godis lyght.
• c1440 Promp. Parv. 227/1 Harde yn knowynge, or warkynge, difficilis.
• 1559 W. CUNINGHAM Cosmogr. Glasse 97 It is as harde, and laborus, to get the Longitude.
• 1611 BIBLE Transl. Pref. 2 So hard a thing it is to please all.
• ...

I actually don't have access to the OED — where I'm quoting this from is this excellent Language Log entry. Highly recommended reading.

To that, I think it is worth adding that difficult is only some 600-odd years old. It is a back formation from difficulty, which in turn was borrowed from Old French in the late 14th century. Hard, on the other hand, has been around for much longer and came straight from Old English. So it's actually the more venerable way of saying "difficult".

You have this in many languages. I can think of at least two of them.

• In German "es ist schwer" means both "it's heavy" and "it's difficult".
• In French "c'est dur" means both it's "hard" and "it's difficult".

I guess you could conclude the Germans used to carry the stones, and the French used to carve them.

And all were complaining anyway !