Difference between "function" and "operation"

Solution 1:

Operation is best. The function of a school would mean basically the purpose of the school, which cannot be "smooth". Operation would mean how things go (how the function is accomplished), so the operation of the school is what can and should go "smoothly".

Also see PLL's comment below: functioning instead of function would easily suffice as a substitute for operation in most cases. So your choices are between "operation" and "functioning" here.

Solution 2:

The two words do have distinct meaning, and in this case, operation would be the correct word to use in that context.

The word function describes the intent and purpose of a thing. In the case of a school, the school's "function" is to serve as a place of teaching by the resident staff for the edification of and learning by the resident students. A purpose and intent lacks any real connotation of how well it might actually operate, it is a definition.

The word operation describes the performance and implementation of the function of a thing. In the case of a school, the school's "operation" is the act of teaching the students by the staff. A smoothly operating school performs its function well...a poorly operating school performs its function badly.