What did they use in Old English or Middle English before 'of course'?

I suspect it was either

1 "gewis" It is not a precise translation but it carries much the same meaning and was relatively common. (Compare the German "Gewiss" See https://dict.leo.org/german-english/gewiss - certainly; of course; sure).

OED:

archaic. A. adj. (gewis)

Certain (subjectively and objectively). Only in Old English.

a1000 Prose Life Guthlac (1848) v. 30 We syndon gewisse þines lifes.

c1000 Gosp. Nicod. iii Myd gewyssum gesceade yrn & clypa..þone

The Middle English is "iwis" (and various other spellings)

2 A good alternative is soþlice

Bosworth Toller: http://bosworth.ff.cuni.cz/028359

-sóþlíce

I. as adv. Truly, really, certainly, verily "Ðú bist sóþlíce ǽr þrím dagum genumen of ðínum líchoman" = certainly before three days thou wilt be taken from the body,(a) Truly, in truth; actually, really; assuredly, certainly; indeed, in fact;

The Middle English is sothli.

(Compare "forsooth")

I like this old term:

certes
adverb archaic
: in truth : CERTAINLY
(Merriam-Webster)

This is commonly found in Shakespeare, for example.