How to rank values inside of a function?
Yes, it is possible.
In the screenshot is non-english delimeter (;) used.
=CORREL(RANK.AVG(Table1[Rank A],Table1[Rank A],1),RANK.AVG(Table1[Rank B],Table1[Rank B],1))
In older Excel versions the formula must be confirmed as an array formula (confirmed by CTRL+SHIFT+ENTER).
See Example in Excel Online.