On average every 1.3rd or 1.3th something?
I would simply avoid the use of ordinals that are not whole numbers. They make no sense, and they can easily be avoided.
If there is a line of people, I cannot say that I pick every 1.5th person.
Every suggests you actually identify an individual occurrence of something, which can never be the 1.1st occurrence.
Change the phrase to something like
10 out of 11 (for 1.1)
10 out of 13 (for 1.3)
2 out of 3 (for 1.5)
Googling for "0.3th quantile" and "0.3rd quantile" reveals that both options are used in scientific literature.