Why isn't repository priority working in yum/dnf in Centos 8?

Case sensitivity should not matter so much here, because both packages virtually provide mariadb-server capability (same case).

The major reason why DNF wants to install MariaDB from AppStream is that it has a preference for modular packages. Which means, that if specific packages (names) or virtual provides are part of a module, and there are packages with the same names available from other repositories and they are not part of the module, then DNF will always want to install packages from the modular repository.

A useful command here, to find whether a specific package is part of a module is:

sudo dnf module provides mariadb-server

This finds any modules which have mariadb-server package.

In AppStream repo, MariaDB packages are with the mariadb module.

By disabling the AppStream we put a blind fold on DNF so it doesn't see what it likes. But the better way is to teach it to like non-modular packages by adding module_hotfixes=1 to the repository of interest, e.g.:

[MariaDB]
name=The MariaDB 10.4 repository
baseurl=http://yum.mariadb.org/10.4/centos8-amd64
enabled=1
gpgcheck=1
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
priority=1
module_hotfixes=1

Note that priority config should be retained, because technically 3:10.3.17 (higher Epoch: 3) version still wins over 10.4.12 (Epoch: none, same as 0:10.4.12).