How to detect if the application being used by a user is mod apk or not

I have an Android application on Play Store and I want to detect if a user is using the original version of the app downloaded from play store or a mod apk from other sources.

Your Question

I have an Android application on Play Store and I want to detect if a user is using the original version of the app downloaded from play store or a mod apk from other sources.

A very simple question that embraces so many things to be fully answered, but from an high level view you can tackle this from within you mobile app or doing it from the outside.

Detecting from Within the Mobile app

Doing it from inside is known as RASP:

Runtime application self-protection (RASP) is a security technology that uses runtime instrumentation to detect and block computer attacks by taking advantage of information from inside the running software.

RASP technology is said to improve the security of software by monitoring its inputs, and blocking those that could allow attacks, while protecting the runtime environment from unwanted changes and tampering.

One thing that I observe often is that many developers are not aware that any protection they may add to the mobile app code to try to secure it can can be bypassed during runtime with an instrumentation framework, even when the code itself is strongly obfuscated. A well known instrumentation framework used to manipulate code at runtime is Frida:

Inject your own scripts into black box processes. Hook any function, spy on crypto APIs or trace private application code, no source code needed. Edit, hit save, and instantly see the results. All without compilation steps or program restarts.

In the case a developer adds a function to their code to detect if the app is the original one, the attacker will eventually find it through static analysis of the binary or through dynamic analysis at runtime, and then use Frida to hook on it to change the outcome, like returning always a result that says it's the original one. Another alternative for the attacker is to recompile the binary without said function, thus removing the protection.

So, Am I saying for you to not use self protecting code or RASP solutions on your mobile app?

No, I recommend you to use all the mechanisms you can afford in order to stop the bad actors, but you also need to be aware that they can bypass them, and try to make as hard as possible to overcome your defences, to the point that it will be time consuming for them and they will just prefer to go elsewhere to look for easier targets.

Detecting from outside the Mobile App

A better alternative is to delegate to outside the app the detection when it's running or not an original version of the binary, and if it doing in a device that is not rooted or jail-broken, and this can be done by using the Mobile App attestation concept, that I explain on this answer I gave to the question How to secure an API REST for mobile app? in the section about A Possible Better Solution.

In a nutshell the Mobile App Attestion is a solution that when full implemented attests if your mobile app is the genuine and untampered version you have uploaded to the play store, and that is running in a trusted device, not jail-broken or rooted.

The Mobile App Attestation solution differs from RASP solutions in the fact that the decisions are made outside the mobile device, therefore cannot be manipulated by instrumentation frameworks, and they also issue a JWT token that allows the backend for the mobile app to know when it can trust in requests is receiving from it.

Summary

RASP solutions fall short, because usually they don't let the mobile api backend know if the request is from a genuine version of the mobile that is running in a trusted environment, aka a device not rooted or jail-broken, but even if they do that, once the logic for doing so is running inside the mobile app, it can be manipulated by the attacker with Frida or similar tools.

On the other hand a Mobile App Attestation solution will make decisions outside the mobile device and allow the mobile api backend to be aware when it can trust or not in the incoming requests.

Do You Want To Go The Extra Mile?

In any response to a security question I always like to reference the excellent work from the OWASP foundation.

For APIS

OWASP API Security Top 10

The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.

For Mobile Apps

OWASP Mobile Security Project - Top 10 risks

The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.

OWASP - Mobile Security Testing Guide:

The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.