CMake: In which order are files parsed (cache, toolchain, etc.)?

cmake

This seems as a trivial question, since CMake is a script language the general answer is: strictly sequential. But I came across several cases where it was important when or in which order CMake is parsing certain files. So I wonder:

Is there a documentation available that describes the order in which files (incl. internal CMake files) are parsed?
Is the file order depending on the CMake version or some CMake options/settings/environment incl. the chosen generator or host environment?

The cases I came across so far, where the above information was important:

The toolchain file is parsed before the compiler is identified, so you have to populate certain CMake variables in the cache first/in the toolchain file: CMake cross-compile with specific linker doesn't pass arguments to armlink
The toolchain file is parsed multiple times, therefore e.g. printing messages from the toolchain file show multiple times: CMake toolchain includes multiple files
Variable watch can be called from a scope outside your main CMakeLists.txt file has been parsed: Execute command or macro in CMake as the last step before the 'configure' step finishes

Maybe you know even more.

To find an answer, I have tried the following: I have setup a simple main CMakeLists.txt as shown below and run cmake --trace … to analyze the parsing order.

cmake_minimum_required(VERSION 2.8)

include(BeforeProjectCmd.cmake)

project(ParserTest CXX)

add_subdirectory(LibTarget1)
add_subdirectory(LibTarget2)

add_executable(ExeTarget Test.cpp)

variable_watch(CMAKE_BACKWARDS_COMPATIBILITY)

When I then run e.g. cmake --debug-output --trace -G"Visual Studio 12 2013" -DCMAKE_TOOLCHAIN_FILE:FILE_PATH=Toolchain.txt I got a long trace that I tried to summarize:

# Begin try to read
CMakeCache.txt
${CMAKE_BINARY_DIR}/CMakeCache.txt
PreLoad.cmake
${CMAKE_BINARY_DIR}/PreLoad.cmake
# End try to read

┌ CMakeLists.txt(1):  cmake_minimum_required(VERSION 2.8 )
│ CMakeLists.txt(3):  include(BeforeProjectCmd.cmake )
│
├─ BeforeProjectCmd.cmake
│
│ CMakeLists.txt(5):  project(ParserTest CXX )
├┬ share/cmake-3.2/Modules/CMakeDetermineSystem.cmake
││
│└─ Toolchain.txt
│
├┬ ${CMAKE_PLATFORM_INFO_DIR}/CMakeSystem.cmake
││
│└─ Toolchain.txt
│
├─ share/cmake-3.2/Modules/CMakeSystemSpecificInitialize.cmake
├┬ share/cmake-3.2/Modules/CMakeDetermineCXXCompiler.cmake
│├┬ share/cmake-3.2/Modules/CMakeDetermineCompiler.cmake
││├ share/cmake-3.2/Modules/Platform/Windows-CXX.cmake
…
││├ share/cmake-3.2/Modules/CMakeDetermineCompilerId.cmake
││├─ share/cmake-3.2/Modules/CMakeCompilerIdDetection.cmake
…
││├ share/cmake-3.2/Modules/Compiler/MSVC-DetermineCompiler.cmake
…
│├ ${CMAKE_BINARY_DIR}/${CMAKE_FILES_DIRECTORY}/3.2.2/CMakeCXXCompiler.cmake
│├ share/cmake-3.2/Modules/CMakeSystemSpecificInformation.cmake
│├┬ share/cmake-3.2/Modules/CMakeGenericSystem.cmake
││├ share/cmake-3.2/Modules/Platform/Windows.cmake
││└─ share/cmake-3.2/Modules/Platform/WindowsPaths.cmake
│├ share/cmake-3.2/Modules/CMakeCXXInformation.cmake
│├┬ share/cmake-3.2/Modules/Compiler/MSVC-CXX.cmake
││├ share/cmake-3.2/Modules/Platform/Windows-MSVC-CXX.cmake
││├┬ share/cmake-3.2/Modules/Platform/Windows-MSVC.cmake
│││└─ share/cmake-3.2/Modules/CMakeRCInformation.cmake
││└ share/cmake-3.2/Modules/CMakeCommonLanguageInclude.cmake
│├ share/cmake-3.2/Modules/CMakeTestCXXCompiler.cmake
│├┬ share/cmake-3.2/Modules/CMakeTestCompilerCommon.cmake
││├ share/cmake-3.2/Modules/CMakeDetermineCompilerABI.cmake
││├ share/cmake-3.2/Modules/CMakeDetermineCompileFeatures.cmake
││├ share/cmake-3.2/Modules/Internal/FeatureTesting.cmake
││└ share/cmake-3.2/Modules/Compiler/MSVC-CXX-FeatureTests.cmake
│└ ${CMAKE_BINARY_DIR}/${CMAKE_FILES_DIRECTORY}/3.2.2/CMakeCXXCompiler.cmake
│
│ CMakeLists.txt(7):  add_subdirectory(LibTarget1 )
│
├─ LibTarget1/CMakeLists.txt
│
│ CMakeLists.txt(8):  add_subdirectory(LibTarget2 )
│
├─ LibTarget2/CMakeLists.txt
│
│ CMakeLists.txt(10):  add_executable(ExeTarget Test.cpp )
│ CMakeLists.txt(12):  variable_watch(CMAKE_BACKWARDS_COMPATIBILITY )
│
│  CMake Debug Log in CMakeLists.txt:
│  Variable "CMAKE_BACKWARDS_COMPATIBILITY" was accessed using UNKNOWN_READ_ACCESS with value "".

-- Configuring done
-- Generating ${CMAKE_BINARY_DIR}
-- Generating ${CMAKE_BINARY_DIR}/LibTarget1
-- Generating ${CMAKE_BINARY_DIR}/LibTarget2
-- Generating done

# Writes
${CMAKE_BINARY_DIR}/CMakeCache.txt

So seeing the above output I came - so far - to following conclusion (which I hope are true and somewhat generic):

The CMakeCache.txt file is only read once when configuration is started and written after the generation is finished. It just persists the state of the "global variables" cache.
The project() command trigger most of CMake's detection magic (including reading from the Toolchain.txt file).
The toolchain file is read twice. Once before the make/compile system is detected and once inside the then generated CMakeSystem.cmake.
The variable_watch() hook can trigger anytime, so the scope in which the optimal "command to execute" is called is undefined.

Solution 1:

There's no official documentation about this particular inner workings of CMake, so please find below a summary of what I've learned about CMake so far ...

What files are parsed depends on the

The host and target operating system
The target compiler
Your host computer's environment (variables, registry, installed software)
Your project's CMake script files, which could include
1. Your toolchain file
2. Your selected programming languages
3. Any external projects/libraries/files/scripts

There are a lot of possible combinations of those parameters, but most of the time CMake does all the magic of automatically detecting the correct settings for you and you don't need to bother how it's done. The good news is - when you need to know - it follows certain intrinsic patterns.

Interesting is that it only marginally depends on the CMake generator you are selecting.

Initial Step: Compiler Detection and Verification

This mainly starts with the project() command. Taking CXX language as an example, the main files for compiler detection are (see also the root files in the question's trace output):

share/cmake-x.y/Modules/CMakeDetermineCXXCompiler.cmake

This basically tries to determine the compiler executable's location and does call it to get a more specific compiler id.

Furthermore it e.g. defines source/output file extensions based on the host computer environment and target operating system.
share/cmake-x.y/Modules/CMakeCXXCompiler.cmake.in

This is the template to store the result of the compiler detection in ${CMAKE_BINARY_DIR}/${CMAKE_FILES_DIRECTORY}/x.y.z/CMakeCXXCompiler.cmake.

Mainly those variables are: CMAKE_CXX_COMPILER, CMAKE_CXX_SOURCE_FILE_EXTENSIONS, CMAKE_CXX_IGNORE_EXTENSIONS and CMAKE_CXX_COMPILER_ENV_VAR

share/cmake-x.y/Modules/CMakeCXXInformation.cmake

This file sets the basic flags for the compiler. It's also where the compiler, host and target does have the most influence on the setup with calls like this:

include(Platform/${CMAKE_SYSTEM_NAME}-${CMAKE_CXX_COMPILER_ID}-CXX-${CMAKE_SYSTEM_PROCESSOR} OPTIONAL)
include(Platform/${CMAKE_SYSTEM_NAME}-${CMAKE_CXX_COMPILER_ID}-CXX OPTIONAL)
include(Platform/${CMAKE_SYSTEM_NAME}-${CMAKE_BASE_NAME} OPTIONAL)
include(Platform/${CMAKE_SYSTEM_NAME} OPTIONAL)

share/cmake-x.y/Modules/CMakeTestCXXCompiler.cmake

This does test everything and e.g. determine compiler features by actually calling the compiler in a simple generated CMake projects.

The results of those steps are stored in cached variables and those files are special in such case the they are safeguarded by variables like CMAKE_CXX_COMPILER_LOADED, CMAKE_CXX_INFORMATION_LOADED or CMAKE_CXX_COMPILER_WORKS to not run with each consecutive CMake configuration step again.

Project Configuration Files: Modify the Defaults

There are several ways you could change a CMake default values without actually having to touch your project's CMakeLists.txt files.

-C <initial-cache> command line option

This can be used if you want to give some preset values (you would normally give via -D ... option) trough several projects over and over again. Like some library search paths on your computer or some presets used in your company.
CMakeCache.txt through e.g. cmake-gui

cmake-gui lets you manually modify your project's options (editing all non-internal variables in CMakeCache.txt) before you finally generate the build environment.
CMAKE_TOOLCHAIN_FILE

Mainly used for cross-compiling, but it can more generally describes as preset values per compiler toolchain used.
PreLoad.cmake

More or less the same as the "initial cache" option (see above), but it's not given through a command line option. It has just to be in the same directory as your project's CMakeLists.txt.

Note: It supports all CMake script commands like if() calls, but PreLoad.cmake has its
- own variable scope (everything non-cached here is not visible in your main CMakeLists.txt)
- limitations what is already known (it runs before everything else, so mostly you can check against CMAKE_GENERATOR)
CMAKE_USER_MAKE_RULES_OVERRIDE, CMAKE_USER_MAKE_RULES_OVERRIDE_<LANG>

This allows to modify non-cached default values after the automatic detection by CMake.

Example: Extending the valid CXX source file extensions by .c files

MakeRulesOverwrite.cmake
```
list(APPEND CMAKE_CXX_SOURCE_FILE_EXTENSIONS c)
```
Then you can call cmake with something like
```
> cmake -D CMAKE_USER_MAKE_RULES_OVERRIDE:PATH=..\MakeRulesOverwrite.cmake ..
```
CMAKE_PROJECT_ParserTest_INCLUDE

This is meant to "inject custom code into project builds without modifying their source" directly after your project() command was processed (and the build environment was detected).

Toolchain.cmake: Parsed Multiple Times

A toolchain file is read multiple time while determining the system, compiler, etc.

Important to know is:

It's read with each try_compile() call. And since try compile must produce a valid executable, you may need - if you are e.g. cross-compiling - to
- CMAKE_TRY_COMPILE_TARGET_TYPE to STATIC_LIBRARY (CMake version 3.6 or above)
- Check IN_TRY_COMPILE global property to add additional options
If you change your toolchain file, CMake will re-trigger the compiler detection (as in the trace above). Which profoundly helps to play with your compiler settings.

CMake Re-Configurations: Everything comes from the Cache

Last but not least, it's important to know that the trace above only shows the initial step. All consecutive project configurations will take almost everything from cached variables and therefore will read much less files in the re-configuration runs.

References

The Architecture of Open Source Applications: CMake
Generic rule from makefile to CMake
CMake error at CMakeLists.txt:30 (project): No CMAKE_C_COMPILER could be found