Possible GCC bug when returning struct from a function

I don't see any UB here; your types are unsigned so signed-overflow UB is impossible, and there's nothing weird. (And even if signed, it would have to produce correct outputs for inputs that don't cause overflow UB, like rdi=1). It's broken with GCC's C++ front-end as well.

Also, GCC8.2 compiles it correctly for AArch64 and RISC-V (to an madd instruction after using movk to construct constants, or RISC-V mul and add after loading the constants). If it was UB that GCC was finding, we'd generally expect it to find it and break your code for other ISAs as well, at least ones that have similar type widths and register widths.

Clang also compiles it correctly.

This appears to be a regression from GCC 5 to 6; GCC5.4 compiles is correctly, 6.1 and later don't. (Godbolt).

You can report this on GCC's bugzilla using the MCVE from your question.

It really looks like it's a bug in x86-64 System V struct-return handling, perhaps of structs containing padding. That would explain why it works when inlining, and when widening a to uint64_t (avoiding padding).

This has been fixed on trunk/master.

Here is the relevant commit.

And this is a patch to fix the issue.

Based on a comment in the patch, the reload_combine_recognize_pattern function was trying to adjust USE insns.

Does this code contain any undefined behaviour that would allow GCC to emit the "incorrect" assembly?

The behavior of the code presented in the question is well defined with respect to the C99 and later C language standards. In particular, C permits functions to return structure values without restriction.