Is there a way to specify assembly references based on build configuration in Visual Studio?

There is a way to do this, but you will have to hand edit your project files. The project files can have a Condition attribute applied to them in many of the elements, including the one for references.

You can add these to your references to specify when the reference should be used:

<Reference Include="Product, Version=1.0.0.0" Condition="'$(Configuration)'=='V1'">
</Reference>
<Reference Include="Product, Version=2.0.0.0" Condition="'$(Configuration)'=='V2'">
</Reference>
<Reference Include="Product, Version=3.0.0.0" Condition="'$(Configuration)'=='V3'">
</Reference>

You then define several build configurations (V1, V2, V3) and each reference will be included only in the relevant chosen build configuration.

Combine this with conditional compilation symbols and #if statements in your code and you should be able to do what you want.

A thing to be careful of if you do this is that it is easy to have Visual Studio remove the conditional attributes from the project file.

<Reference Include="log4net, Version=1.2.11.0, Culture=neutral, processorArchitecture=MSIL">
  <SpecificVersion>False</SpecificVersion>
  <HintPath>..\SharedLibs\log4net\$(Platform)\$(Configuration)\log4net.dll</HintPath>
</Reference>

You can replace the hint path with the properties:

$(Configuration) is equivalent to Release/Debug or whatever other configuration you have. $(Platform) is equivalent to x86/x64/Any CPU

If your configuration includes Any CPU then you will need to put single quotes around $(Configuration)

Also refer to the condition options referenced by adrianbanks