Can I mechanically modify PCIe connectors to accept wider cards?
Yes, you can.
Any PCI-e card should work in any socket (though at the lower common number of lanes). So an x8 card in a x4 slot will use up to 4 lanes.
My PSU is able to source enough power to the card,
You have a PCI-e slot. That is supposed to deliver up to 75 Watt of power via the motherboard. This holds for both x1, x4, x8 and x16 connectors. So assuming a proper motherboard it should just work.
Practical experience/confirmation:
A friend worked for a fruit sorting company (Firewire cards in the x16 slots and a graphics card in an filed open x1 slot, only used to debug). That worked fine.
I just successfully cut open the end of the PCIe x4 slot in my Asus M4A87TD-USB3 motherboard and put in Adaptec RAID 6405 which is intended for PCIe x8 slots. It appears to work just fine. I also had to move the battery socket because it was interfering with the overhanging connector of the card.
This procedure is tricky and you can easily destroy your hardware.
Well, seeing the PCI-e pinout and knowing the working principles of multi-lane combinational serial busses there's two angles to look at this:
- From the Motherboard's perspective
- From the card's perspective
So, 1.: Can you take out the edge on the side where the case has the cut-out for the card/connectors? Yes, if as you say you have tools and handiness to do so, you can. Is it smart? No. The connector is very finely defined to hold the PCB/ribs in place to avoid contact lanes from moving and causing frictional bounce. If you take one of the four edges away, you lower that tightness by a severe margin.
The result can be that if you touch a cable with a hand or foot and the card wiggles a bit. Now the contacts can wiggle right along. Any dirt or courseness of the surface (the latter is unlikely with well-treated, newly bought cards because of the production process and low tolerances, but it cannot be discounted) the spring-loaded contacts in the receptacle can bounce off of those bumps and your system will miss vital parts of the data stream and all other signals will lose their relevance as well, meaning severe glitches.
IF you can take that margin out by further engineering/reinforcement (heavy task without the proper simulation tools) in principle the motherboard will not mind, since it has the lower number of connections and just assumes to find 1x, 2x or 4x on that bus.
Then, 2: Should you put a x16 card in a x4 connector? I cannot say for sure, I am not deep enough into the restrictions and specifications for those balances. It could well be that the standard prescribes the MB to have a x4 connector if it only offers x4. In that case a x16 card with a x16 connector can safely assume that if it fits in the MB it will get signal on all 16 lanes.
Since those lanes pump information simultaneously that gets correlated and decoded through all kinds of magical embedded awesomeness, they may then hard-code the 16 lane path into the card (Firmware or hardware wise), in which case it cannot cope with a 4 lane signal and might do all kinds of weird shit.
It is also possible that cards with 16 lanes are not allowed to make that assumption, and should at least support x8, x4 and maybe x2. In this case it should work from an electrical engineering standpoint, since the mandatory power is on the x4 side of the PCIe connector. But then still, it would be better if it had at least all the ground contact along its entire x16 plug. Some parts of the card's high-speed, high-performance design may depend on all those ground contacts being present, so it may still not be able to do all it could otherwise do on a x4 connection in a proper x16 plug. And those limitations, if they arise, will not be detected by the card itself, because it will definitely assume all the contacts to be there if it counts on those ground contacts in its design. Not to say that some parts will not be grounded, but in simulating noise performance you need to make assumptions about where ground currents will go, and then after those simulations taking away half those paths completely invalidates the results. It may turn out to be no difference, but it may also turn out to be a world of it.
With regards to the PCI-e standards prescribing card behaviour, I need to be back to work now, so I can't search through them, but here's two quite relevant ones, if you want to yourself:
PCI-e Spec V2.0, rev 0.9 - 2006
PCI-e Spec V3.0, rev?? - 2010
If neither of those says anything about card behaviour for x16 types, or implies it through "... for x4 cards, and so on" type constructs, it's not defined strictly and you should assume the worst until you find a reliable spec saying different.
Concluding:
There is a chance it may work, but on balance there are so many risks to trying it, where some of them will afterwards also apply to any other card with an actual x4 connection (the side-missing = risky-contact one, for example), that I would strongly advise you not to try it.