How big is the Nether in Minecraft: XBox 360 Edition?
I tested this myself once I got to the Nether. In the Nether on Minecraft-XBox, maps do not show terrain, but they do report your position's coordinates. I went from one side of the Nether to the other (it is bounded by walls of bedrock). One one side, I was at coordinate Z = +140, On the other side, I was at coordinate Z = -140.
The Nether is therefore approximately 280x280.
This corresponds to the idea that the overworld is less than 1024x1024, since the Nether-to-Overworld distance multiplier is 1:3 in Minecraft-XBox, and 3 times 280 is 840.
If the Overworld is in fact 864x864, then the Nether is actually 288x288. This would match my observation, as there are probably 2-4 layers of bedrock on each edge of the map, to ensure there are no gaps.
On the Minecraft wiki, it states that the factor is 3 instead of 8. So that would make it 341 and 1/3 so it may be around that size. It also says in that page that world size is limited to 1024x1024 like you said and since the Nether is a world it might be that big too (although with my experience in the nether, i do not believe it is that big). All in all, its a 3:1 ratio.