Are the walls slowing down minecarts?

If your mine cart track is in a straight line, then having walls on either side should not slow it down. If your track is running diagonally, or if it turns a corner with the walls tightly around it, then the cart may experience some "dragging"; That is, the cart will slow down as it encounters the wall.

You could try widening the tunnel where the cart goes diagonally, or where the track bends and see if that removes the problem.

If by "between two walls", you mean having walls on both sides of the track, then no, the walls don't affect the cart's speed. Carts lose momentum very quickly with no rider, so that may be why your carts are slowing down fast. Or perhaps your boosters aren't setup properly.

I experienced a slow-down depending on the PC I'm playing on - boosters that work perfectly fine on my good machine totally fail on the slower one. Maybe this is also the case for you, try lowering the viewing distance and set the rendering to fast for comparison.