OSX Software to use for Photo Collage in a large poster size

Vector graphics won't help for this kind of task. All your images are in raster format.
Your limit will really be the number of pixels [WxH] rather than 'print size', though you will need at least 150 dpi to print this at a large poster size.

Gimp can apparently save images up to 45,000 x 45,000 pixels [though I can't find citation for this] which should be just inside your range - though bear in mind this will be a colossal size in RAM.

The clarity of your output image is also very dependant on you input sizes. If you need to enlarge the photos to fit this size you will lose definition.

Alternatively, if your existing package will let you export at 300dpi or higher, then you can just ask your print service to re-scale.