Installing OS X Yosemite from compressed DMG file in VMware Fusion

It doesn't matter what type the DMG is as VMware Fusion cannot install OS X Yosemite by booting it or even booting the Install OS X Yosemite.app downloaded from Apple as that's not how it works in VMware Fusion. VMware Fusion must first make an Installer Image from the Install OS X Yosemite.app and install from the Image it creates.

Have a look at: Installing OS X 10.10 Yosemite as a guest operating system in VMware Fusion (2082109).

To install Mac OS X 10.10 in a virtual machine using the Yosemite app download:

1. Select File > New from the Fusion menu bar.
2. Select Install from disc or image and click Continue.
3. Drag and drop the Install OS X Yosemite application into Use operating system installation disc or image dialog box.
4. Click Continue.
5. Fusion should automatically discover the Guest Operating System. If not, ensure that the Operating System is set to Apple Mac OS X and that the version is set to Mac OS X 10.10. Click Continue.
6. If you want to modify any of the settings, such as memory (RAM), CPU, or hard disk size, click Customize Settings and specify the non-default values.
7. Click Finish. The installation starts.
8. When prompted, select Reinstall Mac OS X and click Continue.
9. Click Continue.
10. Agree to the license agreement and follow the prompts to begin the installation.

After the installation completes, you must install VMware Tools on the virtual machine. To do this, first eject the Mac OS installation disc image from within the new virtual machine. For more information, see Installing VMware Tools in a Mac OS Guest (1032440).

The solution from @user3439894 did not worked for me while installing OS X 10.11 El Capitan using a dmg file. I created a bootable ISO to install it on VMware Fusion 8.0.1 by following the below steps as advised here.

# Mount the installer image
hdiutil attach /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/SharedSupport/InstallESD.dmg -noverify -nobrowse -mountpoint /Volumes/install_app

# Create a ElCapitan blank ISO image of 9900mb with a Single Partition - Apple Partition Map
hdiutil create -o /tmp/ElCapitan.cdr -size 9900m -layout SPUD -fs HFS+J

# Mount the ElCapitan blank ISO image
hdiutil attach /tmp/ElCapitan.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/install_build

# Restore the Base System into the ElCapitan blank ISO image
asr restore -source /Volumes/install_app/BaseSystem.dmg -target /Volumes/install_build -noprompt -noverify -erase

# Remove Package link and replace with actual files
rm /Volumes/OS\ X\ Base\ System/System/Installation/Packages
cp -rp /Volumes/install_app/Packages /Volumes/OS\ X\ Base\ System/System/Installation/

# Copy El Capitan installer dependencies
cp -rp /Volumes/install_app/BaseSystem.chunklist /Volumes/OS\ X\ Base\ System/BaseSystem.chunklist
cp -rp /Volumes/install_app/BaseSystem.dmg /Volumes/OS\ X\ Base\ System/BaseSystem.dmg

# Unmount the installer image
hdiutil detach /Volumes/install_app

# Unmount the ElCapitan ISO Image
hdiutil detach /Volumes/OS\ X\ Base\ System/

# Create a sparseimage and auto-size it
hdiutil convert /tmp/ElCapitan.cdr.dmg -format UDSP -o /tmp/ElCapitan
hdiutil resize -size `hdiutil resize -limits /tmp/ElCapitan.sparseimage | tail -n 1 | awk '{ print $1 }'`b /tmp/ElCapitan.sparseimage

# Convert the ElCapitan sparseimage to ISO/CD master
hdiutil convert /tmp/ElCapitan.sparseimage -format UDTO -o /tmp/ElCapitan

# Remove the unwanted images
rm /tmp/ElCapitan.cdr.dmg
rm /tmp/ElCapitan.sparseimage

# Rename the ElCapitan ISO image and move it to the desktop
mv /tmp/ElCapitan.cdr ~/Desktop/ElCapitan.iso