PulseAudio Loopback Latency

I am using pulseaudio loopback's to link together my system sounds and my microphone. The problem is that my system sounds have a delay of a second or even more sometimes. I have changed ticked simultaneous in paprefs but that has not fixed it. I am using this script to do so.

STREAM_MODULE=$(pactl load-module module-null-sink sink_name=StreamAudio sink_properties=device.description="StreamAudio")

pactl load-module module-loopback source=alsa_output.pci-0000_00_1b.0.analog-stereo.monitor sink=StreamAudio latency_msec=1 >> /dev/null  
pactl load-module module-loopback source=alsa_input.usb-AKM_AK5370-00-AK5370.analog-mono sink=StreamAudio latency_msec=1 >> /dev/null

pactl load-module module-device-manager >> /dev/null  

read -p "Loaded! Press enter to unload!"

pactl unload-module $STREAM_MODULE
pactl unload-module module-device-manager
pactl unload-module module-null-sink
pactl unload-module module-loopback

I have gone through pages of google, but nothing seems to fix it. Is there something obvious I am missing?

alsa_output.pci-0000_00_1b.0.analog-stereo.monitor are all my system sounds.

alsa_input.usb-AKM_AK5370-00-AK5370.analog-mono is my microphone

Here is an example where the delay becomes 3.4 seconds: http://screencloud.net/v/tNAQ while the other loopback has a lot less http://screencloud.net/v/suIF

From what I can tell, for some reason the buffer fills up. Any way I can flush it or something?

I have found a temporary way out, but this still confuses me. I have made a combined module, and after that I link every sound source I want to the combined one, which then routes to my system sounds and to the Streaming sink.

STREAM_MODULE=$(pactl load-module module-null-sink sink_name=StreamAudio sink_properties=device.description="StreamAudio")
COMBINE_MODULE=$(pactl load-module module-combine-sink sink_name=Combined sink_properties=device.description="Combined" slaves=StreamAudio,alsa_output.pci-0000_00_1b.0.analog-stereo)

pactl load-module module-loopback rate=44000 adjust_time=0 source=alsa_input.usb-AKM_AK5370-00-AK5370.analog-mono sink=StreamAudio latency_msec=1 >> /dev/null

pactl load-module module-device-manager >> /dev/null  

read -p "Loaded! Press enter to unload!"

pactl unload-module $COMBINE_MODULE
pactl unload-module $STREAM_MODULE
pactl unload-module module-device-manager
pactl unload-module module-loopback

There is probably a way I can grep the results of pacmd list-clients and then clean out only valid sources, change their output to the combined sink, but that would take too long and complex to do.

Though now a new problem arises, and I even tested it by running the command through terminal. For some reason I can't change the output sink of Java. Here is what I tried and what it returned:

vilsol@skynet:~$ pacmd move-sink-input 6421 124
Welcome to PulseAudio! Use "help" for usage information.
>>> Moved failed.

I tried with others as well and they all worked. 6421 is the Java index, and 124 is the sink index.