Best way to improve UDP transfer speeds with Python?

I have written a Python program for sending and receiving large files over UDP. Right now, I can hit speeds of about .01GB/s when transferring between two computers on a 10 Gigabit Ethernet cable. I'm looking to improve that speed significantly, but I'm unsure what the best approach is.

For what it's worth, I MUST USE UDP for the transfer. The program I wrote is merely a test for a larger project, and the device sending the data for that project can't utilize a TCP stream. Additionally, I'm primarily focused on the best way to receive datagrams quickly, or at least the best way of ensuring that the receiving end isn't where any bottlenecks occur.

Right now, my program works by chunking up a large file into several pieces that will become the datagrams to be sent. These datagrams are sent, and then the receiver does some stuff to ensure that it got the proper data and orders it accordingly.

Sending Code (stripped down to basics)

buf = 32000  #Size of each chunk/packet
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
host ="127.0.0.1"
port = 5005
addr = (host,port)    

def read_in_chunks(infile, chunk_size=buf):
"""Chunk the file before we send it.
Arguments:
infile -- the file to chunk
chunk_size -- the size of the chunk in bytes (default 32KB)
"""
    while True:
        chunk = infile.read(chunk_size)
        if chunk:
            yield chunk
        else:
            # The chunk was empty, which means we're at the end of the file
            return

def run():
    for chunk in read_in_chunks(f):
        if(s.sendto(chunk,addr) and s.sendto(id,addr)):
            #Some acknowledgment stuff - removed for clarity (also noted to not impact performance)
            local_ID += 1

Receiving Code:

UDP_IP = "127.0.0.1"
UDP_PORT = 5005
buf = 32000  # Buffer size in bytes for each chunk
sock = socket.socket(socket.AF_INET,  # Internet
                 socket.SOCK_DGRAM)  # UDP
sock.bind((UDP_IP, UDP_PORT))

try:
    while(dataChunk):
        actualNumChunks += 1
        f.write(dataChunk)
        sock.settimeout(2)
        dataChunk, addr = sock.recvfrom(buf)
        packID, junkAddr = sock.recvfrom(buf)
        packIDString = str(packID)
except socket.timeout:
    f.close()
    sock.close()
    print "File received!"  # --KEEP
    print "A total of " + str(actualNumChunks) +" chunks were received"            --KEEP

I'm not sure if optimizing my code is the issue (yet to be tested), or if there is another (better?) way of getting increased speed for the file transfer. I apologize if the details here are sparse, but let me know if you need any more information.

Thanks!

Solution 1:

Some approaches to try:

1. Varied packet sizes (depending on MTU, might cause the chunk to be fragmented to several packets, or might be too small to optimally use the pipe)
2. Read the file to memory (the network speed might be 10gbps, but the disk is way slower, especially if it isn't an SSD - typically an order of magnitude or more slower), start sending after it has all been cached. You can try sending a the same block of data many times to check if this is the bottleneck.
3. Compression might help. ZLIB is very very very fast, and would mean you could transfer a lot more data in way less time,

Other points:

• Packet loss and ordering might make a proper transfer problematic, even in a simple 1-to-1 link. You'll need error-detection/retry functionality for this to work.
• Kernel/User mode changes might slow you down (doubt it, and this is super micro-optimization)