How to drop first udp packet from a source with iptables?

As a defense against distributed udp flood, I need to drop first udp packet from a source and accept other than first packet. Does iptables has capability of doing this? If yes, how?

I don't know if iptables has a feature like the one you are asking for, but I can explain why such a feature wouldn't help you.

If you receive a flood of unwanted UDP packets, they can either be targeted at a closed or an open port. If they are targeted at a closed port, the kernel will send a proper ICMP error response back. This error response is important because without it debugging network problems becomes difficult, and in certain scenarios this error response is also the only way others can learn about ongoing attacks and mitigate them.

A flood of UDP packets destined at a closed port could lead to a flood of ICMP error messages. But a decent stack will automatically rate limit the error messages, so that shouldn't be a concern. The bandwidth consumed by the UDP packets in the first place is wasted, and you aren't going to change that by dropping the packet after it has already wasted bandwidth.

If the UDP packets are destined at an open port things get a bit more complicated. Due to the stateless nature of UDP, packets get delivered to the application without any chance of the kernel or the application knowing if the source IP is real or spoofed. It is the responsibility of the higher layer protocol to validate the source IP if necessary. DNS and NTP are examples of protocols which have failed at this and can be abused for reflection/amplification attacks.

Many UDP based services can handle large streams of UDP packets due to their stateless nature. If they don't store state, it is just packet in - packet out and forget. The service doesn't really care if the source IP was legitimate or not and due to the stateless nature of the service, a flood of packets with spoofed source cannot cause memory to be wasted on state.

However if you were to deploy a firewall which drops the first UDP packet from a source and let later packets through, then the firewall need to maintain state. Without state it couldn't know if a packet was the first or not.

So now you have put a stateful firewall in front of a stateless service communicating with a stateless protocol. At this point a flood of packets with spoofed source will still not bring down the service, it will however bring down the firewall. The firewall has to remember the source of every single packet it received. So the firewall will run out of memory, and it will have to forget some of the earlier received packets. That means even for legitimate sources, the 2nd, 3rd, etc. packets will get dropped because the firewall simply can't remember that it has seen earlier packets from that source.

So the feature you are asking for would turn a relatively harmless DoS attempt into a much more powerful attack against your firewall.

There is nothing new about this sort of scenario. Decades ago TCP stacks were vulnerable to SYN floods which would DoS a service for the exact same reasons mentioned above. This is why SYN cookies were invented.

If you are designing any new protocol to run on top of UDP (or on top of any other stateless protocol), you have to keep this in mind and apply proper design criteria to make the protocol resistant to spoofing attempts. If you are not designing a protocol, then all you can do is ask the people who are designing the protocol to do so properly.