Linux SCHED_OTHER, SCHED_FIFO and SCHED_RR - differences
SCHED_FIFO and SCHED_RR are so called "real-time" policies. They implement the fixed-priority real-time scheduling specified by the POSIX standard. Tasks with these policies preempt every other task, which can thus easily go into starvation (if they don't release the CPU).
The difference between SCHED_FIFO and SCHED_RR is that among tasks with the same priority, SCHED_RR performs a round-robin with a certain timeslice; SCHED_FIFO, instead, needs the task to explicitly yield the processor.
SCHED_OTHER is the common round-robin time-sharing scheduling policy that schedules a task for a certain timeslice depending on the other tasks running in the system.
Update: since Linux 3.14, there is an additional policy called SCHED_DEADLINE. This policy implements the Constant Bandwidth Server (CBS) algorithm on top of Earliest Deadline First queues. Each task under this policy is assigned a deadline, and the earliest-deadline task is executed. The best resource describing this algorithm is Deadline scheduling in the Linux kernel.
Update 2: since Linux 4.13, SCHED_DEADLINE has replaced CBS with the Greedy Reclamation of Unused Bandwidth (GRUB) algorithm.
Here is the differences between SCHED_OTHER, SCHED_FIFO and SCHED_RR based on Linux Manual (http://man7.org/linux/man-pages/man7/sched.7.html)
SCHED_FIFO: First in-first out scheduling
SCHED_FIFO can be used only with static priorities higher than 0, which means that when a SCHED_FIFO threads becomes runnable, it will always immediately preempt any currently running SCHED_OTHER, SCHED_BATCH, or SCHED_IDLE thread. SCHED_FIFO is a simple scheduling algorithm without time slicing.
SCHED_RR: Round-robin scheduling
SCHED_RR is a simple enhancement of SCHED_FIFO. Everything described above for SCHED_FIFO also applies to SCHED_RR, except that each thread is allowed to run only for a maximum time quantum. If a SCHED_RR thread has been running for a time period equal to or longer than the time quantum, it will be put at the end of the list for its priority.
SCHED_OTHER: Default Linux time-sharing scheduling
SCHED_OTHER can be used at only static priority 0 (i.e., threads under real-time policies always have priority over SCHED_OTHER processes. SCHED_OTHER is the standard Linux time-sharing scheduler that is intended for all threads that do not require the special real-time mechanisms.