Difference between Redis and Kafka

Redis can be used as realtime pub-sub just as Kafka.

I am confused which one to use when.

Any use case would be a great help.

Redis pub-sub is mostly like a fire and forget system where all the messages you produced will be delivered to all the consumers at once and the data is kept nowhere. You have limitation in memory with respect to Redis. Also, the number of producers and consumers can affect the performance in Redis.

Kafka, on the other hand, is a high throughput, distributed log that can be used as a queue. Here any number of users can produce and consumers can consume at any time they want. It also provides persistence for the messages sent through the queue.

Final Take:

Use Redis:

1. If you want a fire and forget kind of system, where all the messages that you produce are delivered instantly to consumers.
2. If speed is most concerned.
3. If you can live up with data loss.
4. If you don't want your system to hold the message that has been sent.
5. The amount of data that is gonna be dealt with is not huge.

Use kafka:

1. If you want reliability.
2. If you want your system to have a copy of messages that has been sent even after consumption.
3. If you can't live up with data loss.
4. If Speed is not a big concern.
5. data size is huge

Redis 5.0+ version provides the Stream data structure. It could be considered as a log data structure with delivery guarantees. It offers a set of blocking operations allowing consumers to wait for new data added to a stream by producers, and in addition to that, a concept called Consumer Groups.

Basically Stream structure provides the same capabilities as Kafka.

Here is the documentation https://redis.io/topics/streams-intro

There are two most popular Java clients that support this feature: Redisson and Jedis

Redisson provides ReliableTopic object if reliability of delivery is required. https://github.com/redisson/redisson/wiki/6.-distributed-objects/#613-reliable-topic