When to use actors instead of messaging solutions such as WebSphere MQ or Tibco Rendezvous?

First off the "older" message systems (MQ) are older in implementation but they are a newer in engineering idea of: transactional persistent queues. Scala Actors and Akka maybe a newer implementation but are built on an older concurrency model of Actors.

The two models however end up being very similar in practice because they both are event message based: See my answer to RabbitMQ vs Akka.

If you're going to code only for the JVM then Akka is probably a good choice. Otherwise I would use RabbitMQ.

Also if you're a Scala developer, then Akka should be a no-brainer. However Akka's Java bindings are not very Java-ish and require casting due to Scala's type system.

Also in Java people don't typically make immutable objects which I recommend you do for messaging. Consequently its very easy in Java to accidentally do something using Akka that will not scale (using mutable objects for messages, relying on weird closure callback state). With MQ this is not a problem because the messages are always serialized at the cost of speed. With Akka they are generally not.

Akka also scales better with large amount of consumers than most MQ. This is because for most MQ (JMS, AMQP) clients every queue connection requires a thread... thus lots of queues == lots of permanently running threads. This is mainly a client issue though. I think ActiveMQ Apollo has a non-blocking dispatcher that purportedly fixes that issue for AMQP. The RabbitMQ client has channels that allow you to combine multiple consumers but there are still issues with large number of consumers potentially causing deadlocks or connections to die so generally more threads are added to avoid this issue.

That being said Akka's remoting is rather new and probably still doesn't offer all the reliable message guarantees and QoS that traditional message queues provide (but that is changing everyday). Its also generally peer-to-peer but does I think support server-to-peer which is generally what most MQ systems do (ie single point of failure) but there are MQ systems that are peer-to-peer (RabbitMQ is server-to-peer).

Finally RabbitMQ and Akka actually make a good pair. You can use Akka as a wrapper to RabbitMQ particularly since RabbitMQ does not help you with handling the consumption of messages and routing the messages locally (in a single JVM).

When to choose Akka

• Have lots of consumers (think millions).
• Need low latency
• Open to the Actor concurrency model

Example system: An interactive real time chat system

When to choose MQ

• Need to integrate with lots of different systems (ie non JVM)
• Message reliability is more important than latency
• Would like more tools and admin UI
• Because of previous points better for long running tasks
• Would like to use a different concurrency model than Actors

Example system: A scheduled transactional batch processing system

EDIT based on concerned comments

I made an assumption that the OP was concerned with distributed processing which both Akka and Message Queues can handle. That is I assumed he was talking about distributed Akka. Using Akka for local concurrency is an apples to orange comparison to most message queues. I say most because you can apply the message queue model locally as a concurrency model (ie topic, queues, exchanges) which both the Reactor library and simple-react do.

Picking the right concurrency model/library is very important for low latency applications. A distributed processing solution such as a message queue is generally not ideal because the routing is almost always done over the wire which is obviously slower than within application and thus Akka would be a superior choice. However I believe some proprietary MQ technologies allow for local routing. Also as I mentioned earlier most MQ clients are pretty stupid about threading and do not rely on non-blocking IO and have a thread per connection/queue/channel... ironically non-blocking io is not always low latency but is generally more resource efficient.

As you can see the topic of distributed programming and concurrent programming is rather large and changing everyday so my original intention was not confuse but rather focus on one particular area of distributed message processing which is what I though the OP was concerned with. In terms of concurrency one might want to focus their searches on "reactive" programming (RFP / streams) which is a "newer" but similar model to the actor model and message queue model of which all of these models can be generally combined because they are event based.

I'm not an expert in messaging systems, but you can combine them with Akka in your apps, getting the best of both worlds. Here's an example that you might find useful for experimenting with Akka and messaging systems, in this case ZeroMQ:

https://github.com/zcox/akka-zeromq-java

Akka-Camel would be a better example than ZeroMQ - ZeroMQ is a direct tcp to tcp communication (hence zero - there is no message queue).

With AkkaCamel you can abstract away the queue and produce/consume messages direct from an actor without any code to deal with the message queue message pushing/pulling.

You can forego akka-zeromq and use Akka directly with remoting. I think akka-zeromq is being removed from the core library but we built a good zeromq library for akka called scala-zeromq (https://github.com/mDialog/scala-zeromq)

Akka has a couple key core use cases:

1) Mutable state

It's easier to handle shared state by hiding it in an actor. As actors handle messages synchronously, you can hold state in an actor and expose that field with high consistency via the actor API

2) Distribution

Concurrency is free in akka so you say it's really about solving distribution problems. Distribution across machines and cores. Akka has build in "location transparency" for sending messages over the wire. It has clustering and patters associated for scaling up a single service as well. This makes it a very good solution for distribution (eg micro-service architecture)

Here is an example of using Akka with ActiveMQ with Akka-Camel (using Java8)

import akka.actor.Props;
import akka.camel.Camel;
import akka.camel.CamelExtension;
import akka.testkit.TestActorRef;
import akka.testkit.TestProbe;
import org.junit.Ignore;
import org.junit.Test;
import akka.camel.javaapi.UntypedProducerActor;
import akka.camel.javaapi.UntypedConsumerActor;
import static com.rogers.totes.TotesTestFixtures.*;
import org.apache.activemq.camel.component.*;

public class MessagingTest {
    @Test @Ignore
    public void itShouldStoreAMessage() throws Exception{
        String amqUrl = "nio://localhost:61616";
        Camel camel = (Camel) CamelExtension.apply(system);
        camel.context().addComponent("activemq", ActiveMQComponent.activeMQComponent(amqUrl));

        TestProbe probe = TestProbe.apply(system);
        TestActorRef producer = TestActorRef.create(system, Props.create((Producer.class)));
        TestActorRef consumer = TestActorRef.create(system, Props.create((Consumer.class)));
        producer.tell("Produce", probe.ref());

        Thread.sleep(1000);
    }
}

class Producer extends UntypedProducerActor{

    @Override
    public String getEndpointUri() {
        return "activemq:foo.bar";
    }
}

class Consumer extends UntypedConsumerActor{

    @Override
    public String getEndpointUri() {
        return "activemq:foo.bar";
    }

    @Override
    public void onReceive(Object message) throws Exception {
        System.out.println("GOT A MESSAGE!" + message);

    }
}