In computer science, an actor system is a way of building programs that are very concurrent and very amenable to scheduling and management.
The idea of an actor system goes back to the PhD work of Gul Agha. His actor model structures programs around a collection of simple agents called (unsurprisingly) actors. Each actor associates a mailbox with a behaviour. The mailbox receives messages from other actors or from outside the actor system. Messages are queued in mailboxes until processed one at a time by the associated behaviour.
The behaviour itself is a piece of code that, when run to process a message, performs bounded computation on the message's contents, which may involve sending messages to other actors and creating other actors (and their mailboxes). The boundedness of the computation is important: an actor is guaranteed to run for a finite amount of time before completing, and so cannot (for example) perform an unbounded loop. An actor's last action before terminating is to nominate a replacement behaviour for its mailbox, which may be the current behaviour or some new behaviour. (A null behaviour that did nothing in response to a message would essentially delete the actor.)
The complexity of the system is clearly going to come from how the behaviours are selected an scheduled. The model says very little about scheduling, leaving the implementation to decide when to process messages (by running the behaviour of the associated actor). A single-threaded implementation might repeatedly select a mailbox at random, check whether it contained messages and, if so, process one. A multi-threaded implementation could have one thread per mailbox running behaviours as messages arrive. There are plenty of other possibilities in between: the point is that an actor program doesn't control the concurrency, it simply induces it by the way it creates actors and sends messages.
A system without unbounded loops can't express general computation, but actor systems do allow unbounded computation: they simply force the programmer to create it using communicating actors. An actor wanting to loop forever could, for example, receive a message, perform some processing, send another message to itself (its own mailbox), and then nominate itself as its own replacement behaviour, which would then receive the self-sent message, and so forth.
If the actor model sounds restrictive, that's because it is deliberately designed that way. Its strength is that it is immune from deadlock, since the finite behaviours cannot become stuck indefinitely. This doesn't preclude the possibility of livelock if the system busily processes messages without actually making progress. However, the boundedness of behaviours means that the scheduler is always guaranteed to get control back on a regular basis, which means that there is always the possibility of an actor being able to run, making actor systems immune to starvation.
It's easy to build something that looks roughly like an actor system in a general-purpose programming language -- and usually pretty much impossible to build something that is actually an actor system. This is because a general-purpose programming language will allow behaviours that include unbounded loops, so you can't guarantee that a behaviour will terminate, and so you lose one of the major features of actor systems: their deadlock-freedom. With suitable programmer care, however, you can build an actor system quite easily, deploying however much concurrency is appropriate for the application and platform the system runs on.