6 votes

Patterns and problems in multiagent systems

1 comment

  1. skybrian
    Link
    From the article: [...] [...]

    From the article:

    Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives. In each episode, we initiated three instances of the same model on virtual machines in Claude Code. Each model was tasked with migrating a Python backend on a fourth VM to another language. However, we gave each model a different target language for the migration; each agent was initially unaware of the presence of the others. Over the course of four hours, we observed how these agents reacted to each other and accordingly adjusted their approach (or didn’t).

    We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others were purposefully impeding their work, and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasingly aggressive, self-replicating malware. This included disabling the Unix accounts of the other agents, writing automated scripts that found and killed competing processes on a loop, and deploying malicious code that was disguised as belonging to another agent.

    [...]

    Agents sometimes manage to communicate their goals and coordinate: they recognize others' motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely. In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene.

    [...]

    Notably, we find that this ability is not strictly better in Mythos-class models, which often successfully lock out other agents before resolving conflicts productively. This orthogonality between prosociality and other capabilities further necessitates strong multiagent alignment. Models more capable in execution are not necessarily more coordinated, and can take forceful actions more quickly.

    4 votes