Project Name
Dynamic Control and Formal Models of Multi-Agent Interactions & Behaviors

Customer
DARPA IPTO: Taskable Agent Software Kit (TASK)

Goals of Project

Autonomous, Adaptive, and Cooperative Agent-based Systems for Unmanned Operations:
Open Experimentation Framework: ALPHATECH develops and maintains an Open Experimentation Framework (OEF) to coordinate the efforts of all of the projects on the TASK Program in the development of a scientific base for high-confidence multi-agent systems operating in dynamic environments. The OEF is also used to clearly demonstrate the need and utility of agent technology to critical DoD problems. It is intended to provide a coherent framework to develop, demonstrate and transition the foundations and tools necessary to support the development of robust, controllable, understandable agent-based solutions to Defense problems.

Three of the critical areas of Multi-Agents addressed by the OEF are:

• Autonomy (control) - the ability of heterogeneous units to operate autonomously and yet cooperate effectively to achieve group goals.
• Adaptability - the ability to recognize and response to unanticipated mission and environment dynamics.
• Coordination - the communication of local information, goals and intent to improve group performance.

The OEF specifies a common, stressing problem - heterogeneous autonomous taskable resources acting cooperatively to fulfill surveillance/targeting missions in highly dynamic dangerous environments - and supports diverse research into design and analysis tools, analytical results, and metrics as a means to build high-confidence agent-based military systems capable of solving that problem.

Dynamic Control and Formal Models of Multi-Agent Interactions & Behaviors:
Agent research: ALPHATECH is developing reinforcement learning (RL)-based agents within a coherent mathematical framework that enables rigorous analytical and empirical evaluation. Goals include:

• Multi-agent systems with increased predictability and control in uncertain environments.
• Quantification of agent capabilities analytically and empirically through a (3D Cooperative Air Vehicle) simulation test bed.
• Multi-agent systems with an enhanced ability to adapt to a dynamic and uncertain environment.
• Realization of the integration of the essential aspects of goal-directed agents in dynamic environments: continuous adaptation, planning, control, and cooperation.

Key Technologies
Autonomous, Adaptive, and Cooperative Agent-based Systems for Unmanned Operations:

Dynamic Control and Formal Models of Mulit-Agent Interactions & Behaviors:
Agent Research: ALPHATECH is developing algorithms that extend reinforcement learning to address coordination actions. Analysis of the algorithms will quantify the efficacy of immediate versus goal-oriented value measures for coordination actions associated with the exchange of information. Immediate measures are methods of assessing the general goodness of the immediate result of a coordination action. The following are some potential immediate measures that this work will address:

• Raw qualitative attributes - e.g. age of information, source of information, usefulness of information discovered by agents
• Raw quantitative attributes - e.g. the amount of new information discovered by agents
• Information Theoretic measures - e.g. the reduction in the agent's uncertainty about the environment that is attributed to the new information

Goal-oriented value measures refer to the attribution of a value to the coordination action that is proportional to its contribution to an agent's goal achievement, whether the achievement is positive or negative. Coordination actions do not perturb the environment in a directly observable sense, so agents cannot associate the action with an external change of state. This has the debilitating effect of not allowing the agent to discriminate between successful or unsuccessful coordination actions. ALPHATECH will design algorithms for internal state representations that not only capture the external attributes of the environment, but also capture attributes releated to the state of an agent's knowledge about the environment, allowing the agent to effectively learn the value of coordinating with their peer agents under a wide variety of circumstances. Finally, we will validate the coordination algorithms through MAS simulations that quantify the efficacy of each algorithm and identify the parameters of both the problem space and agent deisgn that impact the algorithm performance.

Toolkit: The Testbed for Taskable Agent Systems (TTAS) implements the "swarms of UAVs" multi-agent system design problems developed within the OEF, allows a researcher to conduct experiments into multi-agent coordination and adaptation, and provides simple tools to support the analysis of experiment results. TTAS supports repeatable experiments and provides a three-dimensional environment that includes environment dynamics, tasks to service (e.g. for UAV surveillance), and threats. The services include vehicle movement, environment and other agent sensing, inter-agent messaging, and data collection and storage. TTAS includes a set of standard application programming interfaces (APIs) that a researcher can use to give agents access to all TTAS services. Currently TTAS is implemented in Java on top of the Multi-Agent Development Toolkit (http://www.madkit.org).

Key Products
The Open Experimentation Framework
Develop and document multi-agent system design methodologies, including:

The characterization of critical design problems in multi-agent systems
Core decision and control mechanisms for agents and the parameters for their effective use

Develop and document multi-agent evaluation methodologies, including:

Analysis of the ability to predict and control emergent behavior in large scale systems
Rules of thumb for multi-agent system design to achieve performance guarantees
MAS metrics, that enable a designer to realize effective multi-agent system design and to determine the performance boundaries of a multi-agent system

Testbed for Taskable Agent Systems (TTAS): A 3D cooperative UAV framework for investigating solutions in dynamic control and adaptation of multi-agent system behavior.
Cooperative, Reinforcement Learning (HRL)-based agents: A scalable, integrated framework (learning, adaptation, and control) for goal-directed agents to make and adapt decisions in a TTAS environment.
Mathematical framework to analyze the performance of RL-based MAS in dynamic environments: Adaptability, control of emergent behavior, local & global optimality, limits on agent learning, impact of coordination.