Exploiting Spatiotemporal Patterns for Event Response using Digital Pheromones and Reinforcement Learning
A combination of digital pheromones and reinforcement learning was used to effectively anticipate hostile actions taken by adversaries.
Sequences of events resulting from the actions of human adversarial actors such as military forces or criminal organizations may appear to have random dynamics in time and space. Finding patterns in such sequences and using those patterns in order to anticipate and respond to the events can be quite challenging.
Digital pheromones are a computational technique inspired by the chemical substances deposited in the environment for communication by many social insects, such as ants. Essentially, digital pheromones are data added to a geospatial data structure or image that represent event-related information that is not physically present in the environment but can be computed and associated with locations and times in the environment.
Digital pheromones combined with reinforcement learning were employed to predict attacks by adversarial human actors. Environments initially containing only conventional geospatial information and the spatiotemporal locations of sequences of adversary attacks were augmented with a range of application-specific digital pheromones, with a goal of guiding the prepositioning of defensive agents in time and space. The digital pheromones were used to synthetically augment event information to create a more persistent presence in the environment that supported event prediction.
In the first phase of this work, digital pheromones and reinforcement learning were applied to the historical sequence of actual Somali maritime pirate attacks from 2005 to 2012. The algorithm was used to control the movements of a set of simulated defensive naval vessels. Over multiple trials, the simulated naval vessels successfully responded to an average of 333 of the 899 historical pirate attacks. Notably, this level of success significantly outperformed the historical record of 139 successful responses by actual naval forces in the region.
In the second phase of the work, the method was extended with a new partial-state reinforcement scheme, as compared to standard reinforcement learning based only on fully matching states. The enhanced learning was again combined with a set of application-specific digital pheromones. The methodology was used to preposition police patrol cars in response to a historical sequence of business robberies in Denver, Colorado. The method was again more effective than a typical police heuristic for positioning patrol cars within an effective distance of future robberies.
