RFR: Active Vision

Proposed by Cerenaut and the Whole Brain Architecture Initiative
(What is a Request for Research/RFR?)

Summary

Background and Motivation

In the following, active vision means vision with eye movement.

The human eye has higher resolution in the fovea and lower toward the periphery. So, to gather detailed information from the entire visual field, the gaze must move around.

Since human vision requires active vision, AI that mimics humans also would require it. Thus, implementing active vision can be the first step for human-brain-inspired AI.

Active vision involves information processing that combines data collected from more than one time point (data binding), which requires short-term memory and a representation of “scenes” (situations). Eye movement is also apparently related to the mechanism of ‘attention.’ Information processing using short-term memory (working memory), handling of representations of situations, and the mechanism of attention can be regarded as basic functions for general intelligence and important for the realization of AGI (artificial general intelligence).

Objective

The task here is to implement biologically plausible active vision that can perform tasks requiring gaze motion.

For an overview of the system design, see Detailed Project Description below.

Success Criteria

The implementation will be judged with the following criteria:

Biological plausibility
The implementation should be ‘compatible’ with the structure and function of the mammalian brain.
Usability
The implementation should be easily used and maintained together with documentation.
Specifications
See the Detailed Project Description section below.
Performance
Use one or more tasks from the Dataset and Test section below.

Detailed Project Description

Tasks that require Active Vision

Point to Target Task

The agent has to move the gaze to the position where a cue appears.

Fig.1

Image Recognition by Gaze Movement

The agent has to identify “scenes” that extend over the entire field of vision so that they can only be identified by moving the gaze.
Scene 1: The cat is to the left of the door. Scene 2: The cat is to the right of the door.
Fig.2

Oculomotor Delayed Response Task

The agent has to move the gaze to the position where the cue was a few seconds after it disappeared. This simple task requires working memory.

Fig.3

Other tasks

2018 WBA hackathon tasks: 7 tasks were used: i.e., Point To Target, Random Dot, Odd One Out, Visual Search, Change Detection, Multiple Object Tracking tasks. You need to analyze what kinds of abilities are required for the tasks before working on them.

In tasks known as Object Centric Learning, it is required to visually recognize where and what kind of objects are. More broadly, papers with code for visual reasoning can be found here.

Mechanisms to be Prepared

Peripheral and Central Visions

The agent processes the high-resolution image centered on the gaze position from the environment to produce peripheral and central image data.

The peripheral and central image data can be a single set of data, such as an image based on log-polar coordinates, or two sets of data corresponding to each.

The peripheral image could be in monochrome or polychrome.

Saccade and Saliency Map

The rapid eye movements to capture visual objects in the fovea are called saccades. The eye does not move randomly, but rather moves toward the most salient or “interesting” object. The saliency factors include differences in color, brightness, and movement from the surroundings. It has been hypothesized that there is a “saliency map” for the visual field.

Given a saliency map, it is necessary to design:

the conditions (threshold, etc.) and timing of saccade
how physically “realistic” to reproduce the motion (e.g., the eyeball is a sphere with mass and is driven by muscles)
whether the relationship between the saliency map and eye movements should be determined by design or learned

Image Processing

You need to decide how to process the images (bitmaps) from the environment. First, design a method to create image data corresponding to peripheral and central vision; here an image processing library such as OpenCV can be used.

If image recognition is to be performed, the method should be chosen. If you use an artificial neural network (deep learning), you should also decide whether some pre-processing will be performed with image processing libraries. For an artificial neural network, the algorithm to be used should be determined. Since the brain does not use back-propagation in an end-to-end fashion, it would be necessary to incorporate an unsupervised learner such as an autoencoder to make it a brain-inspired AI.

Representing Scenes

Active vision combines data collected from points in time to create representations of ‘scenes.’ You have to design how you will combine the data. If you use an artificial neural network, one idea is to keep data from time points as the internal state of a recurrent neural network (RNN). Also, the representation of the scene must express ‘what and where.’ For this, you can design a certain coordinate system and a mechanism that associates the representation of the place (where) and the representation of the object (what).