II. Formal sensorimotor theory

The purpose of this workpackage is to understand what are the feels of the body, of sensory modality, of external space, and to address these questions in a mathematically rigorous way.

In robotics numerous attempts are being undertaken to make agents — robots — understand the structure of their body and of the environment. The problem of the existing approaches is that they start from an engineering point of view: they assume, for example, that the visual modality is just information provided by a camera and that the body is a set of connected rigid segments. However, the notion of body or vision for an engineer may be different for a living agent.

Here we will assume four primary notions to be given: the agent; the raw sensory inputs which we will call for short: sensations S; raw motor commands which we will call for short: actions A; and the environment E. All are assumed to be Banach space, since these are not limited to being numbers, but can also be functions. We will assume the agent performs experiments by producing actions A and collecting sensations S under different unknown states of the environment E. We define feels as sensorimotor laws or contingencies φ, such that φ(S,A) = 0, relating sensations S and actions A. Different feels arise from the fact that sensations can never be entirely determined by actions because they also depend on the state of the environment E. Hence the agent extracts relationships between certain projections of the sensations and actions — e.g. tactile input from the skin, and muscle output to the fingers via the muscles in the forearms; and only for a particular subset of states of the environment — e.g. when there is a sponge between the fingers. In this case φ would specify the sensorimotor law for the feeling of “softness” and which is valid only for skin and muscles and is applicable only when the sponge is grasped. In some cases sensorimotor laws can be associated together and parametrized with emergent parameters. Thus perception can be seen as the process of making subclasses of sensorimotor experience, constrained by the environment (cf. von Uexküll’s “Umwelt”), with associated parameters within each class. For higher level feels, like body and object, the sensorimotor relationships may be established on the percepts (like softness and color) rather than on the raw sensorimotor flux.