WP5: Interactive continuous learning of cross-modal concepts

An important characteristic of a cognitive system is the ability to continuously acquire new knowledge and new skills in a life-long manner. We refer to such ever-present, life-long learning as continuous learning. Moreover, the learning process in an artificial cognitive system is inherently cross-modal. The environment is perceived through different sensors (e.g. visual, haptic), and is acted upon using different actuators (motor wheels, robot arms). The interaction with the environment and communication (verbal and non-verbal) with a tutor should significantly facilitate incremental learning processes. These processes might induce different levels of tutor involvement and different levels of robot autonomy. Our goal is to analyse these different types of interactive learning and to develop a system (in collaboration with WP 6) that would be able to seamlessly switch between different learning modes in a convenient way.

This workpackage will therefore focus on interactive continuous learning of cross-modal concepts
and on detecting the gaps in categorical knowledge and filling these gaps by updating the corresponding concepts. The system should build its competencies incrementally. Initially, simple concepts (e.g., colours, shapes, etc.) will be built by grounding these concepts to sensory data (i.e. associating them with the features extracted from the sensory data) using a combination of pre-linguistic learning and learning based on communication with the tutor. With the benefit of these acquired concepts as a firm basis, it should continuously build new crossmodal and amodal concepts achieving a progressively richer ability to reason, plan and explain the environment and the robot’s situatedness in this environment.

The main objectives of this WP are to:

• Develop a system for interactive continuous learning of cross-modal concepts.
• Investigate how to best incorporate prelinguistic learning of discrete sets of cross-modal concepts to aid the acquisition of linguistic concepts.
• Investigate how to best support different modes of learning by an interactive self-extending architecture.
• Integrate tutor involvement with an interactively learning system in a user friendly and flexible manner.
• Find representations of skills, concepts and experiences that can form the basis of knowledge boundary identification (i.e. ignorance identification).
• Advance the learning system to be able to actively plan and execute new actions that may increase the system’s knowledge.
• Address the stability/plasticity dilemma by means of introspective management of desiresand active cross-modal validation of concepts and skills.
• Find cross-modal representations appropriate for merging information stemming from vision, haptics, language, manipulation, planning etc.
• Find a mechanism for combining concepts into novel concepts at a higher level of abstraction.