The biological world, especially at the level of behaviour and cognition in animals, is full of fascinating complex organized structures which origins and formation largely remains to be understood. In particular, it is unclear how new structures are generated at both the level of phylogenesis (i.e. through biological evolution of populations of organisms) and at the level of ontogenesis (i.e. through development and learning during the life-time of individual organisms). Darwinian theory proposes that the space of structures is explored through random variations and differential selection of existing structures. But considering that randomness only can account for the generation of new complex organized structures is problematic due to the size of the search space. Similarly, the space of skills that organisms may learn is so vast that random exploration of the interaction of their body with their physical and social environment would be ineffective. It is thus of high interest to understand which are the mechanisms that constrain exploration in evolution and learning and allow at the same time the open-ended creation of novel biological, behavioural and cognitive structures. This talk will present computational robotic models and experiments that can allow us to understand better some of these mechanisms, namely self-organization and active developmental exploration mechanisms. In particular, I will present a model that illustrates how self-organization can enlighten our understanding of the evolution of shared combinatorial vocalization systems in populations of artificial organisms. Then, I will present machine learning architectures that can allow robots to acquire new skills through socially guided learning, and then through curiosity-driven exploration and active self-experimentation. Finally, I will outline a set of maturational constraints as well as mechanisms of morphological computation which should work in concert with curiosity to allow for the organized exploration of an open-ended space of skills and knowledge.