The best example of an intelligent agent is a person! One of the most natural ways people use to interact is via speech dialogue. In this work we are exploring the issues connected to producing speech dialogue systems for intelligent environments. We are focusing on two main aspects; first how to best use contextual information, available from the wider pervasive network, and secondly, the building of natural language dialogue systems that enable complex activities, such as programming digital homes to be accomplished via speech.