Artificial Intelligence will have huge implications on the lives of all citizens, including persons with disabilities who can already benefit of a number of AI-based applications. We nevertheless need strong legal safeguards to protect the rights of all citizens, including citizens with disabilities, from AI-powered technology that could cause them harm. In this context, an assessment of the potential gaps in human rights law that currently protect European citizens is required. Industry-driven guidelines on AI ethics are not sufficient to future-proof the rights of persons with disabilities. Self-regulation and voluntary compliance with ethics guidelines are not enough to offer reassurance to consumers with disabilities. We need a clear legal framework in place, clear accountability and a right to redress.