Evaluation of European Disability Forum implementation of its strategic framework 2017-2021 and workplan 2018-2021
External Evaluator: Niall Crowley, Values Lab
This summary report presents the key findings of an evaluation of the European Disability Forum (EDF) as it reaches the half-way point of its Strategic Framework 2017-2021 and has completed the first two years of a four-year European Commission funded workplan. The evaluation involved: literature review; interviews with eleven EDF members attending the annual general assembly in June 2019; on-line survey of EDF members with twenty-four respondents (24% response rate); interviews with the EDF staff; and interviews with eight external stakeholders.
The European Disability Forum
The EDF pursues a vision for a Europe where persons with disabilities are “fully included in society on an equal basis with others and that our human rights, as outlined in the UN CRPD, are fully respected, protected and fulfilled”. Its mission is to “ensure persons with disabilities’ full inclusion in society and access to our human rights through our active involvement in policy development, implementation and monitoring of the UN CRPD in Europe” (Strategic Framework 2017-2021).
The Strategic Framework of the EDF establishes four core objectives for its work, which provide a framework for this evaluation. These are to:
- Influence the EU, in particular to ensure the UN CRPD is implemented and monitored alongside advancing accessibility and social inclusion for persons with disabilities.
- Strengthen the disability movement in Europe as a movement that is strong and diverse, with a capacity to influence at EU and Member State level.
- Ensure policy-makers, influencers, and the general public are aware of the situation of persons with disabilities, and the work the disability movement
- Advance rights-based international cooperation, influencing the EU’s external policies and engaging with the global disability movement.
The EDF works in a context where the EU has ratified the UN CRPD and committed to implement the Sustainable Development Goals (SDGs). This is also a context where economic policy in many Member States continues to be characterised by austerity policies and a reduced role for the state, to the detriment of persons with disabilities. It is a context where equality and human rights are experiencing a backlash with hostile political and public discourse seeks to divide society.
A broad range of recommendations are made in this evaluation. This is a measure of success and the future potential opened up by the work of the EDF, rather than pointing to problems in this work. As pointers for the future, many of the recommendations could inform a future strategy, being implemented over time.
Relevance is the key evaluation criterion for the EDF strategy. Relevance refers to the match between the work and approach pursued by the EDF under its four objectives and: the aspirations and situation of EDF members; the policy and practice of key stakeholders; and the changing and evolving context for persons with disabilities across Europe. The indicators for relevance are:
- The match between EDF objectives and member ambitions and the extent to which members agree that these ambitions are realised.
- The representativeness and inclusiveness of the voice of persons with disabilities brought forward by the EDF.
- The breadth of stakeholders engaged with and the balance achieved in the level of engagement with each the set of stakeholders from each sector.
There is a strong match evident and articulated between member ambitions and EDF objectives. 21% (5 respondents) reported in the survey that their ambitions were being completely met, 50% (12 respondents) that they were being significantly met. This is seen as being enabled by a secretariat and leadership that is experienced and expert, and supportive, listening and responsive.
The EDF is viewed by external stakeholders as being a representative and inclusive voice for persons with disabilities. Policy makers identify the EDF as a legitimate voice and a voice that, while demanding, is polite and has a capacity to be solutions focused. Civil society organisations identify the EDF as being open, supportive and non-territorial. The engagement with the social partners, employers and trade unions, is more limited than with these other stakeholders.
The EDF is seen as having a relevant policy focus. It is viewed as having an ability to be flexible, to track issues emerging, and to gear up to address these.
Intersectionality is a key focus in the work with civil society organisations. While this work is characterised by innovation, action to explore its conceptual underpinnings and full potential might further strengthen it. Persons with disabilities identify an ambition to be part of mainstream debates and this is proving elusive. Involvement in the SDGs and the development agenda they set out offer opportunities for this.
Recommendation: The EDF should:
- Develop a strategy for its engagement with the social partners and use this to deepen their engagement with employers and trade unions at European level and, through the members, at national level.
- Continue the work on intersectionality and deepen this engagement with relevant civil society organisations through shared exploration of the concept of intersectionality and its implications and potential for the pursuit of more equal societies.
- Expand its engagement over time with the 2030 Sustainable Development Agenda to seek an integration of equality for persons with disabilities in the overall societal development paradigm to be worked out as part of the implementation of this policy agenda.
Effectiveness is the key evaluation criterion for EDF operations. Effectiveness refers to the nature and level of the outputs of the EDF and the extent to which the EDF strategy and plan have been implemented and targets for 2018 and 2019 achieved across the four core objectives of: influencing policy; strengthening the movement; communication; and international cooperation. The indicators for effectiveness are:
- The balances achieved in human and financial resources allocated between: policy making and policy implementation; targeted disability policy and disability mainstreaming; and equality and human rights agendas.
- The quality of routes into and relationships with each of the EU institutions evident in the access achieved to these bodies.
- The level and quality of skills development opportunities for Members.
- Communication strategy operational that enables communication work to function as a tool for achieving change.
- The level and quality of cross-fertilisation achieved between regions within the global movement of persons with disabilities.
Conclusions for Influencing Policy
The strategic unifying vision underpinning the EDF’s work was articulated as being driven by rights-based legislation, specifically the UN CRPD, and an accessible EU, given accessibility is a pre-requisite to participation in all domains. This could be further developed. While a number of gaps in the policy focus were articulated, 79% (19 survey respondents) identified that they saw no gaps in the policy agenda.
The EDF is seen by members as more focused on disability specific policies rather than a disability mainstreaming policy focus. There is a concern to drive a higher level and quality of disability mainstreaming in all policy areas at European level.
Progress that has been made by the EDF on the policy agenda might suggest a rebalancing to focus more on issues of policy implementation, to reap benefits from policy gains. The EDF has developed models of member supports, to enable their work on implementing the UN CRPD and the SDGs, which could be built on.
Human rights predominantly frame the vision and policy focus of the EDF. This could be integrated with a focus on achieving full equality in practice, concerned with equal outcomes for persons with disabilities in access to and distribution of resources, power and influence, status and standing, and respect and solidarity.
The EDF is effective and persistent at following the policy trail as it opens up from an initial breakthrough. This is important for impact. While the EDF is viewed as positive to new thinking and the secretariat is valued for its ability to spot new opportunities, the capacity to imagine and develop new policy ideas needs to be sustained and worked on. The challenge to work out the next ‘big ask’ is noted.
The EDF evidences strong and positive relationships with and channels into key European institutions, particular the European Commission and the European Parliament. Engagement with the Council is more difficult.
Recommendations: The EDF should:
- Develop a statement of vision and core values in the lead up to the preparation of its next strategic framework to guide its development and implementation.
- Assess current models for and implementation of disability mainstreaming in European and Member State policy making with a view to making recommendations and pointing to new approaches as found necessary.
- Develop a strategy for supporting policy implementation through member support, engagement with European level professional bodies, strategic litigation, and promoting models for equality-competent institutions.
- Explore the potential for and implications of an integrated equality and human rights foundation for policy work and communication by the EDF.
- Document and evolve systems for enabling new thinking and the formulation of new agendas for equality and human rights of persons with disabilities.
- Document current approaches with a written procedure for the prioritising and the preparation of policy proposals
Conclusions for Strengthening the Movement
Information provision on developments at global, European, and national levels (42%, 10 survey respondents) and support for national advocacy, in particular on the UN CRPD and the SDGs (33%, 8 survey respondents) are seen as key benefits from EDF membership. 64%, 14 survey respondents, felt effectiveness in this area could be improved.
Capacity building activities are being progressed by the EDF are valued. These range from workshops, training events, toolkits, publications, webinars to visits to members. However, the priority in the work on strengthening the movement is seen as information provision to members rather than skills development. This balance is a concern for members with limited resources.
Recommendations: The EDF should:
- Examine the potential to build on the model developed for work on the UN CRPD and the SDGs for a broader capacity building programme for members.
- Work with European level networks to further enable links by its member with national level equality and human rights infrastructure such as equality bodies, national human rights institutions, and ombudsman offices
Conclusions for Communication
The EDF’s communication objectives are to build public support, secure stakeholder engagement, and support persons with disabilities to know their rights. Communication with members is regular and accessible. External communication has the EDF present on all relevant platforms. There are valuable themes evident in the communication of access, participation, opportunity, and removal of barriers.
Public campaigns (37.5%, 9 survey respondents) and conferences or events (37.5%, 9 survey respondents) were seen as most important in building public support. Conferences (30%, 7 survey respondents), advocacy (30%, 7 survey respondents), social media (22%, 5 survey respondents) and partnership actions (22%, 5 survey respondents) were noted as most important in securing stakeholder engagement; and publications (62%, 15 survey respondents), internal newsletter (50%, 12 survey respondents), and training (37.5%, 9 survey respondents) were seen as most important in supporting persons with disabilities to know their rights.
The core purposes ascribed to communication work are to: transmit what the EDF does to persons with disabilities beyond EDF members; deliver information; address misconceptions of the importance of the EU; and source material and tools to put at the disposition of EDF members and persons with disabilities. Communication is seen as being somewhat limited to a role of giving visibility to policy work, getting feedback on it, or amplifying it.
There is evolving thinking on the potential for communication within the EDF. Values-led strategic communication might be an important tool for the EDF in its work for change. This could be useful in a context where populist discourse is influencing politics and hate speech is increasing. It would require a further evolution in the values identified as core for the EDF for this to happen effectively.
Recommendations: The EDF should:
- Ascribe a broader role to its communications work and reflect this in a communications strategy that looks beyond deliverables, to establish a values-led approach, identify messaging and messaging development processes, and tailoring channels of communication to its various audiences.
Conclusions for International Cooperation
Engaging with the UN policy initiatives (65%, 15 survey respondents), building partnerships on the SDGs (61%, 14 survey respondents), and cooperation with and contributing to the global and regional disability movement (48%, 11 survey respondents) were identified as most important in international cooperation work. This area of work is growing. There is an international cooperation strategy agreed. Additional funding has been secured for this which protects other EDF priorities.
The involvement in and contribution of the EDF to the International Disability Alliance (IDA) is valued. Cross fertilisation of different regional networks has progressed, in terms of shared learning and mutual support, and the IDA programme identifies the need to make further progress on this work. There is a need to create the conditions for this, particularly funding.
Recommendations: The EDF should:
- Support the IDA in creating the conditions necessary for further advancing the work of cross fertilisation between the regional networks within the IDA, in particular access to funding, and offer a strong partnership in the development and implementation of cross-fertilisation initiatives.
Impact is the key evaluation criterion for EDF results. Impact refers to the extent to which the EDF has realised its objectives in relation to: influencing policy; strengthening the movement; communication; and international cooperation. The indicators for impact are:
- The level of take-up of policy recommendations brought forward by the EDF in EU polices and policy documents.
- The demonstrated capacity of national members to engage with the transposition and implementation of EU Directives and policies.
- The level of member demand for and take up of EDF publications.
- The representativeness and influence of the voice of persons with disabilities advanced at a global level.
The EDF can point to policy impact in influencing: the Accessibility Act; the Audio-Visual Media Services Directive with a clause on disability access; the Web Accessibility Act; the Marrakesh Treaty being brought into European Corporate Law with changes to the Copyright Directive so works can be made accessible without being subject to copyright; and the European Electronics Communication Code. UN CRPD implementation by the EU was enhanced by EDF engagement.
Members are supported to operate to an agreed standard with national councils currently being reviewed against this EDF standard. Members identify an impact from EDF membership on the quality of their campaign and advocacy work at national level (33%, 8 survey respondents) and improved engagement with UN CRPD reporting (12.5%, 3 survey respondents).
Members are well informed with improvements in internal communication noted. There is a good demand for EDF publications evident.
A strong movement at global level is evident in taking on and impacting on the UN CRPD and on the SDGs, with the EDF playing a key role as part of IDA. Some cross-fertilisation between the regions has taken place with the EDF sharing its experience and expertise.
Efficiency is one key evaluation criterion for EDF management. Efficiency refers to the manner in which inputs are turned into outputs in the work of the EDF. It is focused on the resources, systems and structures of the EDF and, in particular: the management and deployment of resources; the systems that enable a capacity to respond to a changing and evolving context for persons with disabilities; and the structures in place and their capacity to progress action and decision-making for a network organisation. The indicators for efficiency are:
- Planned and agreed financial outcomes achieved in the annual accounts, with a high rate of expenditure and a growth in the reserve fund.
- The establishment and quality of operation of structures and processes that enable member participation in developing new thinking, contributing to policy positions, and engaging in peer support and learning.
The financial outcomes for 2018 achieved a high rate of expenditure and a contribution to the reserve fund. A strong and efficient infrastructure for the management and deployment of EDF resources is evident. Internal systems are well embedded in EDF for member participation, staff management, and planning.
Members report an organisational culture and approach that enables them to contribute viewpoints and ideas, particularly in relation to policy work. The European NGO coordination meetings organised by the EDF are viewed as valuable. There is only a limited tracking of member engagement and it appears some members are unable to make a full contribution, with their own internal resource constraints serving as a barrier. There is a need articulated to facilitate different ways for members to participate that allow room for creativity, informal exchanges, and new thinking.
Staff note a positive culture, good and supportive internal relations, and space to take on responsibility. Secretariat systems have been strengthened with clearer procedures. A strong team dynamic is reported. The EDF has engaged in a full planning cycle. A strategic framework is in place and its successor is planned.
The EDF’s structures and processes are extensive and appropriate to the needs of the comprehensive and inclusive umbrella nature of the organisation. There is a high level of member involvement in these structures. Elected officials enjoy positive member support. However, the need for turnover and succession planning at this level is noted. Action to reinforce the impact of the Youth Committee and the Women’s Committee is suggested.
Recommendations: The EDF should:
- Put in place a policy to limit and restrict the scale of its engagement as partner in EU funded projects where these do not allow for the recruitment of additional staff, to apply outside of international cooperation programmes.
- Strengthen systems for tracking member engagement in the EDF and to better understand and respond to issues for members in this regard.
- Explore the possibility and potential for creating some deliberative spaces to creatively stimulate member dialogue and new thinking.
- Keep the organisation of the policy team under review as the new system beds down, and develop the organisation of communication team systems.
- Reinforce the work of the Youth Committee and the Women’s Committee by developing a position paper to serve as a guide for ongoing work by the EDF in all fields, from each of these perspectives.
Sustainability is another key evaluation criterion for EDF management. This refers to the potential for the work of the EDF to continue efficiently and effectively over the next period, the challenges to this potential, and the elements that underpin this potential. The indicators for sustainability are:
- The infrastructure of personnel in the EDF secretariat and leadership structures and their standing with members and EU institutions.
- The balance achieved in funding between EU grants, member contributions, and other donors.
The standing of the EDF with its broad membership and strong secretariat is viewed as positive for sustainability. The EDF’s positioning, in particular with the EU institutions, underpins sustainability, with strong relationships and channels to engage with the Commission and European Parliament. While financial sustainability is a challenge, member contributions and sources of funding other than the EC core grant make up some 40% of total funding.