Community-based rehabilitation (CBR) has developed in different contexts over the last 20 years (Miles 1993, Thomas & Thomas 1999, 2002). While the CBR movement has been strongly focused on strategy, organization and activities, not enough attention has been placed on the impact of CBR programmes. Changes in and development of CBR have therefore been based to a large extent on experiences, assessments of process and immediate outcomes, as well as the consequences of political shifts rather than on achieved and lasting results. After several years of input, there has recently been increasing demand from both donors and international organizations to measure and document the results of CBR programmes (Powell, Mercer & Harte 2002 , Thomas 2002, Wirz & Thomas 2002). As argued by Turmusani, Vreede and Wirz (2002), there is also a growing ethical issue as to whether CBR should continue in its current many facetted strategy, since documentation of the usefulness and sustainability of this strategy is lacking.

Evans et al. (2001) argued that an essential component of CBR evaluations should be measurable improvements, and that the search for easily applied outcome indicators has not been particularly fruitful. This is supported by Thibeault and Forget (1997) who see this problem as the greatest challenge in CBR evaluations. According to Thomas (2002), there has been little research on either outcomes or the development of indicators with which to measure success.

The purpose of this article is two-fold: firstly, to present findings from a recent impact assessment of the CBR programme in Palestine, and secondly, to use this Palestinian experience to reflect on the impact assessment process with respect to CBR programmes in general.

A Comprehensive Strategy for Change

The joint position paper by International Labor Organization (ILO), United Nations Educational, Scientific and Cultural Organization (UNESCO) and the World Health Organization (WHO) agreed on the following definition: “CBR is a strategy within community development for rehabilitation, equalization of opportunities and social integration of all people with disabilities. CBR is implemented through the combined efforts of disabled people themselves, their families and communities, and the appropriate health, education, vocational and social services (ILO, UNESCO, WHO 1994)”. Subsequent developments have included human rights, democracy and gender perspectives within the ambitions of CBR programmes. Implied within the definition of CBR are targets and potential results on several levels and in diverse areas. Although the individual with a disability is the main target for CBR, solutions to individual problems or the potential for solving these problems are often found within the family (awareness, attitudes and practice), within the local community (awareness, attitudes, practice, adaptations, integrated programmes, etc.) and even at a higher regional or central level (e.g. through laws, regulations and equal rights activities). Comprehensive CBR programmes are complex in design and, by consequence, also in implementation and in the results they produce (Mitchell 1999).

Previous Impact Assessments

While there are several examples of evaluations of CBR programmes around the world, there are few published examples of attempts at measuring the impact of CBR programmes. Studies have nevertheless been reported from Cambodia (Powel, Mercer & Harte 2002) , Botswana (Lundgren-Lindquist & Nordholm 1996), Philippines (Lopez, Lewis & Boldy 2000) and Zimbabwe (Lagerkvist 1992). These studies share a commonality in design characterized by small sample size, the lack of any control for possible alternative influences, and the lack of a control, comparison group. Furthermore, these studies are primarily outcome evaluations and do not qualify as impact assessments as such.

Measuring Impact

Impact is understood as “the effects of an intervention that can be attributed uniquely to it, i.e. with the influence of confounding effects from other sources controlled or removed” (Rossi, Freeman & Lipsey 1999). In social (or community) development programmes, assessment of impact can, however, only be made with some degree of uncertainty. Such programmes do not take place in a social vacuum, and it may be impossible to distinguish the effects of the programme from other influences such as political changes, competing programmes, parallel initiatives, economic changes, or changes in rules and regulations.

Effects of interventions are often assessed by comparing the outcomes for participants versus non-participants, by making repeated measures of participants before, during and after the intervention, or by other methods that attempt to achieve the equivalent of such comparisons (Rossi, Freeman & Lipsey 1999). In essence, all impact assessments are comparative and it is this comparative element that distinguishes impact from outcome assessment (Schalock 2001).

While the common approach to impact assessment is based on the ideal of experimental design, it follows from the character of a community development programme that alternative approaches will be needed. The assessment strategy depends on the context and the circumstances. Impact assessments are of limited value without considering the socio-political-cultural environment that the programme finds itself in. The judgements of experts, programme administrators, key personnel or participants (users, clients) are acceptable alternatives when conditions for applying a classical experimental design are absent (Oakley 1999).

When basing an assessment on information given by people who have some vested interest in the programme or in the results from the assessment, biased statements may be expected. While prior awareness of possible sources of bias helps to curtail the problem, bias can also be avoided through interventions such as inclusion of controls and hypothetical comparisons in the design, clarity in communication with the interviewees, and incorporating rigorous and systematic rules in the data collection process. All of these initiatives have been incorporated in the Palestinian impact assessment.

Applying multiple methods is a central strategy in the assessment in Palestine. According to several authors (Schalock 2001, Sharma 2001, Hartely 2001, Hartley & Muhit 2003) methodological pluralism, i.e. the use of both quantitative and qualitative research methods, and the use of different measurement approaches will yield complementary types of information that contribute to strengthen the foundation from which the conclusions are drawn.

CBR in Palestine – Programme and Context

The CBR programme in Palestine dates back to 1989. Co-ordinated by the Central National Committee for Rehabilitation (CNCR), the main partners for implementing the programme were 23 non-governmental organizations grouped into five Regional Committees (Nablus, Jenin, South Region, Central Region and Gaza) that co-ordinated the work at the regional and community level. At the heart of CBR Palestine is a multi-level and multi-facetted strategy that comprises a variety of activities initiated nationally, regionally and at the local community level. The practical implementation of the programme has been influenced by the dominating ideas about CBR in the 1990s, primarily represented by the World Health Organization and presented through workshops and the WHO manual (Helander, Mendis, Nelson & Goerdt, 1989). Two donor organizations, the Norwegian Association of the Disabled (NAD) and Swedish Diakonia, have been involved from the programme's inception. Following a gradual expansion, the programme today covers approximately 50% of the population in the West Bank and 75% in Gaza. In each area, a project manager and a group of supervisors are responsible for the management of the programme. At ground level, CBR workers (CRWs) are recruited, for the most part from the same geographical area in which they work. Extensive initial training, followed by regular training periods, has focused on the principles of CBR and practical rehabilitation in the communities.

Community surveys have been important as the first step in the establishment of the CBR programmes in Palestine, both at the pilot stage and later during expansions of the programme. The surveys conducted in Palestine covered the whole population in a specified area through house-to-house visits and the simple mapping of the households with the aim of identifying people with disabilities. After completion of the surveys, CRWs used a two-pronged process in each community (Mendis 1996), working at two distinct levels: (i) the community level: stimulating, mobilizing and promoting the community to organize themselves, and (ii) the individual level: bringing about changes in the situation of the individuals with disabilities and their families (households).

The socio-cultural and political context of Palestine is clearly of importance when studying the impact of any social development programme. The people of Palestine have experienced occupation, suppression, restriction of movement, and war-like conditions for many years. The situation has had a strong negative measurable impact on the living conditions in Palestine. Due to economic dependence on Israel, the restrictions imposed on the Palestinian population have contributed collectively to a high level of unemployment, and the consequent growing burdens of poverty and malnutrition (Hawari 2003).

The CBR programme in Palestine was initiated and is run by a broad spectrum of non-governmental organizations (NGOs) throughout the country and backed by committed popular support. CBR is decentralized with more or less autonomous programmes at regional level. While the CRWs at village level are supervised, they still work independently, most often in the village where they live. Overall, this model has turned out to be particularly robust and suitable for the current conditions.

Human suffering increases during times of struggle and suppression – the situation in Palestine is no exception. Many Palestinians have been disabled either directly due to armed confrontations, or due to the harsh living conditions. The status of people with disabilities in Palestine has been influenced by the fact that many of those disabled obtain the status of heroes in the fight against suppression. This has had positive bearings on the attitudes towards all individuals with disabilities and contributed to strengthen the support for CBR.

Another aspect is the fact that CBR was introduced 10 years ago in a context with weak or non-existent services to individuals with disabilities. Initiating a new concept and new services under such circumstances is in fact beneficial to the success of the programme, as effects will be almost immediate and tangible. There is no doubt that this has also resulted in support for and commitment to the CBR programme, but also that the programme has managed to maintain this positive climate for more than 10 years.

Impact Assessment – Strategy and Selected Results

A variety of methods were triangulated to measure the impact of CBR Palestine. This article will focus on results from a representative baseline follow-up study and a record audit.

Baseline Follow-Up

From its inception, the CBR in Palestine has maintained a good filing system and followed recommendations outlined in the WHO manual (Helander et al. 1989). It is one of the strengths of this programme that activities and results have been well documented over many years, in a consistent manner. Unfortunately, the instrument used to document progress (the “ladder”) was later revised by Helander, Mendis, Nelson & Goerdt (1999) without this having been reflected in the programme. The documentation of activities in the programme has thus not developed in such a way as to reflect important conceptual developments within the field of disability in recent years.

CBR in Palestine does, however, offer a rare opportunity for the statistical analyses of results. There are data available at the individual and family levels that have been entered and stored in a systematic way and under qualified supervision. The WHO assessment form (the “ladder”) has been completed and updated for all individuals that have received services from the CBR programme.

To obtain a sample of individuals having received services from CBR, systematic random sampling (selecting every 20th file) was carried out in all regions, yielding a sample of 1075 individuals out of 19,840 registered and active CBR users. The sample size (5.5%) was chosen to be large enough to allow for breakdown into groups (region, gender, etc.). A standardized questionnaire was developed in English, including the core information in the assessment form as well as additional questions about impact for each user. This questionnaire was completed by the CRWs in charge of the sampled individuals together with their supervisors who were all fluent in English in addition to Arabic. In principle this combines the original assessment for each of the sampled individuals with a reassessment in the same questionnaire. Completed questionnaires were brought to Jerusalem for data entry.

The assessment form provides a basis for composing impact measures on individual as well as aggregate levels. Statistical analyses were performed using programmes available in the statistical package SPSS for Windows (release 11.0).

Table 1 shows a set of activity of daily living (ADL) assessment variables (activity/impact variables) applied by the CRWs in their work with individuals. The form containing this information is completed during the first assessment (of the individual), and later any improvements in the ADL are noted on the form with the date for the new assessment. Table 1 further shows the proportion of individuals in the sample who have improved and by how many units. Considerable progress (measured in number of steps on the scale (or ladder) from 0 to 2 or 3) is shown in Table 1. Across the 20 measures, impact is measured for 51% of the individuals. This is a “gross” measure that does not control for other types of influence. It is difficult to use this information isolated to assess the effectiveness of the CBR programme due to lack of a standard or a comparison group.

Table 1.

Measured impact (% of n for each activity)

Activity/impact variable Progress (number of steps) Total number of individuals
  0 (%) 1 (%) 2 (%) 3 (%)  
Play like other kids 62.6 23.7 13.8   457
Sitting 51.9 24.7 23.4   292
Standing 42.1 31.3 26.6   287
Walking ten steps 41.4 17.6 10.0 31.0 301
Move inside house 41.9 26.2 31.9   315
Move in the village 34.2 35.6 30.2   349
Feeds him/herself 52.4 29.5 18.1   407
Understands easy instructions 30.2 45.4 24.4   279
Expresses his/her needs 30.0 48.0 22.0   305
Understands movements and communication signs 43.7 36.3 20.0   146
Uses movements and communication signs 45.6 34.9 19.5   179
Read lips 38.3 31.0 30.7   86
Talks 34.2 50.4 15 5   299
Stays clean 34.8 47.2 18.0   401
Uses bathroom 46.0 34.0 20.0   423
Puts on and off clothes 33.8 50.1 16.1   403
Goes to school 49.8 13.0 6.1 31.1 212
Join in family activities 52.1 31.4 16.5   553
Join in community activities 27.0 47.6 25.4   531
Does she/he have an income 43.0 28.3 9.9 18.8 99
Variation in n is because not all questions are relevant for all users.

In the second section of the questionnaire the CBR workers were asked to assess the total impact on the individual users after the first assessment had been made. In the table below, n refers to the total number of cases in the sample.

Table 2 reflects a positive view of the impact of CBR, but with a realistic amount of variation. Perhaps one could have expected a higher proportion with no or little impact, but it must be remembered that the question (assessment of total impact after first assessment) in general and that it is broader than the information captured in Table 1.

Table 2.

Assessed impact (n=580)

Level of impact % of n
No impact 0.0
Little impact 2.5
Some impact 19.2
Much impact 47.1
Major impact 31.2
Total 100.0

The respondents were further asked to assess the impact without CBR. It appears from the results presented in Table 3 that for 64% of the users, there would be little or no progress if it was not for the CBR programme. No one suggested that the programme had made no difference.

Table 3.

Progress without community-based rehabilitation (CBR) (n=580)

Progress % of n
Difficult to say 0.4
No progress 39.0
Little progress 25.2
Some progress, but not as much as with CBR 35.4
Same progress even without CBR 0.0
Total 100.0

Record Audit

A record audit was carried out in order to utilize the information in the case file system more fully. In each of the five regions, 10–12 individuals in an active relationship with CBR were selected by the programme managers (n=57) and the supervisors, following specific criteria to ensure a broad representation of individuals (gender, age, type of disability, including both individuals who had profited from CBR and individuals where CBR had not accomplished much) and to reduce selection bias. After the selection was made, each region was visited by the responsible researcher who carried out structured interviews with the relevant CRWs, supported by their respective supervisors, about each separate case. All supervisors were fluent in English and Arabic and assisted with translation when necessary. A major advantage in delving deeper into a number of individuals together with the responsible CRWs, is that questions may be specified and control questions included during the interview. Questions about impact were thus always followed by suggesting alternative explanations and scrutinizing the link between input and impact.

The record audit provided an opportunity to study more closely the understanding and practice of assessment and thus represents a quality control and a comparison of the data derived from the assessment form (baseline follow-up).

Impact is measured along one or more of the indicators in all but six of the 57 individuals. Two of these six users with no impact concerned so-called “strange behaviour”. This came up in several interviews as individuals that were particularly difficult to handle, and it is very likely that impact for such users is limited to increased awareness among family members and to some extent also in the local community. Although such changes may contribute to improve the situation for the individual and their families, progress is slow and affected by ignorance, superstition and stigma. Efforts to increase awareness and to change negative attitudes and practice surrounding strange behaviours are demanding and will not be reflected in the current impact indicators (the “ladder”).

The CRWs were asked to describe the impact on the individual and the family. They were also asked to assess the impact in more general terms on a scale from “no impact” to “major impact”. For the majority of the individual users, impact was assessed to be “much” or “major”. “No” or “very little” impact was reported for only four individuals, and for about 20% impact was assessed to be “little” or “some” impact. These assessments are of course subjective, but the advantage of an interview is that follow-up questions may be asked and the respondent is helped to qualify or disqualify the information. It is reassuring that this result corresponds closely to the result shown in Table 2.

A crucial point in any assessment of impact is whether the progress has been caused by other factors or processes than the input provided by the CBR programme. The impact of external (to CBR) services is in this regard important. The CRWs were asked what kind of services had been provided to the individual with disabilities or his/her family before being identified by CBR. For approximately one-third of the users (17 of 57), the family/individual had received no assistance. Most of these were discovered during the community survey – or they approached the CBR worker when word of the programme started to spread. In itself, this is an important indication of impact; although Palestine as a whole has not been without relevant services for persons with disabilities and their families, it has been limited and has also suffered from the lack of co-ordination and accessibility. The results from the record audit indicate strongly that many would have remained without any qualified help if the CBR programme did not cover their local community.

Referrals are an important part of the work of the CRWs. It is part of their job to consider existing structures and to refer people with disabilities to the proper services. When this is relevant, as was determined for the majority of the individual users in this audit, the sources of impact on the individual and his/her family become increasingly difficult to identify with certainty. In spite of the limitations in the existing health and rehabilitation services, it is nevertheless evident that many individuals with disabilities are given proper assessment, treatment, training, guidance and various devices that contribute to influence their lives positively. Very often, however, it is likely that the alternative to referral by the CBR worker would have been no contact, i.e. no service at all. It is of course impossible to determine exactly how much credit should be given to the CBR programme and thus the estimate of the magnitude of the impact due to CBR versus other sources remains uncertain. There appears to be good reason to give substantial credit to the CBR programme also for the roles played by professionals, NGOs or other resources that have been activated as a result of the CBR programme.

Discussion

This article has attempted to present some central aspects of an impact assessment of CBR in Palestine. It has not been the intention to provide the blueprint for measuring impact. Rather, the intention has been to present a way of thinking about such assessments and to use the current impact assessment as an example.

The assessment has shown substantial impact of the CBR programme particularly on the individual and family levels. Bearing in mind the uncertainty of such assessments, it can nevertheless be concluded that the work carried out by the CBR workers has had a direct and unique impact even when considering and controlling for alternative explanations.

The CBR programme has been gradually implemented since the upstart period in 1989–90. The programme has been in place long enough for some measure of impact to be expected. It is, on the other hand, problematic that targets on different levels are not clearly stated. There may be good reasons for this, in that social development programmes must be flexible in order to adapt to a changing context and conditions and to learn from experience. Evidently, this problem is lowest at the individual level and greatest at community level. For the assessment this means that to some extent the objectives need to be deducted from the programme and/or from a general understanding of wanted and/or anticipated impact on three levels (individual, family, local community) of CBR. This is a general problem related to community development programmes and is not unique to CBR, and it emphasizes the need for measurable formulations at all levels.

Several precautions have been taken to reduce bias in the data material and to verify the results. Firstly, a combination of methods has been used to study the same phenomena. Furthermore, results have been compared across methods and sources of information. The use of multiple methods contributes to more robust results from the study (Schalock 2001). Moreover, different methods will not only confirm or contradict findings, they will also illuminate new things and thus strengthen the study. Secondly, the literature on impact evaluations states that a comparison condition is required. For ethical and practical purposes, this is often not possible in evaluations of social development programmes. In this case we have attempted to reduce this weakness in the design by including a comparative element in the record audit. This means that the causal link between CBR and the result in question is scrutinized and alternative explanations explored in the interviews. A hypothetical situation is described to the respondents (“If it was not for the CBR, how would the situation for this individual/household have been?”), corresponding to Schalock's (2001) “hypothetical comparison group”. Thirdly, a particular advantage in assessing CBR in Palestine has been the well-kept and systematic file system that records information on individual level progress over time. This yields a basis for pre-/post-change comparisons as described by Schalock (2001).

By following the recommendations in the WHO manual, the programme has at hand a valuable data material that was utilized as shown above. It is evident, however, that these individual level indicators have not been developed for evaluation and research purposes, but rather as a support for the CBR worker in his/her work with the individuals and their families. Later revision of the assessment form (Helander et al. 1999) has improved the quality by refining the variables and expanding the scope somewhat. There is, however, still a considerable potential for improving this assessment form to facilitate future impact assessments. There is a need for indicators with better statistical properties, consequent formulations, mutually exclusive answer categories – and not least formulations that are tested for relevance and validity. Furthermore, content needs to be expanded also to cover social and attitudinal aspects, more focus on practice in the family as well as in the local community. The complexity of CBR programmes and thus evaluations of such programmes do not undermine the need for good indicators. On the contrary, it is argued that due to the complexity it is of high importance that assessments are made applying different methods and data sources, and of course that various types of methods and measures follow the highest standard possible. It is, on the other hand, not suggested that indicators will be sufficient to assess impact. The point is that improved indicators will produce better data and that this is needed in combination with other type of data. CBR impact indicators will comprise elements that are valid across contexts as well as context specific elements. Most likely a set of indicators to be applied across contexts may comprise a set of topics, technical standards for formulations and answer categories, and suggested formulations that need to be tested and adapted.

Although there is great potential for improving this way of measuring impact, both by improving the technical quality and by covering a broader set of phenomena, it is argued that assessing impact in a social development programme is too complex to be captured by quantitative indicators alone. A combination of both quantitative and qualitative methods is recommended. There are, however, different ways of combining methods as well as different methods to combine.

The example given in this article reflects a particular context, and the impact must be understood in light of the particular socio-cultural and political situation. In light of the occupation, oppression and economic hardship experienced by the Palestinian people, it may at first be surprising that the CBR programme not only has survived for years but that it also is dynamic and well-organized and engages a large number of dedicated and voluntary CBR workers. In-depth knowledge about the context and the situation does, on the other hand, strongly indicate that the CBR model applied here is particularly suited in an occupied and suppressed population. No doubt individuals' motivation to engage in the CBR programme can be derived directly from the fight against Israel. The decentralized model and the changing role of individuals with disabilities due to the conflict may contribute to explain the relative success of the programme. An intervention in a context with great demands for services will furthermore have good chances for initial positive impact, while maintaining this momentum may turn out to be more demanding. After 10 years of implementation it seems that this CBR programme has managed to deliver services which are of lasting use for individuals with disabilities, their families and the local communities in which they live.

Conclusion

This assessment has shown considerable impact of CBR in Palestine, in particular at the individual level and the family/household level. Bias and uncertainty in the data material with regards to distinguishing impact of CBR from other sources should, of course, always be taken into consideration. Also, there is no standard for assessing whether more could be expected. It is also concluded that the basis for assessing impact can be improved if this is integrated in CBR programmes from the beginning. This includes development of viable impact indicators and application of multiple methods including analyses of the particular role of the socio-cultural context.