Journal of Public Health Advance Access originally published online on October 18, 2005
Journal of Public Health 2005 27(4):331-337; doi:10.1093/pubmed/fdi058
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Evaluating effectiveness of complex interventions aimed at reducing maternal mortality in developing countries
Louise Ross
Louise Ross, Research Assistant, Department of Child Health, University of Aberdeen, Westburn Road, Aberdeen AB25 2ZG, Scotland
Padam Simkhada
Padam Simkhada, Research Fellow, Department of Public Health, University of Aberdeen, Westburn Road, Aberdeen AB25 2ZG, Scotland
W. Cairns S. Smith
W. Cairns S. Smith, Professor and Head of Department of Public Health, Department of Public Health, University of Aberdeen, Westburn Road, Aberdeen AB25 2ZG, Scotland
Address correspondence to Louise Ross. Email: l.ross{at}abdn.ac.uk
Background Reducing the worldwide mortality ratio by 75 per cent between 1990 and 2015 is a key Millennium Development Goal. Randomized controlled trials (RCTs) are the accepted ‘gold standard’ to assess the effectiveness of interventions but they are not always appropriate for practical, ethical or economic reasons in developing countries. This study examines the use of cluster randomized trials and quasi-experimental (nonrandomized) study designs to evaluate complex interventions implemented to reduce maternal mortality.
Methods We systematically searched electronic databases including MEDLINE, EMBASE, CINAHL, BNI, ASSIA, IBSS, CSA and COCHRANE. English language publications between 1990 and 2003 were included. Studies that assessed the effects of complex interventions aimed at reducing maternal mortality in developing countries were included.
Results Four cluster randomized trials and eleven quasi-experimental studies were identified. Two cluster randomized trials examined reduction in prenatal visits with no adverse effects on maternal mortality. Two trials assessed the effects of vitamin A supplementation. Both trials found a significant reduction in maternal mortality ratios after vitamin A supplementation. A decline in maternal deaths was reported in eight of the nonrandomized studies. Measuring maternal mortality was a frequent problem because of insufficient sample sizes and/or poor recording methods. Other limitations include lack of suitable comparison groups and difficulties assessing the effects of confounding factors in the quasi-experimental studies.
Conclusions RCTs may not be appropriate to evaluate complex interventions in maternal mortality and cluster RCTs and quasi-experimental designs may be more suitable. However, further work is required to improve the robustness of such alternative study designs.
Keywords: cluster, maternal mortality, nonrandomized
 |
Introduction |
|---|
TOP
Introduction
Cluster RCTsMaternal mortality is a significant problem in the developing world as a whole, but it is extremely difficult to measure at the local level. Studies of maternal mortality require very large samples for it to be measured accurately. This can be both costly and difficult to implement.1 Evaluation of the effectiveness of complex intervention strategies is also challenging. The randomized controlled trial (RCT) is considered the ‘gold standard’ for evaluation of effectiveness of interventions. However, the RCT, based on random allocation of individuals, is not appropriate for evaluations of complex interventions such as safe motherhood in developing countries, where allocation cannot be conducted on an individual basis. Using cluster randomized controlled trials (cluster RCTs) is one approach but may be difficult to implement for political, economic and technical reasons. An alternative solution is to use quasi-experimental (nonrandomized) study designs.
Many intervention strategies in maternal health programmes are complex in nature with many layers including health care education, information, training, transport funding and/or provision, equipment funding and/or provision and refurbishment or development of existing or new facilities. These multicomponent intervention strategies are a challenge to research designs required to provide robust evidence for policy-makers to utilize.
The purpose of this article is to identify through review, cluster RCTs and quasi-experimental study designs used to evaluate maternal health programmes or intervention strategies in developing countries. Maternal mortality, maternal deaths or case fatality rates are the outcomes of interest. The review was undertaken to inform the development of study design for IMMPACT (Initiative for Maternal Mortality Programme Assessment). This initiative was set up with the overall aim of reducing maternal mortality and severe maternal morbidity in developing countries through improved evidence-based decision-making for safe motherhood.2
|
Cluster RCTs |
|---|
Cluster RCTs involve randomization at group level such as clinic, village or district instead of individual and can be either unrestricted or restricted (using stratification or matching). Cluster randomization requires that both between and within cluster variation be considered. This has implications for sample size and statistical analysis. Thought also needs to be given to the level of the unit of randomization and the level of analysis, i.e. individual or cluster, subgroup analysis and loss to follow-up.3–5
|
Quasi-experimental research designs |
|---|
Quasi-experimental studies can vary widely in design from a simple posttest design with no pretest or control to interrupted time series designs with matching or nonequivalent comparison group. Subjects are not randomly allocated to either group but may be matched for baseline characteristics such as age, geographical location or literacy level, reducing variability between groups. For time series analysis a series of measurements are taken temporally before, during and following the intervention. There may be a comparison group with no intervention or where the intervention may be implemented at a later stage.6–8 Details of the components of quasi-experimental designs are described in framework in Tables 1 and 2. The information provided in these tables is adapted from information provided by Shadish et al.6
|
|
|
Methods |
|---|
Search strategy
Many electronic databases were searched including medical literature (MEDLINE, EMBASE, POPLINE and PUBMED), nursing literature (CINAHL and BRITISH NURSING INDEX), Social sciences literature (Applied Social Sciences Index and Abstracts (ASSIA), Cambridge Scientific Abstracts (CSA), International Bibliography of the Social Sciences (IBSS) and evidence-based medicine (COCHRANE). Bibliographies of selected articles were also searched for relevant studies.
Inclusion criteria
English language publications between 1990 and 2003. Studies were included if based in developing countries and measured maternal death. A second researcher read studies that were considered doubtful.
Definition of maternal or obstetric death used in this review: Death of women in pregnancy during labour and up to 42 days after birth. Outcomes included maternal mortality ratio, case fatality rate and maternal deaths.
Quality assessment
A quality assessment protocol was devised to appraise the nonrandomized studies highlighting particular features of the different types of nonrandomized study design, using a combination of guidelines for appraising studies9 and adding questions designed to elicit information on key features of quasi-experimental research designs obtained from Shadish et al.,6 Cook and Campbell7 and Campbell and Stanley.8 For cluster randomized trials Donner and Klar3 suggested guidelines for the statistical reporting of cluster RCT that were considered in the compilation of the protocol for this review.
Because of the wide variation in types of intervention and measurement of maternal mortality outcomes meta-analysis was not considered appropriate.
|
Results |
|---|
Cluster randomized controlled trials
There were 12 cluster RCT articles retrieved from 237 abstracts, of which four (one being a substudy) met the inclusion criteria after a second reading (Table 3). West et al.10 conducted a double blind trial of two vitamin supplements versus a placebo and was the only study that had sufficient power to measure maternal mortality10 with over 20 000 women followed up. A reduction of 40 per cent in mortality in women taking either vitamin A or beta carotene was reported by West, lowering the maternal mortality ratio from 645 to 385 deaths per 1000 live births. West did not estimate sample sizes to take clustering into account but did use appropriate statistical analysis for cluster RCTs. Christian, a substudy of West (sample size >10 000), estimated maternal mortality in a factorial design as a risk of death with or without night blindness and with or without supplementation.11 Christian reported that night blindness (impairment in vision during pregnancy through vitamin A deficiency) was a risk factor for mortality from infections in pregnancy (RR = 4.99, 95% CI = 2.20–10.58).
|
Two trials, Munjanja12 and Villar,13 evaluated effectiveness of antenatal care programmes and compared maternal deaths between the intervention and control groups. No adverse effects on maternal mortality was found in either study. Both studies were conducted on large samples (>15 000 women) although Munjanja had very few clusters of varying sample size. Only one of the cluster RCTs (Villar) reported using the intracluster correlation coefficient to measure similarities within clusters13. Villar et al. was also the only trial to calculate sample sizes to account for the clustering effect.
Quasi-experimental studies
There were 4581 abstracts identified relating to nonrandomized or quasi-experimental research design. Of these, 28 articles were selected for further examination resulting in 11 quasi-experimental studies meeting the inclusion criteria (Table 4). Only two studies, in Bangladesh, (Fauveau and Ronsmans, a follow up study) used a concurrent comparison group.14,15 A significant association between the implementation of the intervention and a reduction in maternal mortality was found although in Ronsmans direct obstetric mortality also fell in one of the comparison areas. The training programme for midwives in Papua New Guinea16 reported lower mortality rates in all areas, but these were not statistically compared as the two groups were not comparable risk groups. A study on the effect of active prenatal management for pregnant women with sickle cell disease in West Africa17 reported that maternal mortality rates were comparable with the usual rate for the maternity unit. An intervention that included traditional birth attendant education and intervention18 revealed a reduced maternal mortality rate in home deliveries compared to hospital deliveries and national historical data. A series of projects were carried out in Nigeria and Sierra Leone implementing complex interventions that included a variety of components, in different combinations, such as staff training, education, information, transportation, blood transfusion services and upgrading of equipment. Three of the six projects reported a decrease in maternal deaths and case fatality rates,19–21 whereas the others reported fluctuations in the levels of maternal mortality.22–24 All of these projects were carried out on a short (only 6 measurement points) time series basis with no comparison groups. Confounding factors such as political, social and economic unrest and a new better-equipped hospital being set up in the project area may have affected the results in some areas.
|
|
Discussion |
|---|
Main findings
This review identified a small number of articles that used robust cluster RCTs or quasi-experimental designs to evaluate complex intervention strategies to reduce maternal mortality in developing countries. Although most of the studies reported a decline in maternal deaths following implementation of the intervention the findings should be treated with caution because of the inherent biases within the studies, particularly the nonrandomized studies. Variations in the sample sizes, study designs, composition of interventions and reporting of maternal death outcomes made comparability difficult.
What is already known and what this study adds
Many important points can be highlighted regarding the evaluation of complex interventions relating to reduction of maternal mortality. The use of cluster RCTs and quasi-experimental study designs should be considered as useful in programme evaluation, but to ensure validity and robustness some concerns need to be addressed. It may be possible that instead of examining the maternal mortality ratios it might be more useful to look at simple maternal deaths including nonobstetric deaths such as malaria or maternal deaths by obstetric cause or case fatality rates although the latter requires more field testing to improve accuracy.25–27 This action would reduce the need for such large sample sizes and may be more helpful in finding out what actually causes the deaths. More flexible use of the cluster randomized trial could be achieved if combined with quasi-experimental features such as using individual or cluster randomization at the level of intervention or control group along with the introduction of a switching replications design feature described by Shadish et al.6 and also similar to the ‘stepped wedge design’ described by Habicht et al.28 or experimental staged introduction as described by Cook and Campbell.7 A framework describing the strengths and limitations of studies that may be suitable for evaluating public health interventions in developing countries is detailed in Tables 1 and 2.
|
Limitations of this study |
|---|
Searching for quasi-experimental trials was more difficult as many studies did not specify a particular design in their title or methods and some of the specific key words were not recognized under Medical subject headings (MeSH). Some used a combination of designs that incorporated both qualitative and quantitative data (mixed methods) and/or used a variation from the standard nonrandomized or quasi-experimental design as in the shortened time series used in the Safe Motherhood series of 1997.19–24 This made it extremely difficult to classify or categorize the studies following the designs described in Tables 1 and 2. Attempts were made to identify studies in the grey literature through hand searching references but some nonrandomized studies may have been missed.
|
Implications for further research |
|---|
Further research in the combining of cluster randomization and quasi-experimental research methods may provide both the flexibility and the robustness required in the complex area of safe motherhood in developing countries. Other options in providing control groups without randomization or without denying anyone access to a particular service may require a little more consideration but the use of more advanced quasi-experimental designs such as adding switching replications or nonequivalent-dependent variables would lessen the threats to validity that are inherent in nonrandomized designs. Use of the added features that Shadish et al.6 outline enhance not only the validity of the studies but also the flexibility so that policymakers or health care managers appraising the study might be able to assess whether that particular type of maternal health programme could work in his/her area.
Devising guidelines for critically appraising both quasi-experimental studies and cluster RCTs needs to be carried out in order that a standard framework can be set. However, in doing so one must be careful not to lose the necessary flexibility required in the implementation and interpretation of quasi-experimental studies. Adhering to the strict guidelines of any protocol, while appraising the robustness of a particular research method, could result in losing some of the adaptability of the design. If the design does not match up to a set of specific criteria it could be rejected thereby losing some useful evidence.
The lack of evidence regarding intervention strategies to reduce maternal mortality using quasi-experimental research designs only serves to enhance the real need to develop evaluation guidelines for their appraisal. No hierarchal score has been used in categorizing or ranking the designs, as many studies are context dependent. If possible, some form of ranking could possibly be achieved by assessing the number and types of added features. Many quasi-experimental studies are used in situations where it is not practical, ethical or economical to use any other type of study, and it is therefore necessary to make best use of the resources and information available at the time. It is nonetheless vital that quasi-experimental studies be controlled as tightly as possible ensuring that any baseline data collected can be compared satisfactorily with any pre- and postintervention measurements and that sufficient pre-and postintervention measurements are taken to show trends or sustainability of the programme. Areas where there is civil unrest, economic difficulties and poor levels of literacy, features of many of developing countries, pose special challenges in research. This does not mean that validity and reliability can be ignored but that more flexible ideas and attitudes are required about how nonrandomized studies might be both implemented and interpreted within the public health sphere.
|
Conclusions |
|---|
The limitations of these studies make it difficult to draw conclusions. It is possible, however, to conclude that combining different components such as training, education, provision of supplies, upgrading of equipment transport infrastructure, in one programme, may help to reduce maternal mortality. Further research is required in areas of research methodology so that future programmes that may be implemented to reduce maternal mortality are able to be evaluated using robust study designs.
|
Contribution to paper |
|---|
Mrs. Louise Ross was primary author, searching literature and appraising and reviewing papers. Dr. Padam Simkhada reviewed papers and contributed to writing and revision. Prof. W. Cairns S. Smith advised on content, design and contributed to writing and revision.
|
Funding |
|---|
This work was undertaken as part of an international research programme—IMMPACT (Initiative for Maternal Mortality Programme Assessment). See http://www.abdn.ac.uk/immpact/, funded by the Bill and Melinda Gates Foundation, the Department for International Development, the European Commission and USAID.
|
Competing interests |
|---|
The funders have no responsibility for the information provided or views expressed in this article. The views expressed herein are solely those of the authors.
|
Acknowledgements |
|---|
Thanks are due to Jill Mollison, statistician, formerly Department of Public Health, University of Aberdeen for advice on cluster randomization issues.
|
References |
|---|
- Graham W, Brass W, Snow RW. Estimating maternal mortality: the sisterhood method. Stud Fam Plann 1989; 20: 125–135.[CrossRef][ISI][Medline]
- University of Aberdeen. IMMPACT (initiative for maternal mortality programme assessment). Available from http://www.abdn.ac.uk/immpact/ (8 December 2003, date last accessed).
- Donner A, Klar N. Design and analysis of cluster randomisation trials in health research. London: Arnold, 2000.
- Medical Research Council. Cluster randomised trials: methodological and ethical considerations. Available from http://www.mrc.ac.uk/index/publications/pdf-cluster_randomised_trials-link (05 October 2005, date last accessed).
- Campbell MK, Grimshaw JM. Cluster randomised trials: time for improvement. Br Med J 1998; 317: 1171–1172. Available from http://bmj.bmjjournals.com/cgi/content/full/317/7167/1171 (05 October 2005, date last accessed).
[Free Full Text] - Shadish W, Cook T, Campbell D. Experimental and quasi-experimental designs for generalised causal inference. Boston: Houghton Mifflin, 2002.
- Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues. Chicago: Rand McNally, 1979.
- Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally, 1963.
- Crombie IK. The pocket guide to critical appraisal. London: BMJ, 1996.
- West KP, Katz J, Khatry SK et al. Double blind, cluster randomised trial of low dose supplementation with vitamin A or [small beta, Greek] carotene on mortality related to pregnancy in Nepal. Br Med J 1999; 318: 570–575.
[Abstract/Free Full Text] - Christian P, West KP, Khatry SK et al. Night blindness during pregnancy and subsequent mortality among women in Nepal: effects of vitamin A and beta-carotene supplementation. Am J Epidemiol 2000; 152: 542–547.
[Abstract/Free Full Text] - Munjanja SP, Lindmark G, Nystrom L. Randomised controlled trial of a reduced-visits programme of antenatal care in Harare. Lancet 1996; 348: 364–369.[CrossRef][ISI][Medline]
- Villar J, Ba’aqeel H, Piaggio G et al. WHO antenatal care randomised trial for the evaluation of a new model of routine antenatal care. Lancet 2001; 357: 1551–1564.[CrossRef][ISI][Medline]
- Fauveau V, Stewart K, Khan S, Chakraborty J. Effect on mortality of community-based maternity-care programme in rural Bangladesh. Lancet 2003; 338: 1183–1186.
- Ronsmans C, Vanneste AM, Chakraborty J, van Ginneken J. Decline in maternal mortality in Matlab, Bangladesh: a cautionary tale. Lancet 1997; 350: 1810–1814.[CrossRef][ISI][Medline]
- Alto WA, Albu RE, Irabo G. An alternative to unattended delivery – a training programme for village midwives in Papua New Guinea. Social Sci Med 1991; 32: 613–618.
- Rahimy MC, Gangbo A, Adjou R, Deguenon C, Goussanou S, Alihonou E. Effect of active prenatal management on pregnancy outcome in sickle cell disease in an African setting. Blood 2000; 96: 1685–1689.
[Abstract/Free Full Text] - Schaider J, Ngonyani S, Tomlin S, Rydman R, Roberts R. International maternal mortality reduction: outcome of traditional birth attendant education and intervention in Angola. J Med Syst 1999; 23: 99–105.[CrossRef][ISI][Medline]
- Oyesola R, Shehu D, Ikeh A, Maru I. (The Sokoto PMM Team). Improving emergency obstetric care at a state referral hospital, Kebbi State, Nigeria. Int J Gynaecol Obstet 1997; 59: S75–S81.
- Ifenne D, Essien E, Golji N et al. Improving the quality of obstetric care at the teaching hospital, Zaria, Nigeria. Int J Gynaecol Obstet 1997; 59: S37–S46.
- Leigh B, Kandeh H, Kanu M et al. Improving emergency obstetric care at a district hospital, Makeni, Sierra Leone. Int J Gynaecol Obstet 1997; 59: S55–S65.
- Ande B, Chiwuzie J, Akpala W, Oronsaye A, Okojie B. Improving obstetric care at the district hospital, Ekpoma, Nigeria. Int J Gynaecol Obstet 1997; 59: S165–S172.
- Olukoya A, Ogunyemi M, Akitoye C et al. Upgrading obstetric care at a secondary referral hospital, Ogun State, Nigeria. Int J Gynaecol Obstet 1997; 59: S67–S74.
- Olaniron N, Offiong S, Ottong J, Asuquo E, Duke F. Mobilising the community to utilise obstetric services, Cross River State, Nigeria. Int J Gynaecol Obstet 1997; 59: S181–S189.
- WHO. The World Health Report 2005. Make every mother and child count. Available from http://www.who.int/whr/2005/whr2005_en.pdf (05 October 2005, date last accessed).
- Koblinsky M. Safe motherhood indicators: measuring progress. MotherCare Policy Brief 2. Available from http://mothercare.jsi.com/pubs/PolicyBriefs/policy_brief2.htm(05 October 2005, date last accessed).
- WHO/UNICEF. 4. Indicators to monitor maternal health goals. Available from http://www.who.int/reproductive-health/publications/MSM_94_14/MSM_94_14_chapter4.en.html (05 October 2005, date last accessed).
- Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999; 28: 10–18.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||