Skip Navigation



Journal of Public Health Advance Access published online on May 24, 2007
Journal of Public Health, doi:10.1093/pubmed/fdm028
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
29/3/246    most recent
fdm028v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Zimmern, R. L.
Right arrow Articles by Kroese, M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Zimmern, R. L.
Right arrow Articles by Kroese, M.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2007, Published by Oxford University Press on behalf of Faculty of Public Health. All rights reserved

The evaluation of genetic tests


Ron L. Zimmern, Director
Mark Kroese, Consultant in Public Health Medicine

Public Health Genetics Unit, Strangeways Research Laboratory, Worts Causeway, Cambridge CB1 8RN, UK


Address correspondence to Mark Kroese, E-mail: mark.kroese{at}srl.cam.ac.uk

Scientific advances in genetics and molecular biology have been very successful in advancing our knowledge of biological mechanisms in health and disease, and in catalysing a variety of technological innovations. The number of genetic tests available has consequently increased exponentially over the last few years. Their development has not been accompanied by processes and systems to evaluate these tests in a proper and formal manner to establish their clinical validity and utility. A framework for the evaluation of genetic tests has been developed. This paper reviews the current practice of genetic test evaluation, highlighting the limitations and future challenges in this area of public health.

Keywords: biomarker, evaluation, genetic test


   Introduction
TOP
 Introduction
Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
 Funding
 Conflict of interest
 Contributors
 References
 
The Human Genome Project has been a catalyst for an impressive advance in our knowledge of genetic and molecular science and the development of innovative technologies. This has resulted in the availability of an increasing number of genetic tests. There is now concern that processes and systems are not in place to ensure their proper evaluation so that their results may inform decisions about whether or not they should be implemented in healthcare practice.

Genetic diseases are conventionally regarded as those inherited according to known and accepted patterns of inheritance and for which the risk to family members is high. The term genetic test is less well defined. It is capable of being construed in two different ways: first, as a test for an inherited disorder, and second, as a test based on the analysis of human DNA or other gene-based technologies. For the purposes of this paper, a genetic test is defined as one based on the analysis of human DNA, RNA or chromosomes using a variety of technologies, whether or not the disorder in question is inherited.

We also suggest that it is necessary to distinguish a test from an assay. An assay is a method to analyze or quantify a substance in a sample. For example, serum sodium or creatine phosphokinase measurements are assays as are the raw results of a genetic sequence. The term genetic test should be regarded as shorthand to describe an assay to detect:1

  1. a particular genetic variant (or set of variants),
  2. for a particular disease,
  3. in a particular population and
  4. for a particular purpose.

The purpose of a genetic test is crucial to its evaluation. Its effectiveness is necessarily related to purpose since the effectiveness of an intervention is essentially the extent to which the intervention has achieved its purpose.

Genetic tests may be carried out for a variety of purposes. These include diagnosis, prediction, carrier testing, prenatal testing and screening. In addition, there are pharmacogenetic tests, which identify the presence or absence of a particular genetic variant, which can influence an individual's response to a specific drug.

The result of a genetic test is not only relevant to the individual but can also affect family members. A positive diagnostic genetic test can, depending on the condition, be used in the testing of family members to predict the risk of developing the clinical disorder.

Diagnostic tests of all types, not just molecular genetic tests, are sometimes implemented without adequate appraisal.2–4 Genetic test evaluation methods are still under development but considerable progress has been made.5–9 The major framework for such evaluations is the ACCE model developed in the US.10 The framework takes its name from the four components evaluated: analytical validity, clinical validity, clinical utility and the ethical, legal and social implications (ELSi) of genetic testing.9

The United Kingdom has initiated a process of genetic test evaluation for molecular genetic tests for inherited or heritable diseases provided by the National Health Service (NHS). This is the United Kingdom Genetic Testing Network (UKGTN) Gene Dossier evaluation framework based on the ACCE programme.11 In other countries such as Canada, genetic test evaluation has been considered within the context of Healthcare Technology Assessment (HTA). The great majority of genetic test evaluations undertaken so far have been for rare single gene disorders.


   Disease characteristics
TOP
 Introduction
 Disease characteristics
Genetic test evaluation
 Discussion
 Conclusion
 Funding
 Conflict of interest
 Contributors
 References
 
Certain specific characteristics of gene and disease need to be considered when evaluating genetic tests. These are penetrance, genetic heterogeneity and expressivity. Penetrance is the probability that someone with a disease-associated genotype will develop the disease. It includes a time component and is often described in terms of lifetime penetrance, the risk of getting a disease over, for example, an 80-year period. A woman with a mutation in the BRCA 1 or 2 gene has a lifetime risk of between 60% and 80% of developing breast cancer.

Genetic heterogeneity describes the property that there can be many different genetic causes of the same disease. A particular heritable condition may be caused by more than one gene, locus heterogeneity; or by more than one variant within the gene, allelic heterogeneity. Expressivity describes the situation where people with the same disease-associated genotype manifest different clinical features and with varying severity.


   Genetic test evaluation
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
Discussion
 Conclusion
 Funding
 Conflict of interest
 Contributors
 References
 
The critical first step in genetic test evaluation is to precisely define the exact genetic variants that it is intended to assay, the disorder of interest, the purpose of the test, and the population or healthcare setting in which it is going to be used. Without such express specification, the evaluation will produce results of limited value.

Analytical validity
The analytical validity of a genetic test refers to the assay and defines its ability to measure accurately and reliably the genotype of interest. This part of the evaluation is concerned with assessing test performance in the laboratory as opposed to the clinic. Explicit specification of the genotype of interest is needed because the estimation of analytic validity is both method and mutation specific. The key quantative measures of assay performance for analytical validity are analytical sensitivity and specificity. With DNA-based technologies it is possible to achieve analytical sensitivity and specificity close to 100%.12,13

Quality assurance aims to ensure that test results are reliable and reproducible and usually include internal and external control assessments within a quality management framework. A recent survey on quality assurance of genetic testing services in the European Union revealed that the participation of laboratories in external quality assessment schemes was fragmented and incomplete. Many laboratories did not have any accreditation.14 These results suggest that the analytical validity of a significant proportion of the genetic tests currently provided by molecular laboratories within the European Union cannot be assured.

Clinical validity
Clinical validity defines the ability of a genetic test to detect or predict the presence or absence of the phenotype or clinical disease. Genetic association and other scientific studies may demonstrate a clear association between the presence of certain genetic variants and the disease, but will in itself not be sufficient to serve as a demonstration of clinical validity. A formal evaluation of the sensitivity and specificity of the test, and its positive and negative predictive values will also be required.

An exact characterization of the mutations being tested will also be needed. In many instances, all the main causative mutations will not be known and this will necessarily reduce the sensitivity and, hence, clinical validity of the test even if all known mutations are tested. The key causative mutations for a particular disorder can also vary between different populations. For example, studies of the clinical sensitivity of the American College of Medical Genetics panel of 25 mutations for cystic fibrosis has estimated that the clinical sensitivity of the panel was 71.9% for non- Hispanic Caucasians, 41.6% for African Americans and only 23.4% for Asian Americans.15 The clinical sensitivity of this test was limited by the mutations chosen to be included in the panel for testing. It highlights the importance of knowledge of the frequency of specific genetic variants in a defined population.16

Controls need to be included in any formal test assessment. It is only through the use of such controls that an accurate assessment of specificity can be made. Failure to use controls will critically undermine the performance measures obtained. A genetic test with near perfect analytical performance may still produce false positives (those who are test positive but do not have the clinical disorder) and false negatives (those who are test negative but have the clinical disorder) in the clinical situation.1

The pathogenesis of common complex disorders such as diabetes and hypertension are thought to involve interactions between a number of low penetrance genetic variants with various environmental factors. Outwith a small high penetrance single gene subset of the disease, for example BRCA 1 or 2 in beast cancer, individual gene variants will be of low predictive value. There is considerable debate as to whether testing for these variants, even in combination using technologies such as microarrays, will ever be useful in the clinical situation.17 The current limited understanding of the gene–gene and gene–environment interactions involved in the development of complex disorders continues to be a barrier to the development of valid predisposition or susceptibility tests for these conditions.18

Clinical utility
Clinical utility refers to the likelihood that the test will lead to an improved outcome. This is the conclusive stage of the evaluation and is dependent on but not restricted to the information provided by the other components. The clinical utility of a test should be assessed by reference to an integrated package of care rather than as an isolated investigation and should be directed primarily at the purpose of the test. The issues that should be considered are shown in Table 1.


View this table:
[in this window]
[in a new window]

  		Table 1 Clinical utility

 
It is important to distinguish between whether the genetic test replaces a test already in use, is an additional test to current diagnostics or is an entirely new test. If a genetic test is replacing another test then the assessment of clinical utility must include comparison of cost-effectiveness of the new test and the test already in use.

The assessment of utility will always be critically dependent on perspective. This is why the precise definition of purpose is important. Clinicians, healthcare management organizations and commissioners of healthcare, will focus on evidence of measurable health outcomes for example, improved survival, reduction in complications, and will regard these as the purpose of the test. Patients and their families, in contrast, may interpret purpose very differently. The benefit they will attach to a test result will depend on a range of sociological and cultural factors, and in many instances will be based on their wish to establish a diagnosis even though no effective intervention is possible. It is still uncertain whether such knowledge will itself result in improved health behaviour or outcomes.19-23 The uptake of genetic testing will also be affected by individual perception of risk. These considerations speak for the importance of providing detailed information about the test and how it is to be interpreted, and in some cases formal genetic counselling.

Ethical, legal and social implication of genetic testing (ELSi)
This component of the evaluation is perhaps the most difficult to address as it is wide ranging. There is now some consensus that these issues, which include the impact of test results on insurance and employment, privacy and confidentiality, equity of access, and stigmatization, might best be included as a component within the assessment of clinical utility. Assessing ELSi for new specific tests has been very difficult and most of the work so far has instead concentrated on developing general principles or focusing on the use of genetic tests in screening programmes.10;24-29 Detailed discussion of the ELSi topics is outside the scope of this paper.


   Discussion
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
Conclusion
 Funding
 Conflict of interest
 Contributors
 References
 
The requirement to determine test characteristics in the context of a particular population and for a particular purpose can be generalized to all types of biomarkers and molecular diagnostics. The setting in which the test is carried out is crucial. A test may be highly valid and useful in one clinical context and perform poorly in another, not least because the predictive value of the test will depend on the prevalence of the disease in the test population.

The momentum in the development and provision of genetic tests has not been matched to the same degree in efforts to ensure their proper evaluation. Many tests are being developed in the commercial sector with only prima facie evidence of a relationship between the genetic variants and the target disorder but with little formal evaluation of its clinical validity or utility. This raises serious questions about their effectiveness as a clinical test. In the context of inherited or heritable disorders, the provision of a test result without adequate interpretation of the information, counselling or services to provide follow-up and treatment, may have serious negative impacts on users and their families. Tests of low penetrance genotypes in complex conditions require even greater scrutiny of interpretation, since their predictive value may in fact be far lower than that claimed by the test provider. A further consequence of the failure of proper test evaluation is the opportunity cost that will result from inappropriate diversion of finite healthcare resources from the provision of clinically effective interventions to genetic tests of low clinical utility.

Commercial groups are marketing genetic tests directly to clinicians and the public. This is not limited to individual countries; the internet provides an effective conduit for advertising and access to testing across the globe. Such development of direct to consumer advertising could further increase the use of genetic tests particularly in the United States.30 A recent investigation by the US Government Accountability Office raised serious concerns about the validity of some of the genetic testing services available direct to consumers.31 There is also evidence to show that healthcare professionals feel they need more training in genetics.32

The requirements for evaluating genetic tests used in large populations versus those in very small, well-defined groups are very different. A full, formal evaluation of a genetic test will be time consuming and costly, and there is much to suggest that a less formal process may be used in many instances, particularly when addressing tests of extremely rare high penetrance single gene disorders. We suggest that a priority task should be to devise criteria to determine which tests should undergo full evaluation and which might be adequately assessed less comprehensively.

In the United Kingdom, the UKGTN has developed an infrastructure and process for the evaluation of genetic tests for single gene disorders to inform NHS commissioning decisions. This process is still evolving as the network gains more experience of performing such evaluations. However, we believe that there is lacking within the UK (and other developed countries) platforms to gather the data necessary for the proper evaluation of genetic tests and other biomarkers, processes for their analysis and evaluation, and systems that will act to determine the appropriateness for the clinical use of such tests based on the results of evaluation.


   Conclusion
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
Funding
 Conflict of interest
 Contributors
 References
 
It is important that systems and infrastructure for genetic test evaluation are implemented. Although the provision of genetic tests for rare inherited diseases is of the greatest importance, the use of new molecular diagnostics for the common complex diseases such as cancer, coronary heart disease and diabetes may in time have significantly greater impact on population health. The complexities associated with the interpretation of tests for these common disorders and the technologies involved, will be far greater than for genetic tests used in the diagnosis of high penetrance monogenic disorders. Current gaps in methodology, information needs, policies and systems will need to be addressed before such complex evaluations can be undertaken.


   Funding
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
 Funding
Conflict of interest
 Contributors
 References
 
None.


   Conflict of interest
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
 Funding
 Conflict of interest
Contributors
 References
 
None.


   Contributors
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
 Funding
 Conflict of interest
 Contributors
References
 
The authors were equally involved in the preparation and review of this paper. R.L.Z. is a member of the UKGTN Steering Group and M.K. is public health advisor to the UKGTN.


   References
TOP
 Introduction
 Disease characteristics
 Genetic test evaluation
 Discussion
 Conclusion
 Funding
 Conflict of interest
 Contributors
 References
 

  1. Kroese M, Zimmern RL, Sanderson S. Genetic tests and their evaluation: can we answer the key questions? Genet Med (2004) 6:475–80.[Web of Science][Medline]
  2. Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research. Getting better but still not good. JAMA (1995) 274:645–51.[Abstract/Free Full Text]
  3. Bogardus ST Jr., Concato J, Feinstein AR. Clinical epidemiological quality in molecular genetic research: the need for methodological standards. JAMA (1999) 281:1919–26.[Abstract/Free Full Text]
  4. Nierenberg AA, Feinstein AR. How to evaluate a diagnostic marker test. Lessons from the rise and fall of dexamethasone suppression test. JAMA (1988) 259:1699–702.[Abstract/Free Full Text]
  5. Burke W, Atkins D, Gwinn M, et al. Genetic test evaluation: information needs of clinicians, policy makers, and the public. Am J Epidemiol (2002) 156:311–8.[Abstract/Free Full Text]
  6. Holtzman NA, Watson MS. Promoting safe and effective genetic testing in the United States. Final report of the Task Force on Genetic Testing. (1997) http://www.genome.gov/10001733 (3/01/07, last date accessed).
  7. Sanderson S, Zimmern R, Kroese M, et al. How can the evaluation of genetic tests be enhanced? Lessons learned from the ACCE framework and evaluating genetic tests in the United Kingdom. Genet Med (2005) 7:495–500.[Web of Science][Medline]
  8. Burke W. Genetic testing. N Engl J Med (2002) 347:1867–75.[Free Full Text]
  9. Haddow J, Palomaki G. ACCE: A model process for evaluating data on emerging genetic tests. In: Human Genome Epidemiology—Khoury M, Little J, Burke W, eds. (2004) New York: Oxford University Press. 217–33.
  10. Office of Genomics and Disease Prevention and CDC. A Model System for Collecting, Analyzing and Disseminating Information on Genetic Tests. Office of Genomics and Disease Prevention. http://www.cdc.gov/genomics/gtesting/ACCE.htm (3/01/07, date last accessed).
  11. UK Genetic Testing Network. UKGTN. (2007) http://www.ukgtn.org/index.html (3/01/07, date last accessed).
  12. Palomaki GE, Haddow JE, Bradley LA, et al. Estimated analytic validity of HFE C282Y mutation testing in population screening: the potential value of confirmatory testing. Genet Med (2003) 5:440–3.[Web of Science][Medline]
  13. Palomaki GE, Bradley LA, Richards CS, et al. Analytic validity of cystic fibrosis testing: a preliminary estimate. Genet Med (2003) 5:15–20.[Web of Science][Medline]
  14. Ibarreta D, Elles R, Cassiman JJ, et al. Towards quality assurance and harmonization of genetic testing services in the European Union. Nat Biotechnol (2004) 22:1230–5.[CrossRef][Web of Science][Medline]
  15. Palomaki GE, FitzSimmons SC, Haddow JE. Clinical sensitivity of prenatal screening for cystic fibrosis via CFTR carrier testing in a United States panethnic population. Genet Med (2004) 6:405–14.[Web of Science][Medline]
  16. Watson MS, Cutting GR, Desnick RJ, et al. Cystic fibrosis population carrier screening: 2004 revision of American College of Medical Genetics mutation panel. Genet Med (2004) 6:387–91.[Medline]
  17. Janssens AC, Aulchenko YS, Elefante S, et al. Predictive testing for complex diseases using multiple genes: fact or fiction? Genet Med (2006) 8:395–400.[Web of Science][Medline]
  18. Haga SB, Khoury MJ, Burke W. Genomic profiling to promote a healthy lifestyle: not ready for prime time. Nat Genet (2003) 34:347–50.[CrossRef][Web of Science][Medline]
  19. Marteau TM, Lerman C. Genetic risk behavioural change. BMJ (2001) 322:1056–9.[Free Full Text]
  20. Audrain-McGovern J, Hughes C, Patterson F. Effecting behavior change: awareness of family history. Am J Prev Med (2003) 24:183–9.[CrossRef][Web of Science][Medline]
  21. Bowen DJ, Ludman E, Press N, et al. Achieving utility with family history: colorectal cancer risk. Am J Prev Med (2003) 24:177–82.[CrossRef][Web of Science][Medline]
  22. McClure JB. Are biomarkers useful treatment aids for promoting health behavior change? An empirical review. Am J Prev Med (2002) 22:200–7.[CrossRef][Web of Science][Medline]
  23. Yoon PW, Scheuner MT, Peterson-Oehlke KL, et al. Can family history be used as a tool for public health and preventive medicine? Genet Med (2002) 4:304–10.[Web of Science][Medline]
  24. Godard B, Raeburn S, Pembrey M, et al. Genetic information and testing in insurance and employment: technical, social and ethical issues. Eur J Hum Genet (2003) 11(Suppl 2):S123–S142.[CrossRef][Web of Science][Medline]
  25. European Commission Expert Group on the Ethical, Legal and Social Aspects of Genetic Testing. 25 Recommendations on the Ethical, Legal and Social Implications of Genetic Testing. (2004) Brussels: European Commission.
  26. Burke W, Pinsky LE, Press NA. Categorizing genetic tests to identify their ethical, legal, and social implications. Am J Med Genet (2001) 106:233–40.[CrossRef][Web of Science][Medline]
  27. Surbone A. Ethical implications of genetic testing for breast cancer susceptibility. Crit Rev Oncol Hematol (2001) 40:149–57.[Web of Science][Medline]
  28. Godard B, ten Kate L, Evers-Kiebooms G, et al. Population genetic screening programmes: principles, techniques, practices, and policies. Eur J Hum Genet (2003) 11(Suppl 2):S49–S87.[CrossRef][Web of Science][Medline]
  29. Nuffield Council on Bioethics. Genetic Screening: Ethical Issues. 1–115. (1993) London. Nuffield Council on Bioethics.
  30. Genetic testing for breast and ovarian cancer susceptibility: evaluating direct-to-consumer marketing—Atlanta, Denver, Raleigh-Durham, and Seattle, 2003. MMWR Morb Mortal Wkly Rep (2004) 53:603–6.[Medline]
  31. Gregory Kutz. Nutrigenetic Testing: Tests Purchased from Four Web Sites Mislead Consumers. (2006) United States Government Accountability Office. 1–23. www.gao.gov./cgi-bin/getrpt?GAO-06-977T (3/01/07, date last accessed).
  32. Burton H. Addressing Genetics, Delivering Health. A strategy for advancing the dissemination and application of genetics knowledge throughout our health professions. (2003) Public Health Genetics Unit. http://www.phgu.org.uk/resources/educ_project/addressing_genetics_full_300903.pdf (3/01/07, date last accessed).

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
29/3/246    most recent
fdm028v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Zimmern, R. L.
Right arrow Articles by Kroese, M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Zimmern, R. L.
Right arrow Articles by Kroese, M.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?