A critical appraisal of urolithiasis clinical practice guidelines using the AGREE II instrument

Introduction

Urolithiasis is a common urological disease with prevalence rates ranging from 1% to 20% worldwide (1). Its economic burden is expected to increase by a further $1.24 billion per year by 2030 (2). Calcium oxalate stones account for approximately 70% of kidney stones and are usually relevant to hypercalciuria or hyperoxaluria (3). There is also evidence that the incidence of certain stone types may be increased, such as uric acid stones (4). Accordingly, the recurrence rate of urinary stone is 30–50% within 10 years (5,6). Evidence is also emerging that there may be a link between nephrolithiasis and chronic kidney disease (7).

In accordance with Institute of Medicine, clinical practice guidelines (CPGs) are evidence-based documents that are systematically developed to help practitioners and patients in making decisions regarding appropriate health care or treatment options for specific clinical circumstances (8). The value of CPGs is influenced by methodology during the guideline formulation process (9). However, due to the differences in regional economy, medical level and target population, the quality of CPGs formulated by different organizations varies greatly (10). As CPGs aim at providing high-quality, safe alternative care options, more and more urological organizations recognize the importance of standardization in guideline development. Over the last few decades, numerous urolithiasis CPGs were published by medical associations and governments. As far as we know, no reporting on quality assessment of CPGs pertaining to urolithiasis has been reported.

The AGREE II instrument is an international framework produced in 2009 to evaluate the development of practice guidelines and the quality of reporting (11). It covers six domains, has been translated into different languages, and has been advocated by many health organizations. This study used the AGREE II instrument to appraise the reporting quality of eligible CPGs and identify factors limiting the improvement of urolithiasis CPGs.

Methods

Registration

The systematic review has been registered in the International Prospective Register of Systematic Reviews (PROSPERO; registration number: CRD42021286736, https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=286736).

Search strategy

Systematic reviews were performed in the following sources from January 2009 to July 2022: (I) Medline (via PubMed); (II) four CPG databases: National Institute for Health and Care Excellence (NICE), Agency for Healthcare Research and Quality (AHRQ), Guidelines International Network (GIN), Scottish Intercollegiate Guidelines Network (SIGN); (III) six expert organizations of guideline developers: European Association of Urology (EAU), American Urological Association (AUA), Canadian Urological Association (CUA), Urological Association of Asia (UAA), International Consultation on Urologic Diseases (ICUD), Chinese Urological Association (CUA). The comprehensive search strategy in MEDLINE (via PubMed) combined MeSH terms and free text search (Table S1). Additionally, CPGs published in Chinese were identified by reviews of three Chinese electronic databases: China National Knowledge Infrastructure (CNKI), VIP Database, and WANFANG database. The search strategy in websites of guideline developers used the following keywords: “urolithiasis”, “nephrolithiasis”, “ureterolithiasis”, “calculi”, “stone”, “kidney”, “urinary”, “guideline”, “guidance”, “guide”, “recommendation”, “consensus”, and “suggestion”.

Inclusion and exclusion criteria

Inclusion criteria were as follows: (I) CPGs providing evidence-supported recommendations based on assessment of evidence; (II) CPGs developed by a medical association or an expert group; (III) CPGs containing recommendations regarding the management of urolithiasis; (IV) English or Chinese publications; (V) CPGs published between January 2009 and July 2022. Exclusion criteria were as follows: (I) editorials, letters, systematic reviews, or other literature explaining CPGs; (II) old versions or duplicates of CPGs; (III) CPGs limited to the use of a specific technology; (IV) CPGs published in non-English or non-Chinese languages.

Selection and evaluation of guidelines

All articles from the above-mentioned sources were exported to Endnote X9. After eliminating the duplicates, titles and full texts were independently browsed by two reviewers (B. Zou and Y. Zhou). Any discrepancies regarding eligibility were resolved through discussion with two additional experts (X. Duan, G. Zeng).

The reporting and development quality of each eligible CPGs were evaluated using the AGREE II instrument. This instrument was developed and validated by international guideline developers and researchers (AGREE Next Steps Research Consortium) to evaluate six domains of guideline development, followed by two overall items (11).

In this study, each item was evaluated by using a seven-point scale ranging from 1 (totally disagree) to 7 (totally agree), based on AGREE Reporting Checklist (Table S2). Domain scores (from 0% to 100%) were measured using the formula: (obtained score − minimal possible score)/(maximal possible score − minimal possible score) ×100% (11). These scores were followed by two items: overall assessment (from 1= worst quality to 7= best quality) and whether or not they would recommend the CPG for clinical practice. The domains of rigor of development and applicability were assigned double weight. A guideline with overall score of more than 60% was considered “strongly recommended”, “recommended with modifications” for score between 30% and 60%, and “not recommended” for scores less than 30% (12).

Four well-trained reviewers (B. Zou, Y. Zhou, Z. He and X. Zhou) independently evaluated each item, assigning a score of 1–7 points, and then the domain scores were calculated.

Statistical analysis

Three reviewers (X. Zheng, S. Dong, R. Xu) extracted the general characteristics of CPGs: title, date of publication, origin, version, developer, disease concerned, funding, and target population. Agreement among the four appraisers was evaluated using ICC with a 95% CI. ICC was rated as poor reliability (<0.40), moderate (0.40–0.75), and good (>0.75) (13). Statistical analyses were processed using Office Excel 2019 and SPSS version 25.0 (IBM, Armonk, NY, USA).

Results

Literature search

Totally, 189 references were identified by systematic search of databases and websites. Of these, seven references were excluded because of duplicates, and 146 references were removed due to irrelevance or other reasons after the abstract was being browsed. The remaining 36 references were reviewed in full text, and seventeen references were excluded due to irrelevance or lack of systematic literature search. Ultimately, 19 eligible CPGs were identified for evaluation (Figure 1).

Figure 1 Flowchart of urolithiasis guideline search and selection. CUA, Canadian Urological Association; EAU, European Association of Urology; AUA, American Urological Association; UAA, Urological Association of Asia; NICE, National Institute for Health and Care Excellence; GIN, Guidelines International Network; CPG, clinical practice guideline.

CPG characteristics

Table 1 lists the general characteristics of the 19 included guidelines, all of which were published from 2014 to 2022. Of these guidelines, ten (52.6%) focused on urinary stone, seven (36.8%) focused on renal stone, and two (10.5%) focused on ureteral stone. Five (26.3%) were developed by EAU, six (31.6%) in the USA, two (10.5%) in Canada, and the rest in Asia, United Kingdom, Italy, and international union. The funding sources of twelve guidelines (63.2%) were described in detail while seven (36.8%) were not revealed. The grading system used to assess the quality of evidence differed among the eligible CPGs: seven (36.8%) used University of Oxford Centre for Evidence-Based Medicine Levels of Evidence grading system (OCEBM), five (26.3%) used Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, five (26.3%) used self-modified grading system, two (10.5%) used mixed systems. No CPGs in Chinese met the inclusion criteria.

Quality assessment

Table 2 and Table 3 present the domain scores of all included CPGs and their overall recommendations. The ICC values among four appraisers was 0.806 (95% CI: 0.779–0.831), indicating high agreement. According to overall quality, five (26.3%) CPGs were rated as “strongly recommended” with overall scores >60%, while fourteen (73.7%) were considered as “recommended with modifications”, achieving scores of 30–60%. No guideline was considered as “not recommended” (Table 2).

Domain 1: scope and purpose

This domain addresses the main aims of the guidelines and specific health issues. The scores of scope and purpose were relatively high (69.7%; 54.2–86.1%), with seventeen guidelines (89.5%) scoring above 60%, and all guidelines scoring more than 50%.

Domain 2: stakeholder involvement

This domain cares about the diversity of members in the guideline development team and preferences of the target population. The scores in this domain fluctuated dramatically (44.9%; 19.4–84.7%), with only three guidelines (15.8%) scoring more than 60%. Only two (10.5%) disclosed the involvement of methodologist.

Domain 3: rigor of development

This domain investigates the method used to review the evidence and the process to formulate recommendations. The domain of Rigor of development gained a mean score of 54.2% (34.4–78.1%), with five (26.3%) guidelines scoring above 60%. No guidelines scored below 30%.

Domain 4: clarity of presentation

This domain focuses on clarity of reporting and organization of recommendations. This domain received the highest score (76.8%; 59.7–90.3%) with eighteen (94.7%) guidelines scoring above 60%. The lowest score was 59.7% for dietary treatment guidelines for renal stone from CLU Working Group.

Domain 5: application

This domain is concerned with barriers, facilitators to its application, and advice or tools on how the recommendations can be put into practice. The mean score for Application was 48.5% (30.2–72.9%), among which four guidelines scored more than 60% as they reported tools and resources to facilitate implementation.

Domain 6: editorial independence

This domain addresses whether funding and conflicts of interest are properly documented. The mean score for Editorial independence was 59.6% (16.7–81.3%), with two guidelines (10.5%) given a score <30%. CLU Working Group guidelines for renal stone got the lowest score of 16.7%.

Classification of CPGs quality

Table 4 presents the domain scores among different subgroups. Specifically, in the domain of clarity and presentation, the scores of CPGs that focused on renal stones and ureteral stones were higher than their counterparts that emphasized solely on renal stones or ureteral stones (P<0.01). Furthermore, in the domain of clarity, it was found that CPGs developed by medical specialty societies exhibited a slight increase in comparison to those developed by other organizations (P<0.05). Throughout the development of CPGs, an intriguing phenomenon was observed whereby there was a discernible elevation in AGREE II scores over time, particularly in the domain of Applicability (P<0.05). No significant differences were found with regards to the region of development group or the purpose for recommendation, nor were there any significant disparities observed between the first version and its subsequent updates.

Discussion

AGREE II assessment

This is the first research to adopt the AGREE II instrument to thoroughly assess the methodological quality of urolithiasis guidelines published in the past decade. However, this research was an analysis of the methodological quality of the included guidelines and did not compare and validate specific recommendations in the guidelines. Hence, comparison on the differences and consensus among CPGs in different regions are needed in future studies.

Notably, although this study limited the language of guidelines to English and Chinese, no CPGs in Chinese met the inclusion criteria. All four guidelines written in Chinese that were included in the screening process were rejected because they failed to provide evidence-based recommendations. It has to be admitted that the standard framework for guideline development, such as AGREE II instrument and G-I-N Standards (33), has not been promoted and applied in the field of urolithiasis in China.

This analysis indicated that the domain of Clarity of presentation (76.8%) and the domain of Scope and purpose (69.7%) performed best, describing the objectives, health issues, and target populations. The recommendations of these guidelines were also mainly summarized in a box, in bold, underlined or as flow charts or algorithms. The presentation of recommendations would affect the usage of doctors and patients. A concise presentation would allow users to quickly locate the information they need, while ambiguous guidelines would hinder the promotion and application of guidelines. This may be because guideline developers have focused on the professional aspects and neglected the simplicity and accessibility of recommendations.

The domain of Stakeholder involvement and the domain of Applicability obtained the lowest scores (<50%). Most guidelines failed to involve methodologists and patients in the development of guidelines. We have to emphasize again the irreplaceability of methodologists for this domain and even the entire guideline development. The participation of methodologists can maximize the integrity and standardization of CPGs, while the participation of patients is conducive to the emergence of patient-centered decision-making. As for the domain of Applicability, many CPGs neglected to consider barriers and facilitators to its application and missing the suggestions on how to apply the recommendations. Given the importance of this domain in translating recommendations into clinical practice, guideline developers need to conduct pilot tests prior to publication.

Score on the domain of Editorial independence was not that high (59.6%), with many CPGs lacking an explicit statement about the external funding and its influence on the recommendations of the guideline. Independence of recommendation formation plays an important role in the reliability and transparency of the guideline. All strongly recommended CPGs reported conflicts of interest of the development panels in some extent. Finally, poor performance (54.2%) in the domain of rigor of development is of particular concern as it is regarded as the most important domain for methodological quality of guidelines and affect the development of the final recommendations (34).

Recommendations to improve guideline quality

In certain regions, advanced medical equipment and technology may not be available, leading to a lack of certain treatment options or diagnostic tools. On the other hand, differences in diet and lifestyle may result in a higher incidence of specific types of stones among urolithiasis patients in certain regions, and different subtypes of urolithiasis may require distinct treatment approaches. Therefore, to ensure the applicability and effectiveness of clinical practice guidelines in different regions, it is necessary to consider these regional differences and propose optimal treatment plans that correspond to different medical conditions, thereby facilitating the application of the guidelines.

Evidence-based recommendations are the cornerstone of contemporary guidelines, and we emphasize the need for transparent grading of evidence and recommendations to establish user trust. This research has revealed that numerous guidelines fail to disclose conflicts of interest, which can significantly impact the credibility of the guidelines. Thus, we propose that CPG development groups transparently disclose any potential conflicts of interest during the guideline development process. To ensure successful implementation of guidelines, we recommend prioritizing certain areas and soliciting user feedback to identify any barriers. Additionally, CPG development groups should be familiar with tools such as AGREE II, G-I-N Standards (33), Guidelines 2.0 (35) and incorporate them into the guidelines. Besides, guidelines should be reviewed for quality standards by methodologists before peer review. Journal editors should set higher standards for publication, and only guidelines that satisfy the criteria for development can be published, which will in turn promote the spread of normative frameworks for guideline development. Finally, developers should pay more attention to controversial recommendations in different countries and regions which would lead to higher quality evidence and minimize duplication of effort.

Strengths and limitations of the study

This study had some limitations. Firstly, it focused on assessing the methodological quality of CPGs and did not compare the accuracy and feasibility of specific recommendations. Further research should be conducted to confirm consensus and differences between CPGs. Secondly, selection bias may exist because this study was limited to CPGs published in English or Chinese and excluded literature published in other forms (e.g., books, booklets, or government documents). Therefore, the results of this study may not necessarily apply to all CPGs for urolithiasis published globally. Finally, some supplemental materials may have been ignored in this study; consequently, in some cases, the quality of the guidelines may have been underestimated.

Nonetheless, the findings of this study remained reliable. Our team included clinical specialist with experience in urolithiasis CPGs development. Systematic literature searches were conducted, and significant agreement among reviewers was obtained, which ensured the reliability of our conclusions. Additionally, the AGREE II instrument was used to evaluate the reporting quality of eligible CPGs, which is validated by several international health organizations and applied to the development of CPGs worldwide.

Conclusions

This study aimed to evaluate urolithiasis CPGs in an effort to improve the reporting quality of CPGs and patients outcomes for those suffering from urolithiasis. Based on the AGREE II instrument, most of the included urolithiasis CPGs have a well-defined scope and purpose, as well as adequate clarity of presentation. However, future work is still needed in the domains of rigor of development, editorial independence, applicability, and stakeholder involvement.

Acknowledgments

Funding: This work was financed by grants from the National Natural Science Foundation of China (Nos. 82070721 and 82270798), National Key Research and Development Program (No. 2020YFC2002700), Science and Technology Plan Project of Guangzhou (No. 202102010150 and No. 202002030316).

Footnote

Peer Review File: Available at https://tau.amegroups.com/article/view/10.21037/tau-22-846/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://tau.amegroups.com/article/view/10.21037/tau-22-846/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Raheem OA, Khandwala YS, Sur RL, et al. Burden of Urolithiasis: Trends in Prevalence, Treatments, and Costs. Eur Urol Focus 2017;3:18-26. [Crossref] [PubMed]
2. Antonelli JA, Maalouf NM, Pearle MS, et al. Use of the National Health and Nutrition Examination Survey to calculate the impact of obesity and diabetes on cost and prevalence of urolithiasis in 2030. Eur Urol 2014;66:724-9. [Crossref] [PubMed]
3. Xu H, Zisman AL, Coe FL, et al. Kidney stones: an update on current pharmacological management and future directions. Expert Opin Pharmacother 2013;14:435-47. [Crossref] [PubMed]
4. Chou YH, Su CM, Li CC, et al. Difference in urinary stone components between obese and non-obese patients. Urol Res 2011;39:283-7. [Crossref] [PubMed]
5. Uribarri J, Oh MS, Carroll HJ. The first kidney stone. Ann Intern Med 1989;111:1006-9. [Crossref] [PubMed]
6. Rule AD, Lieske JC, Li X, et al. The ROKS nomogram for predicting a second symptomatic stone episode. J Am Soc Nephrol 2014;25:2878-86. [Crossref] [PubMed]
7. Zhe M, Hang Z. Nephrolithiasis as a risk factor of chronic kidney disease: a meta-analysis of cohort studies with 4,770,691 participants. Urolithiasis 2017;45:441-8. [Crossref] [PubMed]
8. Field MJ, Lohr KN. Clinical Practice Guidelines: Directions for a New Program. Washington: Washington (DC): National Academies Press; 1990.
9. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer 2003;89:S4-8. [Crossref] [PubMed]
10. Jiang M, Liao LY, Liu XQ, et al. Quality Assessment of Clinical Practice Guidelines for Respiratory Diseases in China: A Systematic Appraisal. Chest 2015;148:759-66. [Crossref] [PubMed]
11. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting, and evaluation in health care. Prev Med 2010;51:421-4. [Crossref] [PubMed]
12. Jiang M, Guan WJ, Fang ZF, et al. A Critical Review of the Quality of Cough Clinical Practice Guidelines. Chest 2016;150:777-88. [Crossref] [PubMed]
13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. [Crossref] [PubMed]
14. Qaseem A, Dallas P, Forciea MA, et al. Dietary and pharmacologic management to prevent recurrent nephrolithiasis in adults: a clinical practice guideline from the American College of Physicians. Ann Intern Med 2014;161:659-67. [Crossref] [PubMed]
15. Pearle MS, Goldfarb DS, Assimos DG, et al. Medical management of kidney stones: AUA guideline. J Urol 2014;192:316-24. [Crossref] [PubMed]
16. Assimos D, Krambeck A, Miller NL, et al. Surgical Management of Stones: American Urological Association/Endourological Society Guideline, PART I. J Urol 2016;196:1153-60. [Crossref] [PubMed]
17. Assimos D, Krambeck A, Miller NL, et al. Surgical Management of Stones: American Urological Association/Endourological Society Guideline, PART II. J Urol 2016;196:1161-9. [Crossref] [PubMed]
18. Prezioso D, Strazzullo P, Lotti T, et al. Dietary treatment of urinary risk factors for renal stone formation. A review of CLU Working Group. Arch Ital Urol Androl 2015;87:105-20. [Crossref] [PubMed]
19. Lee JY, Andonian S, Bhojani N, et al. Canadian Urological Association guideline: Management of ureteral calculi - Full-text. Can Urol Assoc J 2021;15:E676-90. [Crossref] [PubMed]
20. Bhojani N, Bjazevic J, Wallace B, et al. UPDATE - Canadian Urological Association guideline: Evaluation and medical management of kidney stones. Can Urol Assoc J 2022;16:175-88. [Crossref] [PubMed]
21. Skolarikos A, Straub M, Knoll T, et al. Metabolic evaluation and recurrence prevention for urinary stone patients: EAU guidelines. Eur Urol 2015;67:750-63. [Crossref] [PubMed]
22. Türk C, Petřík A, Sarica K, et al. EAU Guidelines on Interventional Treatment for Urolithiasis. Eur Urol 2016;69:475-82. [Crossref] [PubMed]
23. Türk C, Petřík A, Sarica K, et al. EAU Guidelines on Diagnosis and Conservative Management of Urolithiasis. Eur Urol 2016;69:468-74. [Crossref] [PubMed]
24. EAU Guidelines on Urolithiasis [database on the Internet]2022. Available online: https://uroweb.org/guidelines/urolithiasis. Accessed: 26 July 2022.
25. EAU Guidelines on Paediatric Urology [database on the Internet]2022. Available online: https://uroweb.org/guidelines/paediatric-urology. Accessed: 26 July 2022.
26. Jung H, Andonian S, Assimos D, et al. Urolithiasis: evaluation, dietary factors, and medical management: an update of the 2014 SIU-ICUD international consultation on stone disease. World J Urol 2017;35:1331-40. [Crossref] [PubMed]
27. NICE Guideline - Renal and ureteric stones: assessment and management: NICE (2019) Renal and ureteric stones: assessment and management. BJU Int 2019;123:220-32. [Crossref] [PubMed]
28. Gambaro G, Croppi E, Coe F, et al. Metabolic diagnosis and medical prevention of calcium nephrolithiasis and its systemic manifestations: a consensus statement. J Nephrol 2016;29:715-34. [Crossref] [PubMed]
29. Vermandere M, Kuijpers T, Burgers JS, et al. alpha-Blockers for uncomplicated ureteric stones: a clinical practice guideline. BJU Int 2018;122:924-31. [Crossref] [PubMed]
30. Eisner BH, Goldfarb DS, Baum MA, et al. Evaluation and Medical Management of Patients with Cystine Nephrolithiasis: A Consensus Statement. J Endourol 2020;34:1103-10. [Crossref] [PubMed]
31. Lee MS, Fenstermaker MA, Naoum EE, et al. Management of Nephrolithiasis in Pregnancy: Multi-Disciplinary Guidelines From an Academic Medical Center. Front Surg 2021;8:796876. [Crossref] [PubMed]
32. Taguchi K, Cho SY, Ng AC, et al. The Urological Association of Asia clinical guideline for urinary stone disease. Int J Urol 2019;26:688-709. [Crossref] [PubMed]
33. Qaseem A, Forland F, Macbeth F, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med 2012;156:525-31. [Crossref] [PubMed]
34. Hoffmann-Eßer W, Siering U, Neugebauer EAM, et al. Guideline appraisal with AGREE II: online survey of the potential influence of AGREE II items on overall assessment of guideline quality and recommendation for use. BMC Health Serv Res 2018;18:143. [Crossref] [PubMed]
35. Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ 2014;186:E123-42. [Crossref] [PubMed]