Analysis of COVID-19 Guideline Quality and Change of Recommendations: A Systematic Review

Background. Hundreds of coronavirus disease 2019 (COVID-19) clinical practice guidelines (CPGs) and expert consensus statements have been developed and published since the outbreak of the epidemic. However, these CPGs are of widely variable quality. So, this review is aimed at systematically evaluating the methodological and reporting qualities of COVID-19 CPGs, exploring factors that may influence their quality, and analyzing the change of recommendations in CPGs with evidence published.Methods. We searched five electronic databases and five websites from 1 January to 31 December 2020 to retrieve all COVID-19 CPGs. The assessment of the methodological and reporting qualities of CPGs was performed using the AGREE II instrument and RIGHT checklist. Recommendations and evidence used to make recommendations in the CPGs regarding some treatments for COVID-19 (remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir) were also systematically assessed. And the statistical inference was performed to identify factors associated with the quality of CPGs.Results. We included a total of 92 COVID-19 CPGs developed by 19 countries. Overall, the RIGHT checklist reporting rate of COVID-19 CPGs was 33.0%, and the AGREE II domain score was 30.4%. The overall methodological and reporting qualities of COVID-19 CPGs gradually improved during the year 2020. Factors associated with high methodological and reporting qualities included the evidence-based development process, management of conflicts of interest, and use of established rating systems to assess the quality of evidence and strength of recommendations. The recommendations of only seven (7.6%) CPGs were informed by a systematic review of evidence, and these seven CPGs have relatively high methodological and reporting qualities, in which six of them fully meet the Institute of Medicine (IOM) criteria of guidelines. Besides, a rapid advice CPG developed by the World Health Organization (WHO) of the seven CPGs got the highest overall scores in methodological (72.8%) and reporting qualities (83.8%). Many CPGs covered the same clinical questions (it refers to the clinical questions on the effectiveness of treatments of remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir in COVID-19 patients) and were published by different countries or organizations. Although randomized controlled trials and systematic reviews on the effectiveness of treatments of remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir for patients with COVID-19 have been published, the recommendations on those treatments still varied greatly across COVID-19 CPGs published in different countries or regions, which may suggest that the CPGs do not make sufficient use of the latest evidence.Conclusions. Both the methodological and reporting qualities of COVID-19 CPGs increased over time, but there is still room for further improvement. The lack of effective use of available evidence and management of conflicts of interest were the main reasons for the low quality of the CPGs. The use of formal rating systems for the quality of evidence and strength of recommendations may help to improve the quality of CPGs in the context of the COVID-19 pandemic. During the pandemic, we suggest developing a living guideline of which recommendations are supported by a systematic review for it can facilitate the timely translation of the latest research findings to clinical practice. We also suggest that CPG developers should register the guidelines in a registration platform at the beginning for it can reduce duplication development of guidelines on the same clinical question, increase the transparency of the development process, and promote cooperation among guideline developers all over the world. Since the International Practice Guideline Registry Platform has been created, developers could register guidelines prospectively and internationally on this platform.


Introduction
COVID-19 has become a global pandemic.Up to 31 December 2020, there have been nearly 81.6 million confirmed cases and nearly 1.8 million deaths globally [1].As yet, the WHO Rapid Evidence Appraisal for COVID-19 Therapies (REACT) Working Group has found that compared to patients receiving usual care or placebo, the 28-day allcause mortality was lower in critically ill patients with COVID-19 receiving dexamethasone [2].But the effective therapeutic options for COVID-19 patients are still limited, and those used are mostly supportive or palliative [3].More than 5,000 clinical trials on COVID-19 have been ongoing or completed around the world [4].Clinical practice guidelines (CPGs) that can timely translate the essential results of those studies into clinical practice should be developed and updated to help to respond to the COVID-19 pandemic.
COVID-19 CPGs have been developed by WHO and other developers after the outbreak.Among the published studies on COVID-19, CPGs and consensus statements accounted for one-fifth of the total [5].However, previous studies have shown that CPGs developed early in the COVID-19 pandemic had methodological weaknesses [6][7][8].And when there was no effective treatment against COVID-19, the treatment recommendations between some COVID-19 guidelines are highly consistent [9].
One year has passed since COVID-19 became known worldwide, and the number of COVID-19 CPGs continues to increase.But the methodological quality of some new COVID-19 CPGs, the overall reporting quality of COVID-19 CPGs, the quality change of COVID-19 CPGs in 2020, and the key factors associated with high methodological and reporting qualities of CPGs in the context of a global pandemic remain unknown.Furthermore, randomized controlled trials and systematic reviews on the effectiveness of treatments of remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir for patients with COVID-19 have been published, and the results show that some of the treatments are effective for COVID-19 patients, and some are ineffective [2,.But the consistency of recommendations on those treatments in COVID-19 CPGs remains unknown.Therefore, we aimed to analyze both the methodological and reporting qualities and their change of COVID-19 CPGs published from January 1, 2020, to December 31, 2020, to provide clinicians with a basis for appropriate selection of highest-quality guidelines and to provide guideline developers on how to develop high-quality CPGs during such a public health emergency globally.And we aimed to analyze the consistency of recommendations on treatments of remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir in COVID-19 CPGs and to explore the change of recommendations with evidence published.In other words, we wanted to explore whether the treatment recommendations between COVID-19 guidelines are still highly consistent or not when there were effective treatments against COVID-19.Maximum possible score = 7 strongly agree

Methods
The scaled domain score

Domain 1
Basic information contains the information that need to be reported in the title/subtitle, executive summary, abbreviations and acronyms, and corresponding developer (items 1a-4).

Domain 2
In the background, the guideline authors should give a brief description of the health problem(s), report aim(s) of the guideline and specific objectives, target population, end users, and settings and guideline development groups (items 5-9b).

Domain 3
Evidence is about what should be reported on healthcare questions, systematic reviews, and assessment of the certainty of the body of evidence (items 10a-12).

Domain 4
Recommendations provide how to write recommendations, rationale/explanation for recommendations, and evidence to decision processes (items 13a-15).Domain 5 The process of external review and quality assurance should be reported in the review and quality assurance (items 16-17).

Domain 6
Funding, declaration, and management of interests (items 18a-19b): (1) Funding source(s) and role(s) of the funder (2) Declaration and management of interest Domain 7 Other information (items 20-22): (1) Describe where the guideline, its appendices, and other related documents can be accessed (2) Suggestions for further research (3) Limitations of the guideline Scope and Purpose is concerned with the overall aim of the guideline, the specific health questions, and the target population (items 1-3).

Domain 2
Stakeholder Involvement focuses on the extent to which the guideline was developed by the appropriate stakeholders and represents the views of its intended users (items 4-6).

Domain 3
Rigor of Development relates to the process used to gather and synthesize the evidence and the methods to formulate the recommendations and to update them (items 7-14).Domain 4 Clarity of Presentation deals with the language, structure, and format of the guideline (items 15-17).

Domain 5
Applicability pertains to the likely barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guideline (items 18-21).

Domain 6
Editorial Independence is concerned with the formulation of recommendations not being unduly biased with competing interests (items 22-23).

Inclusion Criteria and Exclusion
Criteria.We included CPGs for the diagnosis, treatment, care, rehabilitation, and management of COVID-19 and their complications.We included both CPGs published in journals and on websites.
The following guidelines were excluded: (1) guidelines exclusively concerning prevention, control of infection, screening, surveillance, or mental health, for most recommendations or measurements in these guidelines are not specific to COVID-19; (2) guidelines on other diseases covering COVID-  4 Health Data Science

Guideline Selection and Data Extraction. Search results
were imported into an EndNote library, and duplicates were identified.Each study was screened by two independent reviewers.The final inclusion of CPGs was agreed on by consensus.Disagreements were solved by consultation with a third author.We developed a purpose-designed spreadsheet for data extraction of the basic characteristics.Extracted characteristics included the following: title, country/organization of publication, month of publication, developer, version (original or updated), evidence-based development, interim CPG (yes/no) (interim guidelines are produced when the available data and information are most certainly incomplete, especially if additional data are anticipated in the near future; they usually have a very focused scope and a short shelf-life [54]), format of publication (published in a journal or on a website), rating system for the quality of evidence and strength of recommendations, methods to reach consensus, funding, conflicts of interest, and fulfilling the definition  The Reporting Items for practice Guidelines in HealThcare (RIGHT) checklist [58], which contains 35 items grouped into seven domains, was used to examine the reporting quality of CPGs.The explanation of the seven domains and 35 items of RIGHT is shown in Table 3.The CPGs were divided into seven pairs of reviewers (Group 1: S Zhao and J Cao; Group 2: Z Wang and Q Shi; Group 3: S Lu and H Zhang; Group 4: H Liu and J Zhang; Group 5: L Wang and X Zhang; Group 6: M Ren and Y Liu; and Group 7: S Wu and Q Guo) for appraisal.Each item was evaluated by both reviewers independently as   Health Data Science either "reported" or "not reported."An example of how to calculate the domain reporting rate (DRR) and the seven overall reporting rates (ORR) for a CPG is shown in Table 4.We also reported the mean DRR and ORR over all CPGs.9 Health Data Science COVID-19 patients) that could prove the effectiveness of those treatments in treating COVID-19 patients.And we thought the evidence could support guidelines to make recommendations consistent.Four reviewers (S Zhao, S Lu, Z Wang, and H Liu) independently accessed the direction of recommendations on those treatments in COVID-19 CPGs.For each treatment or therapy, the direction of recommendations in each CPG was assessed as for if the source recommended the intervention in the treatment of COVID-19 or if the source did not recommend the intervention but declared the intervention could be considered in certain clinical situations.And the direction of recommendations in each CPG was assessed as against if the source did not recommend the intervention (including developers of CPGs did not recommend for or against the intervention due to insufficient evidence, and the source recommended that the intervention should only be used in clinical trials).Then, we also extracted the evidence (randomized controlled trials and systematic reviews) used to make recommendations in the CPGs if available.

Analysis of the
2.6.Statistical Analysis and Quality Control.All reviewers had experience in using AGREE II and RIGHT to evaluate CPGs.The agreement among the AGREE II scores of the reviewers within each domain was measured by the intraclass correlation coefficient (ICC) to assess the reliability.The IBM SPSS Statistics 25.0 software was used to calculate the ICC value.We used Microsoft Excel 2019 to calculate the AGREE II DS and the RIGHT DRR.We compared the effect sizes between the categories of selected dichotomous characteristics (evidence-based process, version (original or updated), interim CPG (yes/no), format of publication (published in a journal or on a website), rating system for the quality of evidence and strength of recommendations, funding, and conflicts of interest) with the mean difference (MD) and 95% confidence interval (CI) using the RevMan 5.3 software.

Search Results
. The literature search yielded 3,840 records.After excluding irrelevant records, 92 CPGs were included.The process of guideline selection is illustrated in Figure 1, and the details of the selected guidelines are reported in Appendix B.

Characteristics of COVID-19 CPGs. The 92 COVID-19
CPGs were published by 19 countries or multinational organizations.Only seven (7.6%) met the IOM definition of CPGs, essentially meaning that their recommendations were informed by a systematic review of evidence (Table 5).In the seven CPGs, three are living guidelines and three are rapid guidelines.
China was the country that developed the most COVID-19 CPGs (n = 29, 31.2%), of which only one (1.1%)met the criteria of IOM [62].Twenty (  We analyzed the CPGs' methodological quality according to different countries or organizations (Figure 2).The AGREE II scores varied greatly across CPGs produced by different countries or organizations.CPGs developed by WHO (n = 6) got the highest OS (51.5%), but the DS of WHO CPGs were highest in only two domains (clarity of presentation, DS = 67:0%; applicability pertains, DS = 47:9%).CPGs developed in Brazil had the highest scores in stakeholder involvement (DS = 53:7%) and rigor of development (DS = 50:7%).CPGs scoring highest in scope and purpose (DS = 54:6%) and editorial independence (DS = 86:1%) were produced by the Netherlands and Turkey, respectively.

The Reporting Quality of COVID-19
CPGs.The mean ORR for all COVID-19 CPGs was 33.0%.The lowest DRR was in the domain "review and quality assurance" with a mean of 13.0%, and the highest was in the domain "basic information" with a mean of 51.4% (Table 7).Ten CPGs had a DRR below 50% in all seven domains.The RIGHT reporting rates varied greatly across CPGs developed by different countries and organizations (Figure 4).
The CPG produced by Brazil had the highest ORR (68.8%), and the CPG developed in Switzerland had the lowest ORR (6.8%).
The overall methodological quality and reporting quality had an increasing trend over time throughout the year 2020 (Figure 6).

Recommendations and Evidence on Some
Treatments.A total of 20 CPGs covered recommendations on remdesivir, 34 covered recommendations on glucocorticoids, 25 covered recommendations on hydroxychloroquine/chloroquine, 12 covered recommendations on interferon, and 21 covered recommendations on lopinavir-ritonavir, which were identified from six countries or organizations (Figure 7).Although randomized controlled trials and systematic reviews on the effectiveness of those treatments have been published, the recommendations still varied greatly across COVID-19 CPGs published in different countries or regions.Take glucocorticoids as an example; in June 2020, the research of the RECOVERY Collaborative Group had proved their effectiveness in the treatment of adult patients with COVID-19 [10], but some CPGs   Health Data Science published in September and October still recommended against glucocorticoids, which may suggest that the CPGs do not make sufficient use of the latest evidence.

Discussion
Our study systematically searched for COVID- ⁎ "Publication of important evidence on these treatments" refers to the direct evidence used in the COVID-19 guidelines that support recommendations on these treatments.And this important evidence could be found in the corresponding numbered references.
⁎⁎ "Publication of guidelines containing recommendations on these treatments by seven countries or organizations" refers to how many guidelines published by the seven countries or organizations in the month that make recommendations about the treatment.So, the number in the or represents number of guidelines which make recommendations for this treatment published by this country or organization during the month.13 Health Data Science interest, and rating systems for the quality of evidence and strength of recommendations were the main factors that impacted the quality of COVID-19 CPGs.
Dagens et al. [6] found that CPGs produced in the early stage of the COVID-19 pandemic had flaws in methodological quality, and no guidelines were based on evidence from systematic reviews.Our previous research [7] also found that both the methodological and reporting qualities of COVID-19 CPGs tended to be low.Luo et al. [9] found that the diagnosis and treatment recommendations between some COVID-19 guidelines are highly consistent when there was no effective treatment against COVID-19.Our results were consistent with some of the findings of these studies.We found that only a small percentage of the COVID-19 CPGs were informed by systematic reviews and used the GRADE approach.The overall quality of the CPGs still needs to be improved, but the methodological and reporting qualities of COVID-19 CPGs tended to improve consistently over time.There was conclusive evidence about some COVID-19 treatments (remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir) [2,, but the consistency of recommendations in the COVID-19 CPGs was not good, which is different from the research results of Luo et al. [9].
Our study has several strengths.We comprehensively searched the literature from literature databases and websites and aimed to include all COVID-19 CPGs published in 2020.Our analysis could thus clearly identify how the quantity and quality of COVID-19 CPGs have changed over the year.In addition, we analyzed why the quality of COVID-19 CPGs was low.We also evaluated both the methodological and reporting qualities of included 92 COVID-19 CPGs.Most importantly, we systematically evaluated the consistency of recommendations and published evidence for the treatment of remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavir-ritonavir in COVID-19 CPGs.To our knowledge, this is the first review of those treatments in COVID-19 CPGs' recommendations produced during 2020.However, our study also had some limitations.Firstly, we excluded guidelines not available in English or Chinese.Secondly, researchers from all over the world have also published hundreds of expert consensus documents of COVID-19, but we did not include these in our analysis.Finally, when we calculated the domain scores of AGREE II and RIGHT of CPGs, we considered all domains to have equal weight, although in practice some domains may be more important than others when determining the overall quality.
Our study provides some important lessons for CPG developers on how to develop evidence-based CPGs during a pandemic such as COVID-19.First, we found a large amount of substantially overlapping CPGs of COVID-19.This means that COVID-19 CPGs which covered the same clinical questions or recommendations were published, and a huge amount of waste of workload funding and other resources could be at least partly prevented by prior coordination and collaboration among developers.The excessive duplication of CPGs which covered the same recommendations has also created confusion for clinicians in the appropriate application of those guidelines.For example, the USA and China both produced dozens of low-quality CPGs for COVID-19.These guidelines covered overlapping clinical questions, but thedirection of recommendations of those CPGs varied, which is confusing for decision-making.One solution to this problem is to promote prospective registration of CPGs on the same platform from the beginning of the development, similar to clinical trials and systematic reviews [68].Registering a CPG can reduce duplication development of guidelines on the same clinical questions, increase the transparency of the development process, and promote cooperation among guideline developers all over the world [69,70].Since the International Practice Guideline Registry Platform has been created, developers could register guidelines prospectively and internationally on this platform (http://www.guidelines-registry.org/?lang=zh_CN).Second, guidelines produced in an emergency should be considered "living" guidelines with a totally transparent development process [6].COVID-19 has rapidly become a disease associated with unbridled uncertainty.To respond to the uncertainties around the disease, CPGs should be updated and strengthened as evidence evolves [3].The Guidelines International Network (GIN) also suggested that guideline recommendations should be clear evidence-based statements that aim to provide guideline users with clear directions for effective delivery of care [71].So, it is necessary to develop a living guideline for it can facilitate the timely translation of the latest research findings to clinical practice.Third, the key messages of most of the guidelines were not fully and transparently reported in the full text.If developers strictly follow rigorous standards for developing trustworthy clinical practice guidelines such as those by the Institute of Medicine (US) Committee [55], the RIGHT checklist [58], and the GIN-McMaster Guideline Development Checklist [71], which are all supported by international guideline organizations, researchers then can better assess the quality of CPGs and users could better understand and apply CPGs.

Conclusion
Both the methodological and reporting qualities of COVID-19 CPGs increased over time, but there is still room for further improvement.The lack of effective use of available evidence and management of conflicts of interest were the main reasons for the low quality of the CPGs.The use of formal rating systems for the quality of evidence and strength of recommendations may help to improve the quality of CPGs in the context of the COVID-19 pandemic.During the pandemic, we suggest developing a living guideline of which recommendations are supported by a systematic review for it can facilitate the timely translation of the latest research findings to clinical practice.We also suggest that CPG developers should register the guidelines in a registration platform at the beginning for it can reduce duplication development of guidelines on the same clinical topic, increase the transparency of the development process, and promote cooperation among guideline developers all over the world.Since the International Practice Guideline Registry Platform has been created, developers could register guidelines prospectively and internationally on this platform.

14
Health Data Science :7% Mean score of Domain 1 = Σ Scaled domain score of each individual guideline ð Þ Total number of guidelines

Figure 1 :
Figure 1: Flow diagram of the identification process for clinical practice guidelines on COVID-19.

Figure 2 :
Figure 2: The AGREE II domain scores and overall scores (OS) of COVID-19 CPGs by developer country or organization.

Figure 3 :
Figure 3: Comparison of methodological quality by selected characteristics.

Figure 5 :
Figure 5: Comparison of reporting quality by selected characteristics.

Figure 6 :
Figure 6: Temporal trends in the methodological and reporting qualities and quantities of COVID-19 clinical practice guidelines (CPGs).

Table 2 :
AGREE II domain score calculation example of three reviewers giving the following scores for Domain 1 (Scope and Purpose) for one guideline.

Table 3 :
Description of the RIGHT domains.

Table 1 :
Description of the AGREE II domains.

Table 4 :
RIGHT calculation example of the domain reporting rate from results by two reviewers in Domain 1 (basic information) for one guideline.

Table 5 :
Clinical practice guidelines (CPGs) that met the Institute of Medicine (IOM) definition or criteria.
* One CPG that was codeveloped by a CDC and a society.

Table 7 :
Examples of the calculation method of a domain score and the overall domain score (overall score (OS)) for six domains of a CPG for AGREE II are shown in Table2.We also reported the mean DS and OS over all CPGs.
[56]domain scores (DS) and overall score (OS) of the AGREE II instrument and the domain reporting rates (DRR) and overall reporting rates (ORR) of the RIGHT checklist over all COVID-19 clinical practice guidelines (CPGs).andcriteriamadeby the IOM in 2011 (yes/no)[55].And this definition is the most authoritative definition of evidencebased clinical practice guidelines available now[56].2.4.Quality Appraisal of Guidelines.The Appraisal of Guidelines for Research & Evaluation II (AGREE II)[57]instrument, which contains 23 items grouped into six main domains, was used to assess the methodological quality of CPGs.The explanation of the six domains and 23 items of AGREE II is shown in Table1.We appraised all COVID-19 CPGs according to the AGREE II User's Manual[57].Eighteen reviewers in six groups of three (Group 1: S Zhao, J Cao, and S Wu; Group 2: Z Wang, Q Shi, and Q Guo; Group 3: S Lu, H Zhang, and L Wang; Group 4: H Liu, J Zhang, and H Lan; Group 5: M Ren, Y Liu, and P Wang; and Group 6: Q Zhou, Q Li, and X Zhang) evaluated all CPGs.The score for each domain (domain score (DS)) was obtained by summing and normalizing all individual item scores in this domain.

system for the quality of evidence and strength of recommendations (reported vs. not reported)
Recommendations and Evidence Used to Make Recommendations.For the six countries or organiza-tions with the most published CPGs, we identified the recommendations regarding remdesivir, glucocorticoids, hydroxychloroquine/chloroquine, interferon, and lopinavirritonavir in COVID-19 CPGs published by the six countries or organizations.There was already high-quality evidence (It refers to the randomized controlled trials and systematic reviews of which the results were based on direct data from

system for the quality of evidence and strength of recommendations (reported vs. not reported)
21.7%) of CPGs reported CPGs was 30.4%.The DS of all domains were below 50% (Table7).Rigor of development had the lowest DS (19.2%), and clarity of presentation had the highest DS (44.7%).
not 67] any system to rate the quality of evidence and strength of recommendations, and 66 (71.7%)CPGs did not report the method of how consensus for recommendations was reached.The highest numbers of COVID-19 CPGs were published in March (n = 19, 20.7%), April (n = 16, 17.4%), and May (n = 9, 9.8%).The characteristics of the included CPGs are provided in Table6.3.3.The Methodological Quality of COVID-19CPGs.The overall agreement among reviewers for the evaluation with the AGREE II instrument was good (ICC = 0:801)[66,67].The mean AGREE II OS over all COVID-19

Management of conflicts of interest (reported vs. not reported)
percentages represent the AGREE II overall score (green) and RIGHT overall reporting rate (red).2. e radius of the circle represents the number of CPGs published each month.3. is radius of the circle represents 5 CPGs.
19CPGs published in English and Chinese and found that the overall quality of the 92 identified COVID-19 CPGs was low.According to our findings, less than one-tenth of the CPGs met the new IOM definition of a CPG.A WHO rapid advice CPG[65]got the highest scores of methodological and reporting qualities of all included CPGs in our study, based on its high scores in methodological (OS = 72:8%) and reporting qualities (ORR = 83:8%).The evidence-based development process, funding, management of conflicts of