Assessment of Family Physicians’ Knowledge, Attitudes and Practice towards Pharmacovigilance and Spontaneous Adverse Drug Reactions Reporting during COVID Pandemic

Received: 16-Aug-2021 Accepted Date: Sep 23, 2021 ; Published: 30-Sep-2021, DOI: O

Introduction

World Health Organization (WHO) has defined pharmacovigilance as “the science and activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other medicine-related problem” [1].

Pharmacovigilance is a field related to detecting, evaluating, and preventing Adverse Drug Reactions (ADRs) or drug-related complications and is critically important as it covers the studies including the complete safety profile of medicine [2]. The use of the ADR reporting system is especially important for detecting rare ADRs [3]. In Turkey, the Turkish Center for Monitoring and Evaluation of Adverse Drug Effects (TADMER) was established in 1985 under the Ministry of Health, General Directorate of Pharmaceuticals and Pharmacy. In 1987, Turkey was accepted as the 27^th member of the WHO (World Health Organization) Pharmaceutical Monitoring Collaboration Center. Until 2004, TADMER continued to collect adverse effect reports throughout Turkey and send them to the WHO International Cooperation Center for Drug Monitoring. A pharmacovigilance program under the name of the “Turkish Pharmacovigilance Center” (TUFAM) has been founded in 2005 to direct the evaluation and monitoring of ADRs reporting in Turkey. In collaboration with international organizations, TUFAM reviews worldwide issues regarding drug safety, takes necessary actions, or mediates the necessary actions to be taken [4-6]. Since the publication of the “Regulation on safety of medicines” on April 15, 2014, a structure for collecting data proportional to risk in the postauthorization period, including the concept of additional monitoring for certain drugs by marking with an inverted black triangle, was put into practice. Spontaneous reporting is the form of ADR reporting preferred by health professionals as online. There is an “Adverse drug reaction notification form” available for patients, Healthcare Professionals (HCPs), and pharmaceutical organizations on the website of TUFAM. Although awareness of pharmacovigilance is improved, still insufficient information may reduce the rate of reporting [5-7]. Underreporting of ADRs is a common problem in Turkey and therefore some improvements are needed to further improve ADRs reporting [6,8]. TUFAM has started its educational activities following the directives. They organize training sessions and also most of the universities had integrated pharmacovigilance education into their medical school curriculum [9]. However, the low number of ADRs reporting demonstrates that this training is not enough and that there is a need for widespread and periodical training to increase reporting rates.

Family physicians in our country provide primary healthcare. Each individual has a family physician and is expected to admit first to his physician. Therefore, large numbers of patients frequently visit their family physicians and ask for advice, especially for the prescriptions. Since family physicians examine and follow-up patients for medicinal products, they can detect and follow unwanted effects. This is the first study to determine the knowledge, attitude, and practice of family physicians, which provide primary care in Turkey. In this study, we also determined the effects of factors such as education, working years, age, and gender. Therefore, the present study was aimed to investigate the knowledge, attitude, and practice and to evaluate the factors that can be effective on ADRs reporting and pharmacovigilance in the family physicians in Turkey.

Material and Methods

Study Design

This is a cross-sectional, questionnaire-based study. The number of participants we aimed was 171 as it covers all the family physicians working in the family health centers of the Uskudar district in Istanbul. A sample was not selected as we aimed to reach all populations. We reached 91.8% of the population. The study was approved by the Ethics Committee of the İstanbul Zeynep Kamil Women and Children’s Diseases Training and Research Hospital (EY.FR.22).

Questionnaire Preparation and Validation

Data were collected by using a questionnaire prepared by the investigators based on the findings of recent studies including surveys about ADR reporting [8,10,11]. The first six questions were about the participants’ demographics and the others meant to determine the knowledge, attitude, and practices of family physicians regarding the ADRs’ reporting, several reports during their practice including the COVID pandemic, and the reasons for not reporting ADRs. Before starting data collection, a pre-test was performed on 10 volunteering health care professionals to eliminate/modify unnecessary or unclear questions and the structured questionnaire was finalized.

Data Collection

The survey was applied online by using Google forms and the link was sent by email or phone message explaining the information and the aim of the study. The purpose of the study and the participants’ rights were explained before the participants filled in the questionnaire and they were asked to fill the informed consent form. The participants who did not complete the survey were excluded from the study. The validated questionnaires were responded to by 157 family physicians, and the response rate was 91.8%.

Data Analysis

Responses to the questions in the questionnaire were categorized according to knowledge, attitude, and practice. A scoring was used for the questions of knowledge. There were 9 questions in total and the scoring scale had a maximum of 9 points to a minimum of zero points depending on the right answer. In scoring, we defined a correct answer between 0-3 as low, between 4-6 as moderate, and between 7-9 as good. Descriptive statistics are shown as frequency and percentage. Mann Whitney U and Pearson Chi-Square tests were used for the analysis of the data. SPSS 22.0 statistical package was used for all statistical analyses. The level of statistical significance was set at p<0.05.

Results

The mean age of the participants in the study was 42.4 ± 9.8 (min 24-max 61) years and most of them were between the age of 41-50 years (34.6%). Among them, 59.8% (n=76) were male. The mean duration of time spent in the profession was 18 ± 10.2 (min 1-max 38) years and 44.8% have worked for 21 years or more. A 49.6% of them had training on ADR reporting during their medical education or postgraduate period (Table 1).

Overall responses to the ADR knowledge questions are presented in Table 2. The definition of pharmacovigilance was made correctly by 69.3% of the participants. A 63.8% (n=81) of the physicians defined adverse drug reaction as “observing the harmful effect of the drug after taking a drug in the normal dose”. While only 60.8% of the physicians (n=76) confirmed that all adverse drug reactions should be reported, 28.8% (n=36) stated that only serious reactions should be reported. Only 44.0% (n=55) of the participants answered correctly where to report the adverse drug reactions in Turkey and 54.4% (n=68) of them did not know the available period to report. The majority of participants answered correctly the question of what serious adverse reactions involve (97.6%) and the aim of ADR reporting (84.3%). However, most of the participants did not know (73.2%) how to get ADR form and 55.1% of the participants also had no idea about what the inverted black triangle means.

When we scored their level of knowledge, only 24.4% of the participants got a good score (7-9) (Table 3). The knowledge scores differed according to education, age groups, duration of experience, and gender (p<0.05) (Table 4).

The proportion of female participants having high knowledge scores were lower than male participants (p<0.05). In addition, the lowest knowledge score was in the age group of 51 years and over (p<0.05).

There was no significant difference between the experience year and knowledge score (p>0.05). However, none of the physicians with 31 years of experience or higher had a low knowledge score. It was seen that being non-trained on ADR was associated with low knowledge score levels (p<0.05).

The awareness of physicians is also evaluated and the findings are presented in Table 5. A 78.8% (n=98) of physicians considered reporting adverse drug reactions as a professional obligation. However, 67.7% of the participants (n=86) have never seen “the adverse drug reaction reporting form”. Of all participants, 72% of the physicians (n=90) evaluated the adverse drug reaction as a serious problem. The physicians with a high score and who had training on ADR and ADR reporting had significantly better attitudes on ADR reporting (p<0.05) (Table 6).

We also evaluated the practice of the participants about ADR reporting and we determined that only 13.5% (n=8) have reported ADR formerly, while 46.5% (n=59) of the physicians met with a situation that could be considered as an adverse effect in their professional life. It was observed that among a total of 8 ADR reporting only one report was present during the COVID pandemic (Table 7). Five of these reports were made by the physicians with 21-30 years of professional experience and 3 by those with more than 31 years of experience. The participants who did not report any ADR had a knowledge score range of 0-3, while those in the range of 4-6 had 6 reports, and those in the range of 7-9 had 2 reports. In the COVID-19 pandemic, only one ADR was reported and the reporting was about the drugs; hydroxychloroquine and favipiravir.

The reasons for non-reporting ADR were mostly due to physicians’ being unsure about it (63.8%; n=81), followed by being uncertain of how and where to report (12.9 %, n=16) and not experiencing such a reaction (12.1%, n=15) (Figure 1).

Figure 1. Reasons for not reporting ADR

The majority of the participants (94.4%; n=118) requested training on ADR and pharmacovigilance.

Discussion

ADR reporting is an integral part of patient safety and the main goal of ADR reporting is important to further clarify the risk profiles of drugs in clinical practice, thereby providing information to support the safe and effective use of drugs.

Adequate training of physicians on pharmacovigilance and ADR reporting will positively affect their reporting. In Turkey, in some medical faculties, these subjects are in the education curriculum. In this study, we determined that approximately half of the family physicians participating in the study had training on pharmacovigilance and ADR reporting as mostly during undergraduate education (45.7%) and much less (3.9%) in postgraduate courses. In a study conducted in Ethiopia, it was reported that only 33.3% of the Healthcare Professionals (HCPs) including nurses, health officers, and pharmacists trained on this subject, which is less than our study [10]. However, in a study in Jordan, it was stated that 8.8% of the HCPs including medical doctors and nurses had attended educational courses about pharmacovigilance [11]. It will be helpful to attend these training both during the medical faculty education and to reinforce them later by the courses.

In this study, only 24.4% of the physicians had a good knowledge score. Additionally, 52% of them were at a medium level whereas 23.6% were poor. Similar to our results, in a recent study conducted in a university hospital in Turkey, it was reported that the knowledge of physicians was not sufficient to fulfill the responsibilities assigned to them by the national pharmacovigilance system [12]. Also, in several studies conducted in other countries among HCPs, it was found that the knowledge of pharmacovigilance and ADRs is low [10-15]. These results show the need for wider coverage of ADR and pharmacovigilance issues in faculty education and it should be supported by several courses at regular intervals. The lower level of knowledge may negatively impact the accuracy and the validity of ADR reporting. We found that the participants who achieved a good score among the questions about the ADR knowledge level were significantly higher in the trained group. Several studies conducted on medical students showed a lack of adequate knowledge and skill of reporting ADR emphasizing the great need for integrating pharmacovigilance education in schools’ curriculums to prepare them [16,17]. In the studies, it was also shown that educational intervention improved ADR reporting [18-20]. In this study, 94% of the family physicians who participated in the study were interested to attend training in pharmacovigilance and/or ADR reporting.

The great interest of family physicians in receiving pharmacovigilance training indicates that training on pharmacovigilance and/or ADR reporting should be planned for all family physicians in Istanbul and also across the country. For this, it is necessary to initiate the continuing education program, then to determine the new training and approaches to be made by making intermittent evaluations and examining the spontaneous ADR reporting.

We also analysed the knowledge scores according to gender, age groups, and experience years and we found that males and those aged 51 and over had significantly higher scores. In contrast to our findings, Seid, et al. reported that an adequate level of knowledge towards ADR reporting did not associate with age or sex [10]. However, it was similar to our study in that is not related to the years of experience.

Another important piece of data in this study is that 46.5% of the participants reported that they had encountered a situation that may be an ADR, but only 6.3% reported it. Similar to our results in a study conducted in Turkey it was reported that although 56% of the physicians encountered 1-10 ADRs in a week, only 8% of them reported [12]. In several studies conducted in other countries, the ADR reporting was found also as low, supporting our data [10,13- 15,18]. However, in another study conducted in Turkey, in which higher ADR reporting was noted, 69.6% of doctors encountered ADR and 40.5% reported it [8].

There was no increase in ADR reporting during the COVID pandemic and only one ADR was reported by a family physician. Every individual has registered to a family physician and since the beginning of the pandemic; individuals with PCR test positivity have been followed up by their family physicians over the phone. Primary healthcare providers followed up their patients at certain intervals either by nurses or family physicians. During this period, it is possible that the adverse effects of the drugs used by the patients who were followed at home were shared with their family physicians. In addition, since some drugs that are not effective and safe in COVID in clinical studies are necessarily used in this process, reporting their adverse effects is more important. The ADR reports include not only negative effects but also ineffectiveness notifications [1,7].

The participants who had a knowledge score range of 0-3 had not performed any ADR reporting, while those in the range of 4-6 had 6, and of 7-9 had 2 reports. Supporting our data, in the study of O’Callaghan, et al., it was stated that those who had reported an ADR more than three times in the past had significantly higher knowledge scores [21]. Guner, et al. also noted that ADR reporting was more common in physicians who have prior knowledge of pharmacovigilance, read literature on ADRs, or prevention of ADRs [8]. In another study conducted in Ireland, the majority of HCPs were aware that ADRs could be reported directly to the Regulatory Authority in Ireland and more than half of the hospital doctors had reported one or more ADR [22]. Also, in a study conducted among stakeholders, it was reported that the most proposed reason for not reporting ADR was poor knowledge or understanding of ADR reporting. It is stated that the correct understanding of the ADR definition is the most important factor for reporting and ADR training via face-to-face workshops which shows that practice and applications are preferred by physicians [6]. The practice for ADR reporting should be included more in the training programs. Educational interventions in pharmacovigilance may also increase the spontaneous reporting of ADRs. In a study made by Gonzalez, et al. ADR reporting was increased by 65.4% in the educational intervention group [23]. The ADR reporting rate in the intervention group has been raised from 28.1 to 39.6 (per 1,000 physicians/year) following the intervention [23].

In addition to scoring the knowledge, we also evaluated the responses to the questions about the knowledge. The definition of pharmacovigilance was made correctly by 69.3% of the physicians, whereas it was 63.8% for the ADR definition. Similarly, in a study conducted in Turkey 74% of the doctors gave the correct response for a description of pharmacovigilance [9]. Another important piece of data of this study was that although 78.8% thought ADR reporting is a professional obligation, 73.2% did not know where to obtain ADR reporting forms and 67.7% had ever seen the ADR form. The participants who had seen ADR form were significantly higher in the group that trained on ADR and ADR reporting. In this study, 55% of the physicians knew that ADRs are reported to Turkey Pharmacology Center. But in a study conducted in Turkey, it was reported that only around 30% of the subjects were aware of the national pharmacovigilance system established in Turkey in which 8% of them had reported one or more ADR [12]. A systematic review including 18 studies reported that 44.4% of the population studied were aware of the existence of the Pharmacovigilance Programme in India [24].

In this study, only 44.9% of the participants knew what the inverted black triangle means. We only asked if the physicians knew what black triangle means, but a question evaluating if the presence/absence of a black triangle alters ADR reporting habits is not asked. However, the very low rate of ADR reporting shows that it is not used in practice and that more training is required emphasizing the importance of inverted triangles for ADR reporting.

Inverted triangle sign is a useful application that provides to identify the drugs subjected to additional monitoring easily. The purpose of adding the inverted triangle is to ensure that additional information is collected as early as possible to further clarify the risk profile of drugs in clinical practice [25]. Additional monitoring is mainly based on the need to increase ADR reporting rates for newly registered drugs whose safety profile is not yet fully defined or drugs with new safety issues that need to be identified. It is stated that drugs on the list of drugs subject to additional monitoring must add an inverted equilateral black triangle symbol to the short product information in the “Regulation on the Safety of Drugs” which was published in Turkey in 2014 [7]. In a study conducted in Australia, it was shown that there was a monthly increase of 0.41 reports per medicine post the black triangle intervention [26]. Another survey study in the United Kingdom, evaluating the significance and implications of the black triangle to ADR reports noted that oncology HCPs had under-reporting of drug-related adverse events and lack of black triangle knowledge [21]. In a study including HCPs, it was reported that awareness of inverted black triangles was at least in nurses and mostly in pharmacists. In parallel, the number of ADR reporting was the lowest in nurses and highest in-hospital pharmacists [17]. We think that the importance and attention to the concept of the black inverted triangle should be highlighted. These should be emphasized in the training programs organized for the medical professionals and also within the scope of pharmacovigilance courses in the medical faculty education programs.

In this study, physicians mostly stated the reasons for not reporting ADR as they are not sure. Besides, to be uncertain of where to report and not experiencing such a reaction is stated as other reasons. Similar to our results, in a study conducted in Turkey, not being sure if it is an ADR was the most common among the reasons for not reporting [8]. In several studies, the common reasons for not reporting ADR were stated as lack of time, not knowing where to report and not knowing where to find the report form, being not sure, what and where to report, difficulty in handling report forms, and lack of awareness of the reporting system requirements [11,15,27-29]. However, similar to our study, “being not sure/uncertainty” was the most common reason for not reporting ADR in studies conducted in several countries [10,11,21]. To overcome this being unsure is the use of scoring systems that help us to assess the causality relation between the medication and the effect.

Scoring and deciding whether there is an ADR or not will increase the rate of reporting by contributing to their confidence. The most commonly used methods for causality assessment of ADRs were Naranjo Probability Scale and the causality assessment system proposed by the World Health Organization Collaborating Centre for International Drug Monitoring, the Uppsala Monitoring Centre (WHO-UMC) [30]. Additionally, there are also more specific scoring systems being developed for different patient groups such as pediatric patients [31].

When the data obtained in our study are considered collectively, it can be suggested that the low number of reporting is due to insufficient practical training on ADR and ADR reporting.

We think that it will be beneficial to give detailed explanations about scoring methods with workshops in a wide range of training on ADR reporting since it will make them more confident to make a decision.

Conclusion

One of the most important outcomes of this study is that ADR reports are very low in number in family physicians and this is related to the level of knowledge. During the pandemic, the drugs that are not effective and safe in clinical studies were used and only one ADR was reported. As it is known, ADRs reporting also includes ineffectiveness reports other than negative effects. To increase the ADR reportings of physicians, training programs are necessary to increase the awareness and knowledge level of physicians to a sufficient level. Pharmacovigilance and ADR reporting should be included more broadly in the education programs of the faculty and also in continuous postgraduate educational programs. In these educational programs, information should also be given about the scoring that physicians can use to be sure of the determination and decision of ADR. In course programs, face-to-face training can be made in small groups for practical application after online theoretical lessons in big groups.

Online counseling centers which may be developed by pharmacovigilance centers can answer the problems faced by physicians in the ADRs reporting and enable a faster solution to increase ADRs reports. Additionally, instead of filling out and submitting a form, functional online systems can be developed.

Strengths and Limitations

This study is important as it is the first study revealing the ADRs reportings of the family physicians and examining the reasons. The findings show that there is an urgent need for educational programs and online systems regarding pharmacovigilance systems.

Limitations: The data were obtained from only one district of Istanbul. Since it has a cross-sectional design, a causality relationship cannot be revealed from these results.

We only asked if they want to train or not. But we did not ask whether they would prefer training online or face-toface. It was seen that the most important reason for not making an ADR notification was being unsure and we think that this may be due to the lack of knowledge about scoring methods. However, questions about whether they have information about scoring methods were not included in the questionnaire.

Declarations

Author Contributions

All authors contributed to the study’s conception and design. Data collection and analysis were performed by Elif Oguz, Hatice Ikiisik, Berna Terzioglu Bebitoglu, Mehmet Akif Sezerol. The first draft of the manuscript was written by Elif Oguz, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Conflicts of Interest

The authors declared no potential conflicts of interest concerning the research, authorship, and/or publication of this article.

Availability of Data and Material

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

References

1. World Health Organization. "Pharmacovigilance: Ensuring the safe use of medicines." No. WHO/EDM/2004.8. World Health Organization, 2004.
2. Beninger, Paul. "Pharmacovigilance: An overview." Clinical Therapeutics, Vol. 40, No. 12, 2018, pp. 1991-2004.
3. Pal, Shanthi N., et al. "WHO strategy for collecting safety data in public health programmes: Complementing spontaneous reporting systems." Drug Safety, Vol. 36, No. 2, 2013, pp. 75-81.
4. Regulation on the Marketing Authorization of Medicinal Products for Human Use, Republic of Turkey Official Gazette, 2005. https://titck.gov.tr/storage/Archive/2019/legislation/1dc73f66-6da6-4353-9d70-47c6898e1962.pdf
5. Khan, Zakir, Yusuf Karataş, and Hazir Rahman. "Adverse drug reactions reporting in Turkey and barriers: An urgent need for pharmacovigilance education." Therapeutic Advances in Drug Safety, Vol. 11, 2020.
6. Ozcan, Gulnihal, et al. "Adverse drug reaction reporting pattern in Turkey: Analysis of the national database in the context of the first pharmacovigilance legislation." Drugs-Real World Outcomes, Vol. 3, No. 1, 2016, pp. 33-43.
7. Regulation on safety of medicines. http://www.resmigazete.gov.tr/eskiler/2014/04/20140415-6.htm
8. Guner, Muberra Devrim, and Perihan Elif Ekmekci. "Healthcare professionals’ pharmacovigilance knowledge and adverse drug reaction reporting behavior and factors determining the reporting rates." Journal of Drug Assessment, Vol. 8, No. 1, 2019, pp. 13-20.
9. Aydinkarahaliloglu, D., et al. "Education and training activities of TUFAM." Drug Safety, Vol. 34, No. 10, 41 Centorian DR, Private Bag 65901, Mairangi Bay, Auckland 1311, New Zealand: ADIS INT LTD, 2011.
10. Seid, Mohammed Assen, et al. "Healthcare professionals’ knowledge, attitude and practice towards Adverse Drug Reaction (ADR) reporting at the health center level in Ethiopia." International Journal of Clinical Pharmacy, Vol. 40, No. 4, 2018, pp. 895-902.
11. Abu Hammour, Khawla, Faris El‐Dahiyat, and Rana Abu Farha. "Health care professionals knowledge and perception of pharmacovigilance in a tertiary care teaching hospital in Amman, Jordan." Journal of Evaluation in Clinical Practice, Vol. 23, No. 3, 2017, pp. 608-13.
12. Ergun, Yusuf, et al. "Knowledge attitude and practice of Turkish health professionals towards pharmacovigilance in a university hospital." International Health, Vol. 11, No. 3, 2019, pp. 177-84.
13. Riordan, David O., et al. "Stakeholders’ knowledge, attitudes and practices to pharmacovigilance and adverse drug reaction reporting in clinical trials: A mixed methods study." European Journal of Clinical Pharmacology, Vol. 76, No. 10, 2020, pp. 1363-72.
14. Stoynova, Veselina, et al. "Physicians' knowledge and attitude towards adverse event reporting system and result to intervention-randomized nested trial among Bulgarian physicians." Medical Herald, Vol. 10, No. 2, 2013, pp. 365-72.
15. Adisa, Rasaq, and Tomilayo I. Omitogun. "Awareness, knowledge, attitude and practice of adverse drug reaction reporting among health workers and patients in selected primary healthcare centres in Ibadan, southwestern Nigeria." BMC Health Services Research, Vol. 19, No. 1, 2019, pp. 1-14.
16. Meher, Bikash Ranjan, et al. "A questionnaire based study to assess knowledge, attitude and practice of pharmacovigilance among undergraduate medical students in a Tertiary Care Teaching Hospital of South India." Perspectives in Clinical Research, Vol. 6, No. 4, 2015, pp. 217-21.
17. Alwhaibi, Monira, et al. "Pharmacovigilance in healthcare education: Students’ knowledge, attitude and perception: A cross-sectional study in Saudi Arabia." BMC Medical Education, Vol. 20, No. 1, 2020, pp. 1-7.
18. Ganesan, Subramaniyan, et al. "The impact of the educational intervention on knowledge, attitude, and practice of pharmacovigilance toward adverse drug reactions reporting among health-care professionals in a tertiary care hospital in South India." Journal of Natural Science, Biology, and Medicine, Vol. 8, No. 2, 2017, pp. 203-09.
19. Varallo, Fabiana Rossi, Cleopatra S. Planeta, and Patricia de Carvalho Mastroianni. "Effectiveness of pharmacovigilance: Multifaceted educational intervention related to the knowledge, skills and attitudes of multidisciplinary hospital staff." Clinics, Vol. 72, No. 1, 2017, pp. 51-57.
20. Farha, Rana Abu, et al. "Effect of educational intervention on healthcare providers knowledge and perception towards pharmacovigilance: A tertiary teaching hospital experience." Saudi Pharmaceutical Journal, Vol. 26, No. 5, 2018, pp. 611-16.
21. O’Callaghan, Joan, et al. "Knowledge of adverse drug reaction reporting and the pharmacovigilance of biological medicines: A survey of healthcare professionals in Ireland." BioDrugs, Vol. 32, No. 3, 2018, pp. 267-80.
22. Ergun, Yusuf, et al. "Knowledge attitude and practice of Turkish health professionals towards pharmacovigilance in a university hospital." International Health, Vol. 11, No. 3, 2019, pp. 177-84.
23. Lopez-Gonzalez, Elena, et al. "Effect of an educational intervention to improve adverse drug reaction reporting in physicians: A cluster randomized controlled trial." Drug Safety, Vol. 38, No. 2, 2015, pp. 189-96.
24. Bhagavathula, Akshaya Srikanth, et al. "Health professionals' knowledge, attitudes and practices about pharmacovigilance in India: A systematic review and meta-analysis." PloS One, Vol. 11, No. 3, 2016, p. e0152221.
25. Black Triangle Scheme. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/396808/Black_Triangle_Scheme-new_medicines_and_vaccines_subject_to_EU-wide_additional_monitoring.pdf
26. Li, Raymond, et al. "Effect of the black triangle scheme and its online educational campaign on the quantity and quality of adverse drug event reporting in Australia: A time series analysis." Expert Opinion on Drug Safety, Vol. 19, No. 6, 2020, pp. 747-53.
27. Thorne, Rebecca J., et al. "Awareness and compliance with pharmacovigilance requirements amongst UK oncology healthcare professionals." Ecancermedicalscience, Vol. 12, 2018.
28. Hohl, Corinne M., et al. "Why clinicians don’t report adverse drug events: Qualitative study." JMIR Public Health and Surveillance, Vol. 4, No. 1, 2018, p. e9282.
29. Hazell, Lorna, and Saad AW Shakir. "Under-reporting of adverse drug reactions." Drug Safety, Vol. 29, No. 5, 2006, pp. 385-96.
30. Zaki, Syed. "Adverse drug reaction and causality assessment scales." Lung India, Vol. 28, No. 2, 2011, pp. 152-53.
31. Conroy, Elizabeth J., et al. "A pilot randomised controlled trial to assess the utility of an e-learning package that trains users in adverse drug reaction causality." International Journal of Pharmacy Practice, Vol. 23, No. 6, 2015, pp. 447-55.