5 Discussion

5.1 Main findings

• The pediatric pneumococcal vaccination program that was implemented in Iceland in 2011 achieved excellent coverage. Over 95% of all children in the first six vaccine eligible birth-cohorts received two or more doses of PHiD-CV10 before 24 months of age.

• An impact of PHiD-CV10 introduction was detected on several different facets of otitis media incidence. Pediatric primary care visits due to acute otitis media decreased by 21% and emergency department visits to Children’s Hospital Iceland decreased by 12-21%. Hospital-based treatment of otitis media with parenteral ceftriaxone decreased by 42%.

• All-cause outpatient antimicrobial prescriptions decreased by 8% among children younger than three years of age following the introduction of PHiD-CV10. The incidence of AOM-associated antimicrobial prescriptions decreased even further, 21%.

• The incidence of tympanostomy tube placements did not decrease following the introduction of PHiD-CV10, despite a measurable impact on otitis media visits to primary care and pediatric emergency departments and antimicrobial prescriptions. Approximately one-third of all Icelandic children undergo at least one tympanostomy tube procedure before five years of age.

• Pediatric hospital admissions for pneumonia declined by 20% following PHiD-CV10 introduction. Pneumonia hospitalizations were also shown to have decreased among Icelanders 20-39 years of age, implying a strong herd-effect among adults likely to by the parents of young children.

• Invasive pneumococcal disease requiring hospitalization decreased by 93% among vaccine-eligible children younger than three years of age. No vaccine-type invasive disease was diagnosed among the vaccinated cohorts.

• The pediatric pneumococcal vaccination program in Iceland was cost-effective. During the first five years of the program, PHiD-CV10 resulted in a net-savings of 7,463,176$ when only considering averted costs of prevented cases. When work-loss was also considered, the estimated net-savings increased to 8,164,894$.

5.2 Data collection and sources (Papers I-VI)

The data used for the papers in this thesis were collected from several population-based registries. Their quality and scope is extensive.

Three of the registries are maintained by the Icelandic Directorate of Health. Firstly, data on every administered dose of pneumococcal conjugate vaccine were obtained from the National Vaccine Registry. The NVR receives data directly from the electronic health record system. Secondly, information regarding every outpatient antimicrobial prescription was extracted from the National Drug Prescription Registry, which receives electronic data directly from Icelandic pharmacies when each prescription is filled. Compliance is required by law. Finally, primary care visits for respiratory infections were extracted from the Primary Care Registry, which contains records on all visits to primary care physicians in Iceland.

Information on emergency department visits and hospital admissions for respiratory infections were obtained from Landspitali University Hospital’s patient registry. Landspitali University Hospital is the sole tertiary hospital in Iceland, and includes Iceland’s only pediatric hospital. It accounts for 91% of all hospital beds in the country, provides primary and secondary care for 65% of the Icelandic population, and provides tertiary care for the whole population. The patient registry is maintained by administrative staff at Landspitali University Hospital.

Landspitali’s patient registry included a detailed breakdown of costs for each visit and hospitalization. Cost was broken down into categories which included wages of physicians, nurses, and other support staff, as well costs of medications and diagnostic testing. The costs incurred by each department involved in the patient’s care was tabulated. These data were used in the cost-effectiveness analysis.

The reimbursement database of Icelandic Health Insurance provided data on all outpatient otolaryngological procedures. While the reimbursement database is not directly connected to electronic medical records, there is still strong reason to believe that the database is accurate and contains information on all outpatient procedures. Health care in Iceland is a single-payer system with the government guaranteeing equal access for all permanent residents through a single national health insurance, which is funded by taxation. Health care providers are either salaried governmental employees or independent practitioners who work within a framework agreement with Icelandic Health Insurance, and are reimbursed on a per case basis. It is unreasonable to assume that any Icelandic resident would routinely forgo their national health insurance and opt to pay the full cost out-of-pocket. Likewise, independent practitioners have a strong incentive to seek compensation for their labor. The data available from the reimbursement database therefore reflect the total number of procedures.

Immigration and emigration data on children younger than five yeas of age were provided by Statistics Iceland. The data augmented birth-cohort analyses, allowing longitudinal time-to-event analyses to be performed. Children who immigrated to Iceland after birth could be excluded and the follow-up time of children who emigrated could be censored at the time of emigration.

Data from each registry were transferred directly to the Directorate of Health where the data was anonymized. Each record was associated with a unique study identification number that was created from the individual’s national identification number by staff at the Directorate of Health. Because each Icelandic citizen receives one and only one national identification number over the course of their lifetime, this allowed the data from the various registries to be reliably linked. Unique individuals could be tracked between the different registries but could not be identified.

The linkage between several large population-based registries overcame a common limitation of epidemiological data. Registries record information about an event that occurred, e.g. an antimicrobial prescription, a primary care visit or a hospital admission. Because of this, they generally lack information about those individuals who did not experience the event. Epidemiological studies are therefore often forced to either restrict their scope to individuals who experienced one or more events, or infer the number of individuals who did not experience an event. Individual-level data on those who did not experience the event is missing. By linking large concurrent population-based registries, we were able overcome this common constraint by accurately identifying unique individuals who were alive and living in Iceland during the evaluated period.

The data underlying this study were observational in nature, and their quality was enhanced by several factors. The breadth of the data was extensive. The study contains individual-level information on 375,383 Icelandic citizens over a 13 year period from 2005-2017. Six years of data were collected prior to the introduction of PHiD-CV10 into the pediatric vaccination program, which allowed for an analysis of secular trends in the pre-vaccine period. Similarly, the study included five to seven years of post-vaccine data, providing an opportunity to evaluate the immediate and delayed effect of vaccine introduction. According to Statistics Iceland, the aggregate number of Icelandic citizens was 293,577 on 1 January 2005, and was 338,349 on 1 January 2017 – which strongly suggests that our data contains individual-level information on all Icelandic citizens with few exceptions.

Though the data were extracted from the registries after events occurred, they were collected prospectively, and retain the properties of prospectively collected data. Retrospective observational studies identify a population of individuals who have experienced a certain event, and then determine what risk-factors they had prior to experiencing the event. A prospective study identifies a population of interest and follows them over time to ascertain whether they experience a certain event. Prospective data can produce estimates of relative and absolute risk, and relative and absolute rates.

Electronic medical records were consistent during the entire study period. Not only were all medical records in Iceland stored electronically, but the same software, Saga, was used by all health care providers and institutions throughout the study period. Likewise, the International Classification of Diseases, 10th revision (ICD-10), was the only diagnostic coding system in use in Iceland during the study period. Furthermore, all medical procedures were coded with the NOMESCO Classification of Surgical Procedures (NCSP), and drugs were classified using the Anatomical-Therapeutic-Chemical (ATC) classification system of the World Health Organization. Continuity between data systems enhanced the quality of results.

Our data included the date of birth, date of death, as well as data on immigration and emigration of Icelandic children. Through linkage of the registries, individual-level information of those who did not experience an event became available, and accurate time-based at-risk denominators could be constructed. This allowed the study data to be analyzed using survival methods, and repeated events within the same individual taken into account.

Individual-level information was available for each pneumococcal vaccine dose that was administered in Iceland. Unlike prior studies, which were forced to infer the proportion of individuals who were vaccinated using aggregate sales data, we were able to directly check if a person had been vaccinated. We were also able to confirm that few children had been vaccinated in Iceland before the introduction of PHiD-CV10 into the Icelandic pediatric vaccination program. Among birth-cohorts 2005-2009, the proportion of children who received two or more doses of a pneumococcal conjugate vaccine before two years of age was below 3%. Of the non-vaccine eligible birth-cohorts, only the 2010 birth-cohort saw an increase in PCV uptake, especially among children born in the latter half of 2010. This is likely due to heightened awareness among parents because of the impending introduction of the vaccine. In some sense the result is that an unplanned catch-up occurred, with close to 40% of children born in the latter half of 2010 receiving two or more doses of a pneumococcal conjugate vaccine, before two years of age.

5.3 Epidemiology and management of otitis media in Iceland and the impact of PHiD-CV10 introduction (Papers I, II, III, V and VI)

Following the introduction of PHiD-CV10, changes were noted in several facets of otitis media epidemiology and its management. These changes were summarized in papers I, II, III, V and VI.

5.3.1 Epidemiology of acute otitis media in Iceland (Papers II and V)

Acute otitis media is often a benign temporary infection that will in many cases resolve without intervention (Thornton et al. 2011). The epidemiology of AOM is influenced by the distribution of risk-factors in the population, and by cultural factors that influence the propensity of parents to consult physicians (Blank et al. 2014; Fortanier et al. 2015). The epidemiology of AOM is therefore often highly variable between countries, as documented in Table 1.1. Despite this, AOM is the most common reason for physician visit among children, a fact which has been frequently documented in multiple countries (Arguedas et al. 2010; Marchisio et al. 2012; Monasta et al. 2012).

The epidemiology of acute otitis media visits to primary care was described in paper II. During the 11 year study period from 2005 to 2015, the overall incidence of AOM episodes among children zero to three years of age was 42 per 100 person-years. Prior to the introduction of PCV into the Icelandic pediatric vaccination program, 59% of children had visited a primary care physician one or more times by their third birthday and each child had experienced 1.6 episodes on average. The impact of PHiD-CV10 is discussed in Chapter 5.3.2. There were large variations in the incidence rate between both gender and age. The incidence was consistently higher in males than females. This gender difference has been documented in several different prospective studies of AOM epidemiology (Baraibar 1997; Macintyre et al. 2010; Paradise et al. 1997). However, the mechanism by which gender affects AOM incidence is unknown. The incidence of AOM increases from birth and peaks among children eight to 15 months of age. The abrupt peak at eight to 15 months of age and subsequent decrease may be hypothesized to be a function of two competing processes; the exposure of the child to pathogens due to daycare center attendance and the development of immunity to common pathogens (Hoog et al. 2014).

The incidence of AOM is highly variable between countries. Generally, the rate of AOM among European children ranges from 15 to 40 episodes per 100 person-years (Adam and Fehnle 2008; Esposito et al. 2007; Gisselsson-Solen 2017; Gribben et al. 2012; Lau et al. 2015; Liese et al. 2014; Marchisio et al. 2012; Todberg et al. 2014; Usonis et al. 2016), while the rate among children in the United States ranges from 95 to 200 episodes per 100 person-years (De Wals et al. 2009; Grijalva et al. 2006; Grijalva, Nuorti, and Griffin 2009; Poehling 2004; Zhou et al. 2008). With an incidence rate of 42 episodes per 100 person-years, the rate of AOM in Iceland seems to be higher than published estimates from most other European countries, but lower than published estimates from the United States. To our knowledge, only one previous study has reported the cumulative incidence of AOM in a modern cohort, demonstrating a 60% cumulative incidence by seven years of age (Todberg et al. 2014). Additionally, two studies from the twentieth century showed an 83% cumulative incidence by three years of age and 66% by two years of age (Bjarnason, Friðriksson, and Benediktsson 1991; Teele, Klein, and Rosner 1989). We observed a cumulative incidence of 59% by four years of age. Though not directly comparable to the other studies, this seems to suggest that the cumulative incidence is higher than a concurrent modern cohort in Denmark, but lower than historical cumulative rates. These differences between countries may be due to a multitude of different factors – some related to the distribution of risk factors in the population, others due to cultural differences that influence the ascertainment rate of AOM cases. Finally, some of the variation is likely due to design of the studies reporting the epidemiology of AOM.

Of studies that have examined the epidemiology of AOM, ours is the only longitudinal population-based study. We followed 11 birth-cohorts from birth until four years of age and accounted for censoring due to emigration. We accounted for follow-up visits for the same episode of AOM by excluding visits within 30 days of the index visit. Most of the other studies obtained aggregated AOM visits within a portion of the population and divided this with the aggregated number of children (De Wals et al. 2009; Gisselsson-Solen 2017; Grijalva et al. 2006; Grijalva, Nuorti, and Griffin 2009; Lau et al. 2015; Poehling 2004; Zhou et al. 2008). By using aggregated AOM visits as the numerator, they cannot exclude re-visits for the same AOM episode, which artificially inflates the incidence of AOM. The remaining studies followed individual children, but were not population-based and ascertained cases at variable time-points after six months of age (Adam and Fehnle 2008; Esposito et al. 2007; Gribben et al. 2012; Liese et al. 2014; Usonis et al. 2016).

The pathophysiology and microbiology of AOM are discussed in chapter 1.1.1. The most common bacterial causes of otitis media are non-typable Haemophilus influenzae (NTHi), Streptococcus pneumoniae and Moraxella catarrhalis (Bluestone, Stephenson, and Martin 1992; Casey and Pichichero 2004; Casey, Adlowitz, and Pichichero 2009; Ngo et al. 2016; Pumarola et al. 2013). The relative contribution of these three pathogens is remarkably stable between countries and over time (Ngo et al. 2016). This is likely a consequence of how common they are in the nasopharyngeal flora of children. A systematic review of studies from 1970-2014 which reported the etiology of otitis media, found that Streptococcus pneumoniae caused 30% of acute otitis media in Europe (Ngo et al. 2016). Our research group has shown that prior to PHiD-CV10 introduction in Iceland, the proportion of otitis media with tympanic perforation that was caused by pneumococcus was 20% (Quirk et al. 2018). This is similar to other countries (Ngo et al. 2016). The contribution of other pathogens has not yet been evaluated in Iceland. There does not seem to be strong evidence to suggest that the distribution of pathogens is different in Iceland than in other high-income countries, and thus other factors likely explain the difference in epidemiology.

The propensity to seek medical care may be influenced by several cultural and socio-economic factors. Access and cost of health care may influence whether parents consult a physician and may result in measurable changes in incidence between countries (Fortanier et al. 2015; Hadley 2003; Smolderen 2010). In Iceland, all permanent residents are provided health insurance by the government and there is excellent access to urgent care. Children are provided health care free of charge. The employment rate of Icelandic adults 20-64 years of age is 91% and 84% for males and females respectively, which means that Iceland has the highest employment rate in Europe (Eurostat 2018). These factors may push parents to seek early care for otitis media and would result in higher estimates of AOM incidence that would be closer to the true incidence in the population. This may also lead to overdiagnosis and overtreatment. Employment can however not explain the higher incidence reported in studies from the United States, as the employment rate is considerably lower, with only 76% and 66% of working age men and women employed (OECD 2019).

Iceland is ranked third in formal daycare attendance by The Organisation for Economic Co-operation and Development (OECD), with up to 60% of children under three years of age attending a daycare center for 38 hours per week or longer. This is compared to the OECD average of 35% attendance for 30 hours (“Enrolment in childcare and pre-schools” 2013). Both daycare attendance and the number of hours per week are known risk factors for acute otitis media (Ramakrishnan, Sparks, and Berryhill 2007) These factors could result in a higher incidence of AOM in Iceland relative to other countries and may explain some the observed difference between Iceland and other European countries. However, this does not explain why studies conducted in the United States consistently show higher incidence rates of AOM than Icelandic estimates. The rate of early daycare attendance in the United States is half that of Iceland, and below the average of OECD countries (“Enrolment in childcare and pre-schools” 2013).

Taken together, our population-based study of all primary care visits for acute otitis media over an 11 year period is likely to have ascertained a large proportion of true cases. To our knowledge, we report the only population-based follow-up of AOM incidence from birth to four years of age. The study reminds us that ratio statistics do not tell the whole story. Longitudinal studies are important to capture other perspectives on disease incidence. We report a 59% cumulative incidence of AOM by four years of age. The mean number of AOM episodes per child was 1.6, and 42% of children experienced two or more episodes. This better represents the burden of disease then ratio statistics such as 42 AOM episodes per 100 person-years, though such statistics also have their place.

5.3.2 Impact on primary care visits for otitis media (Papers II and VI)

Pneumococcal conjugate vaccines have been shown to reduce the incidence of acute otitis media in both randomized and observational trials. Eight randomized control trials examined the effect of PCV on AOM, demonstrating large decreases in pneumococcal AOM and moderate decreases in all-cause AOM (Black et al. 2000; Dagan et al. 2001; Eskola et al. 2001; Fireman et al. 2003; Kilpi et al. 2003; O’Brien et al. 2008; A. Palmu et al. 2015; Prymula et al. 2006; Tregnaghi et al. 2014; Vesikari et al. 2016). Two systematic reviews of observational studies examining the impact of PCV on the incidence of AOM identified nine studies, of which three accounted for pre-vaccine trends in AOM incidence and only one was population-based (Ben-Shimol et al. 2014; Grijalva et al. 2006; Grijalva, Nuorti, and Griffin 2009; Poehling 2004; Poehling et al. 2007; Lau et al. 2015; Magnus et al. 2012; Marom et al. 2014; Singleton et al. 2009). We report the results of two separate population-based studies estimating the impact of PHiD-CV10 on otitis media visits in primary care. The studies used different methods which complemented each other with regards to underlying assumptions, weaknesses and strengths.

Paper II reported the results of a population-based, individual-level birth-cohort study. The crude incidence rate of AOM visits to primary care decreased from 45 to 40 per 100 person-years among children zero to three years of age following vaccine introduction. Accounting for repeated episodes, the estimated impact of PHiD-CV10 was 21%. Of the previously identified observational studies, only three included individual-level data (Ben-Shimol et al. 2014; Magnus et al. 2012; Poehling et al. 2007).

Ben-Shimol et al. (2014) reported a prospective population-based study of middle-ear fluid cultures from a single-center in Israel following sequential PCV7 and PCV13 introduction. The study found a 96% and 85% impact on vaccine-type pneumococcal AOM, and a 60% reduction in all-cause AOM, but only obtained data on cases serious enough to be referred to the center for tympanocentesis. The propensity of physicians to refer otitis media cases for tympanocentesis was not evaluated. Changes in referral patterns may have potentially confounded the observed reduction in all-cause AOM, but would not be expected to confound vaccine-type AOM.

Magnus et al. (2012) utilized prospectively collected data from the Norwegian Mother and Child Cohort Study to estimate the rate of AOM before and after the introduction of PCV7. Mothers were recruited from 39 of 50 maternity units in Norway and data was collected by questionnaires at six, 18 and 36 months of age. The questionnaire only asked the mother whether their child had experienced AOM between 12-18 months of age, and 18-36 months of age, and did not specify the frequency or whether a physician was consulted. The study reported a 14% and 8% vaccine impact on the parent-reported prevalence of one or more AOM episodes in children 12-18 months of age and 18-36 months of age respectively.

Poehling et al. (2007) examined the hazard ratio of frequent otitis media between birth-cohorts enrolled in an state-insurance program before and after the introduction of PCV7. Four birth-cohorts 1998-2001 were included and followed to five years of age; the first two (1998-1999) were considered unvaccinated and the later two (2000-2001) were considered vaccinated. Impact was reported by comparing the 1998 birth-cohort to the 2000 birth-cohort, and by this metric the vaccine impact on frequent AOM was 17%-28%. However, the largest decrease was seen between the birth-cohorts 1998 and 1999, with almost no difference seen between the 1999 and 2000 birth-cohorts and no explanation or theory was provided in the discussion chapter.

Paper II adds to the current literature by providing robust population-based, individual-level observational evidence of PHiD-CV10 impact on all-cause AOM. As an outcome-measure, physician diagnosed AOM in primary care is more applicable for policy decisions than is the parent-reported prevalence of AOM, frequent AOM and otitis media requiring tympanocentesis. To our knowledge, this was the first study to directly account for repeated episodes within the same individual and employ survival methods that accounted for confounding by age and censoring. This allowed us to examine vaccine impact from perspectives that are not normally available to observational studies. In addition to impact on traditional ratio measures, we were able to show that the proportion of children who had never experienced an AOM episode increased from 40% in the vaccine non-eligible cohorts to 43% among vaccine eligible children. The vaccine impact was shown to independently protect against a child’s first and second episode, but in the subset of children who had already experienced two or more episodes of AOM, the vaccine was not shown to decrease the hazard of experiencing an additional episode. Longitudinal data on each child allowed us to calculate the mean number of AOM episodes at 36 months of age, and show that this had decreased from 1.61 among vaccine non-eligible children to 1.31 among vaccine eligible children.

By utilizing population-based data, our study avoided selection bias, which may have confounded the results of previous studies. Selection bias is possible in Ben-Shimol et al. (2014) if providers deferentially referred children for middle-ear sampling based on their vaccination status and may have occurred in Magnus et al. (2012) if the propensity of mothers to continue submitting questionnaires at 18 and 36 months was modified by the frequency of AOM in the child. Though Poehling et al. (2007) used a similar registry approach as our study, their population included only children of insured families. Despite these weaknesses, it should be noted the results of the above studies were congruent with a positive impact of PCV, which our study also confirmed.

Our study was strengthened by its long observational period which included six vaccine non-eligible birth-cohorts and five vaccine eligible birth-cohorts. Though paper II did not directly adjust for observed trends, the long pre-vaccine period provided context to the large and abrupt decrease in the hazard of AOM noted between the last vaccine non-eligible cohort (2010) and the first vaccine eligible cohort (2011). Such context would have been useful in interpreting Poehling et al. (2007), were the largest decreases in frequent AOM was noted between the first and second unvaccinated cohorts. Despite the strengths of paper II, the lack of adjustment for secular trends remained an important weakness.

Paper VI estimated the impact of PHiD-CV10 on physician diagnosed AOM in primary care taking into account pre-vaccine trends using a time series methodology. Four different methods were independently used to predict vaccine impact, which were then stacked by maximizing the predictive performance of the final model. This approach optimally accounts for secular trends and allows estimation of herd-effect in older children who were not vaccine eligible in the post-vaccine period. Using this method, the vaccine impact was 26% among children younger than one year of age, 28% among one year olds, 12% among two year olds and 14% among children three to four years of age.

Of the identified observational studies evaluating the impact of PCV on otitis media, only three have attempted to correct for secular trends (Grijalva et al. 2006; Lau et al. 2015; Marom et al. 2014). Grijalva et al. (2006) reported the rate of otitis media visits in the United States before and after PCV7 introduction. They estimated that PCV7 was associated with a 20% decline in the rates of otitis media among children younger than two years of age. The rate ratio of otitis media visits between children younger than two years of age and children three to six years of age were compared visually between the pre-vaccine period, transition year and a single post-vaccine year. The ratio of rate ratios was then calculated between the pre-vaccine period and post-vaccine year, and used as a measurement of impact. The largest decreases in the rate of otitis media visits among children younger than two years of age were observed before the introduction of PCV7, yet no explicit adjustment was made for pre-vaccine trends.

Lau et al. (2015) used an interrupted time series approach to estimate the sequential impact of PCV7 and PCV13 on otitis media in general practice, and reported a 21.8% reduction in the rate of otitis media visits in children younger than 10 years of age.
The pre- and post-vaccine periods were appropriately long and the study was well conducted, though the methods only allowed adjusting for linear trends, which may both over- and underestimate underlying trends (Bernal, Cummins, and Gasparrini 2016; Kontopantelis et al. 2015). The data was extracted from the IMS Disease Analyser; a longitudinal electronic health care database maintained by a for-profit health care information company. The database receives information from general practitioners who applied to participate and received compensation for manually contributing deidentified data. This process may select for general practitioners who are more self-reflective and adhere more strictly to evidence based medicine than do the general population of physicians.

Finally, Marom et al. (2014) used data on privately insured children younger than seven years of age in the United States from 2001-2011 to estimate the rate of otitis media visits before and after the introduction of PCV13 in 2010, using a linear time series analysis. They employed a similar approach as Grijalva et al. (2006), calculating rate ratios of otitis media visits between children younger than two years of age and children three to six years of age. They reported a stable rate ratio of 1.38 in the pre-vaccine period, which decreased to 1.01 in 2011 following the introduction of PCV13. Marom et al. (2014) did not report a single estimate of PCV13 impact. However, if the had calculated impact similarly to Grijalva et al. (2006), the estimated impact would have been 27%.

Paper VI adds to the current literature by providing population-based data on the impact of systematic vaccination with PHiD-CV10 on acute otitis media visits to primary care that is adequately adjusted for secular trends. Our study compliments previous studies examining the impact of PCV on otitis media visits and improves on them in several ways.

We used four models to adjust for secular trends, each with their own strengths and weaknesses. Two simple time series models were constructed, with and without a population offset. These two models were comparable to those that have been used by previous studies to adjust for secular trends, but improved upon them by allowing trends to be non-linear (Lau et al. 2015; Marom et al. 2014). Two additional models were included which incorporated data on primary care visits for other causes. A synthetic control framework was used, in which the relationship between otitis media visits and visits for other indications was estimated in the pre-vaccine period, and this relationship was used to predict the number of otitis media visits that would have occurred in the post-vaccine period, had PHiD-CV10 not been introduced (Bruhn et al. 2017; Shioda et al. 2018). The models were then optimally combined to maximize the predictive performance of the final stacked model. Using these methods we were able to adjust for secular trends and were not forced to assume that trends were linear and predictive uncertainty was minimized by incorporating data on observed visits to primary care for other indications.

By using population-based data, the possibility of selection bias was eliminated. Grijalva et al. (2006) and Marom et al. (2014) only include privately insured children and Lau et al. (2015) only includes visits to a self-selected subset of general practitioners. This selection has unknown significance with regards to the outcome being measured. It should however be noted, that while our study improves upon previous studies in many ways, the results of our study is congruent, demonstrating a large impact of PCV on otitis media.

In any vaccine ecology study, one must be careful when interpreting the results, as confounding may have contributed to over- or underestimation of the estimated vaccine impact. The impact of PHiD-CV10 on AOM in Iceland is quantified using two different methods that complement one another. Paper II provided individual-level evidence of vaccine effect, and the longitudinal data allowed us to report unique measurements of impact, such as cumulative incidence and the mean number of episodes as a function of age. However, the methods did not allow adjustment for trend. Paper VI used several different methods to capture pre-vaccine trends and predict how many AOM episodes were prevented by PHiD-CV10 introduction. Thus we were able to adjust for trend to a degree that has not previously been published in impact studies evaluating AOM. Despite the different methodologies, the results of paper II and paper VI are consistent. The point-estimate in both papers are similar and the diminishing vaccine impact seen in older children with the time series methodology is mirrored in the incidence rate ratios between the vaccine eligible and vaccine non-eligible birth-cohorts displayed in Figure 4.9.

Taken together, we believe that the impact of PHiD-CV10 on AOM has been demonstrated beyond a reasonable doubt. Using population-based data, we have removed the confounding threat of selection bias. The individual-level cohort study provided robust evidence with well delineated exposures and time at-risk, and the time series analysis adjusted the estimated impact for secular trends. This is an important addition to the current literature on PCV, as otitis media is the most common childhood infection that leads to physician visits and antimicrobial prescriptions (Arguedas et al. 2010; Marchisio et al. 2012; Monasta et al. 2012).

5.3.3 Impact on pediatric emergency department visits for acute otitis media (Paper I)

Acute otitis media is generally a benign self-limiting condition, and cases that require treatment usually respond well to oral antimicrobials (Ahmed, Shapiro, and Bhattacharyya 2014). However, AOM can progress to recurrent or chronic infection and require more invasive treatment (Cullen, Hall, and Golosinskiy 2009; Vlastarakos et al. 2007). One example of this is mastoiditis – a rare but serious complication of AOM that invariably requires hospital admission and administration of intravenous antimicrobials (Finnbogadóttir et al. 2009; Groth et al. 2011).

Paper I examined the impact of PHiD-CV10 introduction on the incidence of AOM visits to the emergency department of Children’s Hospital Iceland. Following vaccine introduction, the incidence decreased significantly from 4.7 visits per 100 person-years in the pre-vaccine period to 4.1 visits per 100 person-years in the post-vaccine period. The estimated vaccine impact was 12%. The study employed a simple pre/post design, and was not able to adjust for secular trends due to the nature of the study data. However, we did show that the AOM visits decreased despite an increase in all-cause visits. The design was similar to most previously published observational studies of PCV on AOM (Vojtek, Nordgren, and Hoet 2017).

Emergency department visits for AOM are likely to represent a subset of more serious cases than those that present to primary care centers. It is therefore important to establish impact in both settings, as they represent different aspects of AOM epidemiology and burden of disease. This observed decrease in the incidence of emergency department visits probably reflects the effect of PHiD-CV10 to reduce not only AOM in general but also more serious manifestations of AOM. A robust time series approach with synthetic controls is warranted to further investigate the impact of PHiD-CV10 on pediatric emergency department visits for acute otitis media, as was done in paper VI for primary care.

5.3.4 Impact on outpatient antimicrobial prescriptions for otitis media (Paper III)

Antimicrobial consumption is directly linked to antimicrobial resistance on both the individual and population levels (Blommaert et al. 2014; Bruyndonckx et al. 2015; Costelloe et al. 2010; Goossens et al. 2005). Otitis media is responsible for the majority of antimicrobial prescriptions in children, and thus contributes significantly to antimicrobial resistance (Austin, Kristinsson, and Anderson 1999; Grijalva, Nuorti, and Griffin 2009).

The impact of the introduction of PHiD-CV10 on outpatient antimicrobial prescriptions for AOM was explored in Paper III, using population-based registries. Data were obtained from the Icelandic National Drug Prescription Registry which included all filled outpatient antimicrobial prescriptions during a thirteen year period from 2005 to 2017. These were linked to visits to primary care physicians for AOM which were extracted from the Primary Care Registry. The vaccine impact on AOM associated antimicrobial prescriptions was 22%, adjusting for the number of previous antimicrobial prescriptions. The impact was 6% on all antimicrobial prescriptions regardless of indication.

Several randomized controlled trials have estimated the vaccine efficacy of PCV on outpatient antimicrobial consumption, with results ranging from 5% to 15% (Dagan et al. 2001; Fireman et al. 2003; Palmu et al. 2014). Dagan et al. (2001) specifically examined efficacy against otitis media associated antimicrobial days, and reported 20% vaccine efficacy. Our results are within reasonable bounds of these findings.

Though blinded randomized controlled trials do provide robust estimates of vaccine efficacy, they do so under artificial conditions. Parents and physicians may behave differently knowing that their actions are being observed and quantified by researchers, and this may reduce the incidence of inappropriate prescribing. In that respect, observational studies provide additional information by demonstrating whether vaccine impact can still be observed in true clinical settings. To our knowledge, only one study has previously assessed the impact of PCV on otitis media associated antimicrobial prescriptions, which demonstrated a 20% vaccine impact (Lau et al. 2015). Two other observational studies on outpatient antimicrobial prescriptions demonstrated an association between PCV introduction and a decline in prescriptions (Gefenaite et al. 2014; Howitz et al. 2017).

To our knowledge, our study was the first to estimate the individual-level impact of pneumococcal vaccination on antimicrobial prescriptions and directly account for repeated prescriptions within the same child. This allowed a much sharper delineation between vaccinated and unvaccinated children. When data are aggregated and analyzed based on calendar-time, the period from the introduction of the vaccine until the majority of children under observation are vaccinated will span several years. During that time, other secular trends exert their confounding effects on the outcome, and the start of the post-vaccination period is not well defined. Contrast this with our cohort study were children in the 2010 cohort, who were not vaccine eligible, were directly compared to the vaccine eligible children in the 2011 cohort. Though the children in the 2010 birth-cohort were one year older on average, the two cohorts experienced the same viral epidemics, were treated by the same physicians and interacted with each other daily in daycare centers.

By including six non-eligible cohorts and five eligible cohorts, we were able to adequately visualize secular trends. A decreasing trend in the hazard of AOM associated antimicrobial prescriptions was noted in the vaccine non-eligible cohorts, such that the hazard was significantly higher in the 2007 compared to the 2010 birth-cohort. Though this may change the point-estimate, the large, abrupt and significant hazard decrease between the 2011 and 2010 birth-cohorts strongly suggests a true vaccine impact.

Our study adds to the current literature by providing population-based, individual-level data on the impact of PHiD-CV10 on AOM-associated antimicrobial prescriptions. Assessing the impact of vaccination on antimicrobial prescriptions is an important outcome measure as it captures a subset of outpatient AOM cases that is serious enough to warrant treatment. It encompasses both a physician’s diagnosis of AOM, and the decision to treat. Our data suggest that PCV and other vaccines may be an important tool in the fight against antimicrobial resistance, by decreasing antimicrobial prescriptions among children.

5.3.5 Evidence of herd-effect of PHiD-CV10 on the incidence of otitis media in the unvaccinated population (Papers II and VI)

Vaccination decreases the susceptibility of vaccinated individuals to become infected, and later decreases the risk of passing the pathogen to others. Therefore, there is strong reason to believe that introducing PHiD-CV10 into the population could confer indirect protection against pneumococcal disease to non-vaccinated members of the population. In vaccine studies, this indirect effect is termed herd-effect, and has been convincingly demonstrated for invasive pneumococcal disease following introduction of pneumococcal conjugate vaccines (Tsaban and Ben-Shimol 2017). There is a paucity of published literature demonstrating herd-effect for acute otitis media following pneumococcal conjugate vaccine introduction (Jennifer D Loo et al. 2014).

In paper II, a large decrease in the incidence of AOM was found among children younger than four months of age. The incidence rate ratio between children in the vaccine eligible cohorts compared to children in the vaccine non-eligible cohorts was 0.60 (0.51-0.69), which translates to a 40% vaccine impact in this age-group. Children three to four months of age may have already received the first dose of PHiD-CV10, but will not yet have received any direct benefit (Nicholls, Leach, and Morris 2016). Thus the data strongly supports existence of a herd-effect for AOM, and that vaccinating children at three, five and 12 months of age with PHiD-CV10 confers protection to children too young to be themselves directly protected. One previous study reported fewer positive pneumococcal cultures from samples taken from the middle ear of children younger than four months of age, following the sequential introduction of PCV7 and PCV13 in Israel, suggesting a possible herd-effect (Ben-Shimol et al. 2016). Though population-based, their case ascertainment was dependent on referral for tympanocentesis from primary care physicians, and it is unclear from the publication whether the proportion of cases referred changed following vaccination. Any such change could introduce an artificial decrease following vaccination. Unlike Ben-Shimol et al. (2016), we included all cases of AOM presenting to primary care physicians. Our study is the first to suggest herd-effect against all-cause physician-diagnosed AOM among children too young to receive the vaccination.

In paper VI, a time series methodology with synthetic controls was applied to ascertain the PHiD-CV10 impact on otitis media visits in primary care. The study period was 2005-2015, during which none of the vaccine eligible children had reached five years of age. The vaccine impact was estimated separately in children younger than one, one, two, three to four, five to nine, 10-14 and 15-19 years of age. The estimated vaccine impact was 12%, 17% and 11% in children five to nine, 10-14 and 15-19 years of age respectively. For these estimates the 95% credible intervals did not cross the ratio value of one, which translates to a 97.5% or higher probability that the impact was larger than or equal to 1%. The result is consistent with a strong herd-effect of PHiD-CV10 against otitis media in unvaccinated older children. We are unaware of any previously published study which confirms the existence of herd-effect against AOM in older children and believe our results to be unique. Because the data regarding visits to primary care for other indications were aggregated for children zero to 11 months of age, we were unable to test for herd-effect among children younger than four months of age using the time series methodology. Visual examination of Figure 4.9 does not suggest a decreasing pre-vaccine trend in AOM episodes among children younger than four months of age, and therefore the result would be expected to be the same.

We believe the results of paper II and paper VI to be an important addition to the current literature regarding herd-effect of pneumococcal vaccination. AOM is associated with a large individual and societal burden of disease (Greenberg et al. 2003; Monasta et al. 2012). Our studies show that countries that introduce PCV into their national immunization programs can and should expect a decrease in AOM episodes among age-groups not covered by the vaccination program, in addition to a larger direct decrease among vaccinated children. This provides further evidence of PCV benefit and should inform future cost-utility analyses.

5.3.6 Impact on acute otitis media with treatment failure (Paper I)

There is a paucity of published studies that examine the impact of pneumococcal conjugate vaccines on acute otitis media with treatment failure. Treatment failure is generally defined as the persistence of symptoms despite antimicrobial treatment, though precise definitions vary (Casey and Pichichero 2004). There is evidence to suggest that treatment failure occurs more commonly in AOM caused by Streptococcus pneumoniae and Haemophilius influenzae than in AOM caused by other pathogens (Casey and Pichichero 2004; Pichichero et al. 2008). Otitis media with treatment failure represents an important subset of cases that is associated with a higher burden of disease.

In paper I, the impact of PHiD-CV10 on the incidence of AOM episodes treated with parenteral ceftriaxone at Children’s Hospital Iceland was examined. Data regarding the use of ceftriaxone and visits for AOM were extracted from Children’s Hospital Iceland’s patient registry for the period from 2008 to 2015. When ceftriaxone is used in the treatment of AOM at Children’s Hospital Iceland, it is done exclusively in cases of treatment failure, difficult recurrent infections or in culture-proven antimicrobial-resistant pneumococcus. The study demonstrated a statistically significant 52% impact of PHiD-CV10 on ceftriaxone treated AOM episodes among children younger than four years of age.

To our knowledge, this is the first study to show a significant decrease in AOM with treatment failure following PCV introduction. Because AOM with treatment failure is not objectively defined in the literature, a proxy measurement such as ceftriaxone use is needed. Parenteral antimicrobial use is avoided unless absolutely necessary and is not administered at primary care clinics. It is therefore an appropriate and clinically relevant proxy measurement for the worst cases of otitis media with treatment failure.

We do however acknowledge that we are not currently in possession of data that proves that a trial of oral antimicrobials was attempted prior to initiation of parenteral ceftriaxone. It is an assumption that is grounded in established clinical norms of pediatric medicine. Unlike the other study data upon which this thesis is based, the data used in paper I was not linkable to population registries, which precludes formal evaluation of this assumption.

Several evaluations were performed to examine other possible explanations for the observed decrease. To exclude the possibility that the observed reduction was due to a decrease in AOM cases, the analysis was repeated using the number of AOM visits to Children’s Hospital Iceland as the rate denominator, If this statistic were used, the estimated vaccine impact would have been 42%. No changes were observed in ceftriaxone usage among older children, and among children younger than four, ceftriaxone use for indications other than AOM and pneumonia did not change significantly, with an estimated incidence rate ratio 0.96 (95% CI: 0.87-1.06). Though not statistically significant, the observed rate ratio does not prove an absence of effect. The result is consistent with as much as a 13% decrease in the incidence of ceftriaxone treatment for indications other than AOM and pneumonia. This may be partly explained by the inclusion of indications other than AOM and pneumonia that are possibly related to PCV, such as unspecified fever, cough and sepsis.

No changes in institutional guidelines regarding the use of ceftriaxone or treatment of severe AOM occurred during the study period. When the totality of evidence is considered, our study suggests a true independent association between the introduction of PHiD-CV10 into the Icelandic pediatric vaccination program, and the observed decrease in AOM treated with parenteral ceftriaxone. Clinical experience tells us that this may be considered a proxy for AOM with treatment failure.

5.3.7 Impact of PHiD-CV10 on tympanostomy tube placements (Paper IV)

Tympanostomy tube placements are the most common pediatric surgical procedure requiring general anesthesia (Black 1984; Cullen, Hall, and Golosinskiy 2009). The most common indications for the procedure are recurrent acute otitis media and otitis media with effusion, however evidence of benefit is inconsistent (Venekamp et al. 2018; Browning et al. 2010). In paper IV, we report the impact of PHiD-CV10 introduction into the Icelandic pediatric vaccination program on tympanostomy tube placements. This population-based study obtained data on tympanostomy procedures from the reimbursement database of Icelandic Health Insurance and Landspitali University Hospital’s patient registry. The estimated impact was -5% (95%CI -15% to 4%), indicating a non-significant increase in the hazard of tympanostomy tube placement following vaccine introduction.

The results are neither congruent with the results of other studies included in this thesis, or previously published randomized controlled trials and observational studies. Papers II and III demonstrated a clinically meaningful decrease in primary care visits, and antimicrobial prescriptions for AOM and all-cause outpatient antimicrobial prescriptions. The results of randomized controlled trials are summarized in Table 1.3. Black et al. (2000) demonstrated a 20% efficacy of PCV7 for tympanostomy tube procedures, and four other randomized studies showed a non-significant preventative effect (Eskola et al. 2001; O’Brien et al. 2008; Prymula et al. 2006; A. Palmu et al. 2015). Two observational studies have examined the effect of PCV on tympanostomy tube procedures; one was conducted in the United States reported reductions in the risk of tympanostomy procedures following the introduction of PCV7, but did not consider the possibility of secular trends (Poehling et al. 2007). The other was conducted in Australia carefully evaluated secular trends and other confounders, and reported a 23% reduction in the rate of procedures associated with the introduction of PCV7 (Jardine et al. 2009).

We report an incidence of 106 procedures per 1,000 person-years in children zero to five years of age, and a cumulative incidence of 32% by six years of age. This represents both the highest incidence rate and cumulative incidence of tympanostomy procedures that has been published to date (Table 1.2). We demonstrated that among children who underwent tympanostomy tube placement, the proportion of children who had never visited a primary care physician or filled an antimicrobial prescription prior to the procedure was higher in the vaccine eligible cohorts. The study design does not allow us to estimate the cause of this unexpected increase. We know of no plausible biological explanation for the consistently high proportion of Icelandic children who undergo invasive treatment for a benign self-limiting disease. Our study provides evidence that children who underwent tympanostomy placement after the introduction of PHiD-CV10 had fewer AOM associated visits and had filled fewer antimicrobial prescriptions at the time of surgery. We believe that the high incidence and cumulative incidence of tympanostomy tube procedures both before and after systematic pneumococcal vaccination in Iceland warrants further study and scrutiny by both researchers and the Icelandic Directorate of Health.

5.4 Impact of PHiD-CV10 on hospital admissions for respiratory infections and invasive pneumococcal disease (Papers V and VI)

Infections that require close monitoring or parenteral antimicrobials are admitted to hospitals for inpatient treatment. Hospital-based treatment is reserved for serious infections, and is therefore an important manifestation of disease burden. Pneumonia is a large contributor to pediatric hospitalizations, accounting for 3%-18% of admissions (Madhi et al. 2012).

While less common, invasive pneumococcal disease is associated with high mortality rates. The case-fatality ratio of hospitalized IPD in Europe was 2.4% in children younger than five years of age, 9.1% in individuals 5-64 years of age, and 18.6% in adults 65 years of age and older (Torné et al. 2014).

We report the results of two separate population-based studies that estimate the impact of PHiD-CV10 on hospital admissions. The studies used different methods which complemented each other with regards to underlying assumptions, weaknesses and strengths.

5.4.1 Impact on hospital admissions for otitis media and its complications (Paper V)

The impact of PHiD-CV10 on pediatric hospital admissions for otitis media was examined in paper V. Hospital admissions for otitis media were compared between the vaccine eligible and non-eligible birth-cohorts using a Cox regression model, resulting in a statistically significant 43% estimated vaccine impact. However, this difference seemed to be largely driven by a sharp decrease in the incidence of otitis media associated hospitalization between the first two vaccine non-eligible cohorts (2005 and 2006). This decrease occurred before even selective vaccination of high-risk children was common practice, and several years before the introduction of PHiD-CV10.

Four observational studies that estimate the impact of PCV on otitis media associated hospitalizations have previously been published (Durando et al. 2009; Gisselsson-Solen 2017; Marom et al. 2017; Tawfik et al. 2017). A retrospective study in the United States reported a 66% relative risk reduction in hospitalizations in the period following the introduction of PCV13, compared to the period prior to PCV7 introduction – nine years apart (Tawfik et al. 2017). No attempt was made to correct for secular trends. The study did not include a single observational year prior to the introduction of PCV7 in 2000. The included figures revealed that the entire observed decrease occurred in the first year of the study, between 2000 and 2001 (Tawfik et al. 2017). Gisselsson-Solen (2017) similarly reported a 42% reduction in AOM associated admissions in a period following the introduction of higher valent PCV compared to the period prior to the introduction of PCV7. The influence of secular trends were not discussed or adjusted for, and the figures revealed that a large portion of the observed decrease occurred immediately following PCV7 introduction. Durando et al. (2009) implemented a pre- and post cohort design, and compared the rates of otitis media hospitalizations using a simple comparison of means. They included hospitalizations for urinary tract infections as a control, and reported a 36% relative risk reduction. Trends were not considered. Finally, Marom et al. (2017) did not observe a decrease in hospitalizations.

A careful examination of our results does not suggest an observable decrease in otitis media associated hospitalizations in Iceland that can be attributed to the introduction of PHiD-CV10 into the pediatric vaccination program. This is largely congruent with the results of previous studies despite their variable interpretation. Our findings underscore the importance of carefully considering the results of vaccine ecology studies, as secular trends may exert great influence over a long time period.

5.4.2 Impact on pediatric pneumonia hospitalizations (Papers V and VI)

The impact of PHiD-CV10 on pneumonia hospitalizations was reported in two population-based studies, papers V and VI.

Four randomized trials have evaluated the efficacy of PCV for clinical pneumonia and their results are summarized Table 1.4. (S. B. Black et al. 2002; Cutts et al. 2005; Kilpi et al. 2018; Tregnaghi et al. 2014). Only Kilpi et al. (2018) reported efficacy for hospital-diagnosed pneumonia, which was 27%. A systematic review and meta-analysis of observational studies of higher valency PCV examining impact on clinical pneumonia identified 11 studies and concluded that among children aged 24 months and younger, the introduction of PCV resulted in a 17% (95%CI 11% to 22%) reduction in clinical pneumonia (Alicino et al. 2017). Of the observational studies identified by the systematic review, three were population-based (Berglund et al. 2014; Nair et al. 2016; Saxena et al. 2015). Only two studies identified by the review discussed or attempted to adjust for secular trends (Sgambatti et al. 2016; Simonsen et al. 2014).

Paper V reported the impact of PHiD-CV10 introduction on pneumonia hospitalizations among children younger than three years of age, and demonstrated a 20% reduction in pneumonia hospitalizations following the nationwide introduction of PHiD-CV into the pediatric vaccination program. The study adds to a growing body of literature by providing population-based, individual-level observational evidence of PCV impact on pneumonia hospitalizations.

As in any vaccine ecological study, careful consideration must be paid to the possibility of unmeasured variables unrelated to the vaccination which could influence the outcome. Because the study followed all children in Iceland for 11 consecutive birth-cohorts, sampling bias was eliminated. This means that differences in the distribution of risk factors among children in the vaccine non-eligible cohorts compared to vaccine eligible cohorts can only be due to systematic changes in the whole population. We are unaware of any systematic changes that would have reduced the incidence of pediatric pneumonia requiring hospitalization, except for the introduction of PHiD-CV. The Children’s Hospital Iceland has remained the only pediatric secondary and tertiary care hospital in Iceland. The proportion of the pediatric population the hospital serves as a primary hospital has increased, rather than decreased during the study period. During the same period, hospital admissions due lower respiratory tract infections other than pneumonia increased by 32% in the vaccine eligible cohorts. This would be expected to increase the susceptibility of children for pneumonia. Despite the strengths of paper V, the lack of adjustment for secular trends remained an important weakness.

In paper VI, a time series methodology was employed and several different methods used to correct for secular trends as has been previously discussed. Among children zero to four years of age, the posterior estimate of the rate ratio for pneumonia hospitalizations was 0.67, with the 95% credible interval spanning 0.51-1.39. Though the 97.5% credible limit is above the threshold value of one, there is a 94% probability that the rate ratio is lower than one, and a 90% probability that it is lower than 0.83. The cumulative number of prevented hospital admissions for pneumonia in this age-group during the first seven years of the vaccination period was 142 (95% credible intervals -115 to 307) – an impressive decline considering that the baseline rate of pneumonia hospitalization was 65-75 per year in this age-group.

We believe that our methods are considerably more robust than the two previously identified observational studies examining PCV impact that discussed or adjusted for secular trends. Sgambatti et al. (2016) reported the incidence of pneumonia hospitalizations among children two to 35 months of age during the two year period before and after the introduction of PHiD-CV10 in Brazil. They examined whether a trend had occurred in the pre-vaccine period by performing a linear regression and concluded that no trend was detectable and therefore no adjustment necessary. Impact was reported as a simple unadjusted rate ratio between the pre- and post-vaccine period, and was 13%. However, linear trends are difficult to estimate in short seasonal time series, and depend strongly on the included period (Bernal, Cummins, and Gasparrini 2016; Kontopantelis et al. 2015).

Simonsen et al. (2014) reported the incidence of hospital admissions for pneumonia, empyema and invasive pneumococcal disease before and after the introduction of PCV13 using data from a convenience sample of 500 hospitals in the United States. They demonstrated 21% impact of PCV13 on all-cause pneumonia among children younger than two years of age, and a 17% impact among children two to four years of age. The pre-vaccine period used to estimate trends was appropriately long (2005-2012) but the trend was again assumed to be linear. Interestingly, sensitivity analyses revealed that excluding the adjustment for linear trend effaced the estimated impact on all-cause pneumonia among children younger than five years of age. The authors did not provide figures showing the observed number of all-cause pneumonia hospitalizations and estimated trend and readers are therefore unable to verify why this might have occurred. However, one could hypothesize that the linear trend was possibly overestimating the number of hospitalizations that would have occurred in the post-vaccine period, had PCV13 not been introduced.

Contrast this with our study, which estimated the relationship between pneumonia hospitalizations and hospitalizations due to other specific diagnoses (synthetic controls) during the six year pre-vaccine period 2005-2010, and used this relationship to predict the monthly number of pneumonia hospitalizations in the post-vaccine period (2013-2017), had PHiD-CV10 not been introduced. This method does not require trends to be linear and correctly propagates the uncertainty of the prediction by incorporating observed synthetic controls in the post-vaccine period. Our methods are based on Bruhn et al. (2017). In their paper they demonstrate their methodology by analyzing population-based data on pneumonia hospitalizations in five countries; Brazil, Chile, Ecuador, Mexico and the United States, before and after the introduction of PCV7 and PHiD-CV10. Their data was aggregated to slightly different age-groups. The rate ratio between predicted and observed pneumonia hospitalizations ranged from 0.55 to 0.86 among children younger than 12 months of age and the 95% credible intervals did not cross the threshold value of one. However, the rate ratio for children 12-23 months of age and 25-59 months of age was more variable, ranging from 0.76 to 1.05 and 0.77 to 1.06 respectively. Our results are largely congruent.

Taken together, papers V and VI convincingly demonstrate a robust impact of PHiD-CV10 on pneumonia hospitalizations among Icelandic children. The methods used by each paper compliment each other, with both papers offsetting methodological weaknesses of the other. Paper V provides individual-level data on pneumonia hospitalizations among all Icelandic children, and corrects for censoring due to hospitalization, death or emigration, providing accurate time at-risk for the whole population. Paper VI used robust methodology to correct for secular trends that explicitly controls for changes in the rates of hospitalization for indications unrelated to the vaccination. This corrects for both observed trends in pneumonia hospitalizations in the pre-vaccine period and trends in health care utilization in general both before and after vaccine introduction. Both studies are population-based and encompass a 13 year observational period. Our studies add to the current literature by demonstrating a robust, measurable decrease in the incidence of pneumonia hospitalization among children following the introduction PHiD-CV10 into a national immunization program.

5.4.3 Impact on hospital admissions for culture confirmed invasive pneumococcal disease

Invasive pneumococcal disease represents the most serious manifestation of pneumococcal infection. Randomized controlled trials evaluating PCV for IPD have consistently shown large efficacy for both vaccine-type and all-cause IPD, ranging from 77% to 100% and 46% to 93% respectively (Black et al. 2000; Cutts et al. 2005; O’Brien et al. 2003; Palmu et al. 2013; Tregnaghi et al. 2014). The results of the randomized controlled trials are summarized in Table 1.5.

A large number of observational studies have been published on the impact of PCV on IPD, and their results are considerably more variable than those of randomized trials. Myint et al. (2013) conducted a systematic review of observational studies of the direct impact of PCV7 on vaccine-type and all-cause IPD. The review identified 18 studies which reported the impact on vaccine-type IPD, which ranged from 39.9% to 99.1%, with a median impact of 90.1%. For the outcome of all-cause IPD, the review identified 30 studies, which reported impact estimates ranging from 1.7% to 80.2% and a median impact of 45.0%. The variation in the study estimates was likely due to variable baseline seroprevalence of vaccine-type pneumococci, vaccine uptake, case-ascertainment, observation period and other methodological differences. Study design was not specifically reviewed (Myint et al. 2013).

We explored the impact of PHiD-CV10 introduction on hospitalized culture-confirmed IPD among vaccine-eligible children in papers V and VI. Both papers utilized culture data from Iceland’s reference laboratory at the Department of Clinical Microbiology at Landspitali University Hospital, which receives all invasive pneumococcal isolates for the whole country. Culture data were then cross-referenced with hospitalization records.

In paper V, individual-level population-based data were analyzed by birth-cohort. They were divided into vaccine non-eligible and vaccine eligible cohorts and followed until three years of age. No vaccine-type IPD developed in the 22,336 children in the vaccine eligible cohorts during the 57,507 person-years at-risk. This is compared to 18 vaccine-type IPD hospitalizations during 84,949 person-years at-risk accumulated by the 29,050 children who comprise the vaccine non-eligible cohorts – an incidence rate of 21.2 IPD hospitalizations per 100,000 person-years. The impact of PHiD-CV10 on hospitalized vaccine-type IPD was therefore 100%.

The rate of all-cause IPD decreased from 24.7 (25 hospitalized cases) per 100,000 person-years in the vaccine non-eligible cohorts to 1.71 (one hospitalized case) per 100,000 person-years in the vaccine eligible cohorts. The adjusted hazard ratio was 0.07 (95% CI 0.01-0.50), representing an impact of 93%. These impact estimates compare nicely with the results of randomized controlled trials, and the results of previous observational studies, and adds to the current literature by providing individual-level population-based evidence of extensive protection against IPD (Myint et al. 2013). However, the methods used in paper V were unable to correct for secular trends, and a slight decreasing trend in the incidence of IPD had been observed in the pre-vaccine period.

Paper VI used a time series methodology to estimate the impact of PHiD-CV10 on IPD among children zero to four years of age, adjusting for secular trends in IPD and all-cause hospital admissions. Due to the extremely low event-rate, and the statistical uncertainty introduced by including the contribution of secular trends, the credible intervals of the rate ratio were wide. The rate ratio between observed and predicted IPD hospitalizations was 0.27 (95% credible interval 0.05-3.00). However, the results were consistent with a 90% probability that the rate ratio was equal to or less than 0.75 and a 93% probability that the rate ratio was lower than one.

The cumulative number of prevented IPD hospitalizations was 14 (95% credible intervals -2 to 67). There was a 90% probability that the vaccine had prevented seven or more IPD hospitalizations. Because aggregate data were analyzed, the number of IPD hospitalizations occurring in the post-vaccine period was three, compared to one in paper V. This is due to several factors. The aggregated age-group included more children (zero to four years of age, compared to zero to two years of age in paper V), and person-time at-risk was not censored due to immigration or emigration.

Taken together, papers V and VI demonstrate a large impact of PHiD-CV10 on vaccine-type and all-cause IPD among vaccine eligible children. Our results represent the highest quality of observational evidence. The outcome measure of culture proven IPD is specific and sampling bias is not possible do to the population-based nature of the data. The observational period is long compared to previous studies, which allows adequate estimation of trends (Myint et al. 2013). Paper V provides individual-level evidence and allows accurate estimation of rates because of careful censoring of person-time. Paper VI adds to this by adjusting for secular trends in both rates of IPD and all-cause hospitalizations. ### Evidence of herd-effect of PHiD-CV10 on pneumonia hospitalization in the unvaccinated population (Paper VI)

To our knowledge, only seven previous publications have examined the herd-effect of pneumococcal conjugate vaccines on pneumonia hospitalizations among the unvaccinated population (Andrade et al. 2017; Bruhn et al. 2017; Griffin et al. 2013; Grijalva et al. 2007; Jardine, Menzies, and McIntyre 2010; Simonsen et al. 2011, 2014), five of which were identified by a recent systematic review (Tsaban and Ben-Shimol 2017). Two of the included studies, Simonsen et al. (2011) and Griffin et al. (2013), did not adjust for trends. The results of the remaining five publications generally showed a reduction in the incidence of pneumonia hospitalizations in children zero to four and five to 17 years of age, and among adults 18-39 and 40-64 years of age, though the exact bounds of the age-groups and point-estimates varied between studies. Impact estimates ranged from 3% to 24% among five to 17 years of age; 0% to 26% among adults 18-39 years of age; and 0% to 19% among adults 40-64 years of age. All but two of the studies suggested an impact among adults 65 years of age and older, with estimates ranging from 3% to 15%, though none reached significance at the pre-specified alpha of 0.05. Bruhn et al. (2017) reported the impact of PCV on pneumonia hospitalizations in five countries and divided the oldest age-group into adults 65-79 years of age and adults 80 years of age and older. Using a sophisticated synthetic-control methodology, they did not find evidence of impact among these age-groups. The results of Andrade et al. (2017) stand out from the rest, reporting a statistically significant 16% increase in pneumonia hospitalizations among adults 65 years of age and older following the introduction of PHiD-CV10 into the national immunization program in Brazil. They discuss this in detail citing a baseline increasing trend in this age-group prior to vaccine introduction.

The herd-effect of PHiD-CV10 on pneumonia hospitalizations in Iceland was examined in paper VI. Determining the etiology of pneumonia is often difficult as direct sampling of lung tissue is not feasible (Cilloniz et al. 2016; Feikin et al. 2017). In a prospective study of 310 consecutive pneumonia hospitalizations at Landspitali University Hospital in 2008, a potential causative pathogen was only identified in 52% of admissions, despite the active gathering of blood and sputum cultured, oropharyngeal swabs for polymerase chain reaction analysis and urine for antigen testing (Bjarnason et al. 2018). Because of this, and because the results were later to be used in a cost-effectiveness analysis, we used the most sensitive but least specific outcome measure of all-cause pneumonia. We employed a time series methodology to adjust for both secular trends in the outcome measure and trends in hospitalizations for other indications, and demonstrated a robust impact on pneumonia hospitalizations in children five to 19 years of age (26%), and adults 20-39 (32%), 65-79 (25%) and 80 years of age and older (24%) (Bruhn et al. 2017; Shioda et al. 2018). Additionally, the data indicated an 8% impact among adults 40-64 years of age and were compatible with a 77.5% probability that the impact was equal to or larger than 1%. The posterior estimate of impact for children five to 19 years of age was compatible with a 90% probability that the impact was equal to or larger than 1%. Likewise, among adults 20-39, 65-79 and 80 years of age and older, the results were compatible with a 98.5%, 96% and 96% probability that the impact was equal to or larger than 1%.

Our findings of herd-effect among adults older than 65 years of age are discordant with Bruhn et al. (2017) – a publication that introduces some of the methods used in our study. There are several possible reasons for this disagreement. Our study examined the impact of PHiD-CV10, while theirs was primarily a study of PCV7. In Iceland, the uptake of PHiD-CV10 was immediately high, achieving over 97% uptake of the primary doses in the vaccine eligible birth-cohorts. Contrast this with the United States where uptake of two primary doses was initially 18% in 2002 and increased to 46% in 2004, which was the final year included in Bruhn et al. (2017; McLaughlin et al. 2016). For the other countries included in Bruhn et al. (2017), uptake for three doses during the first year of PCV introduction was 9% in Mexico, 17% in Ecuador, 55% in Chile, and 82% in Brazil (Oliveira et al. 2016). We included seven years of data following vaccine introduction, compared to two to five years of post-vaccine data in Bruhn et al. (2017). This allowed for a longer period for herd-effect to develop and present itself. We fitted four different models to the data (of which the synthetic control model of Bruhn et al. (2017) was one) and produced a final stacked model by maximizing the leave-one-out cross-validation likelihood in the pre-vaccine period. This produced the optimal model, given the pre-vaccine data.

Our results provide robust evidence for the existence of herd-effect on pneumonia hospitalizations following the introduction of PHiD-CV10 into a national immunization program. They support the findings of previous studies that have indicated herd-effect, and improve on them in many ways. The data underlying the study were population-based excluding the possibility that selection bias confounded our findings, and the results were adjusted for secular trends in hospitalization rates and the pre-vaccine incidence of pneumonia. As did all previous studies, the largest and most robust indirect decrease in pneumonia hospitalizations was noted among adults 18-39 years of age. According to Statistics Iceland, the median age at which Icelandic mothers give birth to their first child is 27 years of age. When all births are considered, the median age of mothers and fathers are 29 and 32 years of age respectively. Furthermore, data provided by Statistics Iceland shows that this age-group consistently represents 50% of all daycare staff in Iceland. Adults 18-39 years of age are therefore the primary providers of care for young children and would be expected to benefit the most from indirect protection, as is confirmed by our study. Though purely speculative, the relatively small indirect impact among adults 40-64 years of age could possibly be explained by this age-group having little direct contact with young children. They could represent adults who are unlikely to have young children of their own, and are on average at the height of their professional careers, limiting direct care of grandchildren. This might also explain why the indirect impact increases again among adults 65-79 years of age. The proportion of grandparents would be higher, parents would be entering the height of their own careers and grandparents would at this age possibly have more leeway to provide direct care as needed.

5.4.4 Evidence of herd-effect of PHiD-CV10 on hospital admissions for invasive pneumococcal disease in the unvaccinated population (Paper VI)

The herd-effect of pneumococcal conjugate vaccines on invasive pneumococcal disease has been extensively studied. Two systematic reviews identified 262 observational studies published between 1 January 1994 and 6 January 2016 that examined the direct and indirect impact of PCV on vaccine-type and all-cause IPD (Davis et al. 2013; Shiri et al. 2017). No summary of the included studies was attempted by Davis et al. (2013). To our knowledge, only two of the 262 included studies adjusted for pre-vaccine trends (Andrade et al. 2016; Moore et al. 2015).

The publication by Shiri et al. (2017) was also a meta-analysis, which used a Bayesian mixed-effects model to translate the included studies into a single estimate. For each age-group, they reported the yearly risk ratio of IPD among unvaccinated individuals in the post-vaccine period compared to the pre-vaccine period. They also reported the cumulative reduction of IPD as a function of years since PCV introduction and predicted the mean time until a 50% and 90% reduction in IPD was achieved. The study demonstrated a yearly post-vaccine risk ratio of vaccine-type IPD of 0.79 (95% credible intervals 0.75-0.81), translating to a mean period to attain a 50% population reduction of vaccine-type IPD of 2.3 years (95% credible interval 1.9-2.7), and 8.9 years (95% credible interval 7.8-10.3) to attain a 90% reduction. When stratified by age-group, the yearly risk ratio of vaccine-type IPD was 0.77 among adults 65 years of age and older, and the time until 50% and 90% reduction was 4.1 and 10.3 years respectively.

Interestingly, the results for all-cause IPD were different. The yearly risk ratio for all age-groups was 0.99 (95% credible intervals 0.96-0.99) and the time until 50% and 90% reduction were not estimated, presumably due to the credible intervals approaching infinity, though this was not commented on in the paper or supplementary files. When stratified by age-group, a mean 50% reduction of all-cause IPD was achieved 12 years after the introduction of PCV in children younger than five years of age, children five to 18 years of age and adults 19-49 years of age. Among adults 50-64 years of age, the mean reduction at 12 years was 39%, but the 95% credible interval included 50%. However, among adults 65 years of age and older, the mean predicted reduction at 12 years was 30% and the 95% credible intervals did not include 50%.

The indirect effect of PHiD-CV10 on IPD hospitalizations in Iceland was studied in paper VI. Because the data in paper VI were meant to inform a subsequent cost-effectiveness analysis, and serotype replacement is known to blunt the impact of PCV, we only studied all-cause IPD (Weinberger, Malley, and Lipsitch 2011). Because only 338 IPD hospitalizations occurred during the whole study period, we were only able to estimate impact among few age-groups and data were aggregated by year-quarter instead of by month.

We demonstrated that among individuals five to 64 years of age, the introduction of PHiD-CV10 prevented 29 (95% credible interval 1 to 65) cases of IPD serious enough to warrant hospital admission in the first seven years of the immunization program. Before the vaccine introduction, this population experienced 16 IPD hospitalizations per year. The rate ratio between the observed and predicted number of IPD hospitalizations was 0.44 (95% credible intervals 0.31-0.68), which translates to a 56% vaccine impact in this age-group and is compatible with a 99% probability that the impact is equal to or larger than 1%. Examination of the rolling rate ratio presented in Figure 4.34, shows that impact was first detectable in the latter half of 2013, and achieved a 50% reduction in the beginning of 2014.

Conversely, the evidence of herd-effect among adults 65 years of age and older was less clear. The cumulative number of prevented cases were 10 (95% credible intervals -16 to 45) and rate ratio was 0.94 (95% credible intervals 0.62-1.53), which is consistent with a 50% probability that the vaccine impact was 6% or larger in this age-group, and a 62.5% probability that the impact was equal to or larger than 1%. Examination of the rolling rate ratio presented in Figure 4.34 does not seem to suggest that the impact is increasing as time passes from the vaccine introduction.

Our study adds to the current literature by presenting population-based data and taking into account secular trends prior to vaccine introduction. Our findings are largely congruent with previous studies examining the herd-effect of PCV on all-cause IPD. We show a robust indirect protection among individuals five to 64 years of age, after adjusting for any secular trends in the pre-vaccine period, and the result is consistent with visual examination of the raw data (Figure 4.31). This is an important finding for policy decisions, as this age-group represents the population of working adults in any given country. Though our findings are consistent with slight decrease in IPD hospitalizations among adults 65 and older, the effect is not as obvious. While surprising, this result is consistent with the literature which seems to suggest a large and robust impact on vaccine-type IPD in this age-group, but a marginal impact on all-cause IPD (Davis et al. 2013; Shiri et al. 2017). Though one can only speculate, this may be due to a long tradition of 23-valent polysaccharide vaccination in this age-group in Iceland, as demonstrated by our data.

5.5 Cost-effectiveness of introducing PHiD-CV10 into the Icelandic pediatric vaccination program (Paper VI)

The health care system operates under resource constraints and funding must be optimally allocated to maximize the benefit for the population. Cost-effectiveness analyses are one of many tools that can be used to inform policy decisions in this regard (Gray et al. 2011). A large number of cost-effectiveness analyses of pneumococcal conjugate vaccines have been published (Saokaew et al. 2016; Vooren et al. 2014; Wu et al. 2015). Of those, 21 studies examined the cost-effectiveness of PHiD-CV10 or PCV13 (Blank and Szucs 2012; By et al. 2012; Castiglia et al. 2017; Chuck et al. 2010; Delgleize et al. 2016; Díez-Domingo et al. 2011; Earnshaw et al. 2012; Gouveia et al. 2017; Klok et al. 2013; Knerer, Ismaila, and Pearce 2012; Kuhlmann and Schulenburg 2017; Newall et al. 2011, 2016; M. A. O’Brien et al. 2009; Robberstad et al. 2011; Rozenbaum et al. 2010; Rubin et al. 2010; Strutton et al. 2012; Talbird et al. 2010; Hoek, Choi, et al. 2012; Zhou et al. 2014). The methodology employed by each study was reviewed extensively in chapter 1.3. Each found the pneumococcal conjugate vaccine to be cost-effective compared to no vaccination.

We reported the cost-effectiveness of introducing PHiD-CV10 into the Icelandic vaccination program. Cost-effectiveness was evaluated from both the health care and societal perspectives. Though we did directly measure vaccine uptake and serotype coverage, we did not need to explicitly model these variables as the study’s ecological design implicitly captured their effects. The pre- and post-vaccination incidence of otitis media, inpatient pneumonia and IPD were directly measured in the population, and a previously published Bayesian time series methodology was employed to estimate what the incidence would have been, had the vaccine not been introduced (Bruhn et al. 2017; Shioda et al. 2018). These methods resulted in a posterior predictive distribution, which allowed us to estimate uncertainty empirically, and avoid the need for assuming an arbitrary uncertainty distribution.

Costs associated with the administration of the vaccine were directly obtained from the Directorate of Health and not based on assumptions or list pricing. The direct costs associated with the outcome were sampled from individual-level otitis media visits, and hospital admissions for pneumonia and IPD. The distribution was empirically estimated through resampling of the observed costs, again allowing us to avoid assuming arbitrary uncertainty distributions. Days of work lost due to hospitalized pneumonia and IPD were modeled as a function of the individually observed hospital lengths of stay and the distribution was estimated through direct resampling. All cost and outcome data were included in an overall Bayesian model, which propagated the uncertainty of each of the model parameters and produced a posterior distribution of the cost-effectiveness that includes an empirical probabilistic sensitivity analysis.

Our results showed that the introduction of PHiD-CV10 was cost-saving by 7,463,176$ in constant 2015 USD from the health care perspective. The direct cost of introducing the vaccine was 2,652,364$ as of 31 December 2015. However, this cost was offset by the cost-savings associated with averted otitis media visits, pneumonia admissions and hospitalized IPD, which totaled 10,115,540$. When the societal perspective was considered, and averted lost workdays were also included, the vaccine introduction was cost-saving by 8,164,894$. The direct savings resulting from vaccine-prevented cases of AOM was 1,389,900$. The incremental cost-effectiveness ratio per prevented case of AOM was -543$ (95% credible interval -1,508$ to -48$) – that is, the health care system’s monetary gains exceeded the initial expenditure, resulting in each additional case averted saving rather than costing money. Similar numbers were seen for hospitalized disease. The ICER for each additional prevented pneumonia hospitalization was -5,640$ (95% credible interval -10,336$ to -1,032$) and -119,992$ (95% credible interval -387,183$ to -9,542$) per prevented IPD hospitalization.

Our results are quantitatively similar to the body of cost-effectiveness literature of PCV. Most show that introducing PCV into national immunization programs is cost-effective when compared to no vaccination. However, our study improves on prior studies in several important ways. We included more granular data than have previously been incorporated into a cost-effectiveness analysis of PCV. Because they are in essence predictive models, cost-effectiveness analyses are particularly sensitive to the accuracy of the modelling assumptions (Gray et al. 2011). Most of the prior studies did not collect detailed data on vaccine uptake, serotype coverage, incidence of disease in the population, disease sequelae, or direct and indirect costs (Vooren et al. 2014; Wu et al. 2015). Efficacies were based on the results of randomized controlled trials, but the existence and magnitude of herd-effect and serotype-replacement were usually based on assumptions and expert opinion (Vooren et al. 2014; Wu et al. 2015). Utilities were invariably based on studies conducted in other populations and time-periods (Herdman et al. 2016). Contrast this with our study, in which all inputs were directly measured in the population.

Our study included a built in probabilistic sensitivity analysis of all parameters using empirically derived distributions. In general, a sensitivity analysis is necessary to explore the cost-effectiveness outcomes over a range of plausible input parameters, due to the subjective nature of underlying assumptions. Consensus statements from the World Health Organization and the International Society for Pharmacoeconomics and Outcome Research (ISPOR) require, at minimum, a one-way sensitivity analysis of each of the modelling assumptions (Mauskopf et al. 2018; Walker, Hutubessy, and Beutels 2010). Despite the considerable uncertainty associated with utilities, they were often not examined with sensitivity analyses (Blank and Szucs 2012; Chuck et al. 2010; Earnshaw et al. 2012; Gouveia et al. 2017; Klok et al. 2013; Newall et al. 2016; Strutton et al. 2012; Talbird et al. 2010). Similarly, cost inputs that were often purely assumed, based on expert opinion, or based on national tariffs given without any reference, were in many studies not included in a sensitivity analysis (Chuck et al. 2010; Earnshaw et al. 2012; Gouveia et al. 2017; Klok et al. 2013; Newall et al. 2016; Strutton et al. 2012; Talbird et al. 2010).

Our study is inherently different than most previous studies, in that it examines the cost-effectiveness of an intervention that has already been introduced. The most obvious strength of a post-implementation ecological design, is that it absolves the need to rely on untestable assumptions regarding herd-effect and serotype-replacement, which are instead directly observed. To our knowledge, only one previous study has reported the post-implementation cost-effectiveness of PCV (Newall et al. 2016). They used a time series methodology, but with access to only three years of annual pre-vaccine incidence rates, they were unfortunately only able to make crude adjustments and were not able to leverage the strengths of the post-implementation approach, as they acknowledge in the discussion chapter.

Our study improves on the current literature by providing post-implementation evidence of the cost-effectiveness of PHiD-CV10. We demonstrate a large cost-savings that is robust with regards to extensive sensitivity analysis of all parameters. Our results demonstrate that initially expensive vaccine interventions can be shown to produce such a decrease in health care consumption, that the resulting cost-savings offset the initial cost – all the while resulting in reduced suffering in the population. Our study highlights the importance of careful post-implementation studies; both as a tool to validate and calibrate the predictions made by pre-implementation cost-effectiveness studies, which rely heavily on unverifiable assumptions, and to provide evidence of vaccine benefit for policy makers.

References

Thornton, Ruth B, Paul J Rigby, Selma P Wiertsema, Pierre Filion, Jennifer Langlands, Harvey L Coates, Shyan Vijayasekaran, Anthony D Keil, and Peter C Richmond. 2011. “Multi-species bacterial biofilm and intracellular infection in otitis media.” BMC Pediatrics 11 (1): 94. https://doi.org/10.1186/1471-2431-11-94.

Blank, Sarah J, David J Grindler, Kristine A Schulz, David L Witsell, and Judith E C Lieu. 2014. “Caregiver Quality of Life Is Related to Severity of Otitis Media in Children.” Otolaryngology-Head and Neck Surgery 151 (2): 348–53. https://doi.org/10.1177/0194599814531912.

Fortanier, Alexandre C, Roderick P Venekamp, Marieke L A de Hoog, Cuno S P M Uiterwaal, Anne C van der Gugten, Cornelis K van der Ent, Arno W Hoes, and Anne G M Schilder. 2015. “Parent-Reported Symptoms of Acute Otitis Media during the First Year of Life: What Is beneath the Surface?” Edited by Herminia de Lencastre. PLOS ONE 10 (4): e0121572. https://doi.org/10.1371/journal.pone.0121572.

Arguedas, A., K. Kvaerner, J. Liese, A. G.M. Schilder, and S. I. Pelton. 2010. “Otitis media across nine countries: Disease burden and management.” International Journal of Pediatric Otorhinolaryngology 74 (12): 1419–24. https://doi.org/10.1016/j.ijporl.2010.09.022.

Marchisio, Paola, Luigi Cantarutti, Miriam Sturkenboom, Silvia Girotto, Gino Picelli, Daniele Dona, Antonio Scamarcia, Marco Villa, Carlo Giaquinto, and Pedianet. 2012. “Burden of acute otitis media in primary care pediatrics in Italy: a secondary data analysis from the Pedianet database.” BMC Pediatrics 12 (1): 185. https://doi.org/10.1186/1471-2431-12-185.

Monasta, Lorenzo, Luca Ronfani, Federico Marchetti, Marcella Montico, Liza Vecchi Brumatti, Alessandro Bavcar, Domenico Grasso, et al. 2012. “Burden of disease caused by otitis media: Systematic review and global estimates.” PLoS ONE 7 (4): e36226. https://doi.org/10.1371/journal.pone.0036226.

Baraibar, Román. 1997. “Incidence and risk factors of acute otitis media in children.” Clinical Microbiology and Infection : The Official Publication of the European Society of Clinical Microbiology and Infectious Diseases 3 Suppl 3: S13–S22. https://doi.org/10.1016/S1198-743X(14)64947-8.

Macintyre, EA, CJ Karr, M Koehoorn, P Demers, L Tamburic, C Lencar, and Michael Brauer. 2010. “Otitis media incidence and risk factors in a population-based birth cohort.” Paediatrics and Child Health 15 (7): 437–42.

Paradise, Jack L., Howard E. Rockette, D. Kathleen Colborn, Beverly S. Bernard, Clyde G. Smith, Marcia Kurs-Lasky, and Janine E. Janosky. 1997. “Otitis media in 2253 Pittsburgh-area infants: prevalence and risk factors during the first two years of life.” Pediatrics 99 (3): 318–33. http://pediatrics.aappublications.org/content/99/3/318.long{\#}T2 http://www.ncbi.nlm.nih.gov/pubmed/9041282.

Hoog, Marieke La a de, Roderick P Venekamp, Cornelis K van der Ent, Anne Schilder, Elisabeth Am M Sanders, Roger Amj M J Damoiseaux, Debby Bogaert, Cuno Spm P M Uiterwaal, Henriette a Smit, and Patricia Bruijning-Verhagen. 2014. “Impact of early daycare on healthcare resource use related to upper respiratory tract infections during childhood: prospective WHISTLER cohort study.” BMC Medicine 12 (1): 107. https://doi.org/10.1186/1741-7015-12-107.

Adam, D, and K Fehnle. 2008. “Safety and effectiveness against respiratory tract infections for pneumococcal conjugate vaccine co-administered with routine vaccine combinations.” Vaccine 26 (47): 5944–51. https://doi.org/10.1016/j.vaccine.2008.08.058.

Esposito, Susanna, Alessandro Lizioli, Annalisa Lastrico, Enrica Begliatti, Alessandro Rognoni, Claudia Tagliabue, Laura Cesati, Vittorio Carreri, and Nicola Principi. 2007. “Impact on respiratory tract infections of heptavalent pneumococcal conjugate vaccine administered at 3, 5 and 11 months of age.” Respiratory Research 8 (January): 12. https://doi.org/10.1186/1465-9921-8-12.

Gisselsson-Solen, Marie. 2017. “Trends in Otitis Media Incidence After Conjugate Pneumococcal Vaccination; A National Observational Study.” The Pediatric Infectious Disease Journal 36 (11): 1. https://doi.org/10.1097/INF.0000000000001654.

Gribben, Barry, Lesley J Salkeld, Simon Hoare, and Hannah F Jones. 2012. “The incidence of acute otitis media in New Zealand children under five years of age in the primary care setting.” Journal of Primary Health Care 4 (3): 205–12. http://www.ncbi.nlm.nih.gov/pubmed/22946068.

Lau, Wallis C Y, Macey Murray, Aisha El-Turki, Sonia Saxena, Shamez Ladhani, Paul Long, Mike Sharland, Ian C K Wong, and Yingfen Hsia. 2015. “Impact of pneumococcal conjugate vaccines on childhood otitis media in the United Kingdom.” Vaccine 33 (39): 5072–9. https://doi.org/10.1016/j.vaccine.2015.08.022.

Liese, J. G., S. A. Silfverdal, C. Giaquinto, A. Carmona, J. H. Larcombe, J. Garcia-Sicilia, A. Fuat, et al. 2014. “Incidence and clinical presentation of acute otitis media in children aged <6 years in European medical practices.” Epidemiology and Infection 142 (8): 1778–88. https://doi.org/10.1017/S0950268813002744.

Todberg, Tanja, Anders Koch, Mikael Andersson, Sjurdur F. SF F Olsen, Jørgen Lous, Preben Homøe, W Nystad, et al. 2014. “Incidence of otitis media in a contemporary danish national birth cohort.” Edited by C M Schooling. PloS One 9 (12): e111732. https://doi.org/10.1371/journal.pone.0111732.

Usonis, Vytautas, Teresa Jackowska, Sigita Petraitiene, Alicja Sapala, Andrea Neculau, Izabella Stryjewska, Raghavendra Devadiga, Monica Tafalla, and Katsiaryna Holl. 2016. “Incidence of acute otitis media in children below 6 years of age seen in medical practices in five East European countries.” BMC Pediatrics 16 (1): 108. https://doi.org/10.1186/s12887-016-0638-2.

De Wals, Philippe, Steven Black, Ray Borrow, and David Pearce. 2009. “Modeling the impact of a new vaccine on pneumococcal and nontypable Haemophilus influenzae diseases: A new simulation model.” Clinical Therapeutics 31 (10): 2152–69. https://doi.org/10.1016/j.clinthera.2009.10.014.

Grijalva, C. G., K. A. Poehling, J. P. Nuorti, Y. Zhu, S. W. Martin, K. M. Edwards, and M. R. Griffin. 2006. “National Impact of Universal Childhood Immunization With Pneumococcal Conjugate Vaccine on Outpatient Medical Care Visits in the United States.” PEDIATRICS 118 (3): 865–73. https://doi.org/10.1542/peds.2006-0492.

Grijalva, Carlos G, J Pekka Nuorti, and Marie R Griffin. 2009. “Antibiotic prescription rates for acute respiratory tract infections in US ambulatory settings.” JAMA : The Journal of the American Medical Association 302 (7): 758–66. https://doi.org/10.1001/jama.2009.1163.

Poehling, K. A. 2004. “Population-Based Impact of Pneumococcal Conjugate Vaccine in Young Children.” PEDIATRICS 114 (3): 755–61. https://doi.org/10.1542/peds.2003-0592-F.

Zhou, Fangjun, Abigail Shefer, Yuan Kong, and J Pekka Nuorti. 2008. “Trends in acute otitis media-related health care utilization by privately insured young children in the United States, 1997-2004.” Pediatrics 121 (2): 253–60. https://doi.org/10.1542/peds.2007-0619.

Bjarnason, Skúli, Ingþór Friðriksson, and Jón Benediktsson. 1991. “Tíðni bráðrar miðeyrabólgu hjá börnum á svæði heilsugæslustöðvarinnar Borgarnesi.” Læknablaðið 77 (4): 137–40. http://www.hirsla.lsh.is/lsh/bitstream/2336/90415/1/L1991-04-77-F3.pdf.

Teele, D W, J O Klein, and B Rosner. 1989. “Epidemiology of otitis media during the first seven years of life in children in greater Boston: a prospective, cohort study.” The Journal of Infectious Diseases 160 (1): 83–94. http://www.ncbi.nlm.nih.gov/pubmed/2732519.

Bluestone, C D, J S Stephenson, and L M Martin. 1992. “Ten-year review of otitis media pathogens.” The Pediatric Infectious Disease Journal 11 (8 Suppl): S7–11. https://doi.org/10.1097/00006454-199208001-00002.

Casey, Janet R., and Michael E. Pichichero. 2004. “Changes in frequency and pathogens causing acute otitis media in 1995-2003.” The Pediatric Infectious Disease Journal 23 (9): 824–8. https://doi.org/10.1097/01.inf.0000136871.51792.19.

Casey, Janet R, Diana G Adlowitz, and Michael E Pichichero. 2009. “New Patterns in the Otopathogens Causing Acute Otitis Media Six to Eight Years After Introduction of Pneumococcal Conjugate Vaccine.” The Pediatric Infectious Disease Journal 29 (November): 1. https://doi.org/10.1097/INF.0b013e3181c1bc48.

Ngo, Chinh C., Helen M. Massa, Ruth B. Thornton, and Allan W. Cripps. 2016. “Predominant Bacteria Detected from the Middle Ear Fluid of Children Experiencing Otitis Media: A Systematic Review.” Edited by Sean Reid. PloS One 11 (3): e0150949. https://doi.org/10.1371/journal.pone.0150949.

Pumarola, Felix, Josep Marès, Isabel Losada, Isabel Minguella, Fernando Moraga, David Tarragó, Ulla Aguilera, et al. 2013. “Microbiology of bacteria causing recurrent acute otitis media (AOM) and AOM treatment failure in young children in Spain: Shifting pathogens in the post-pneumococcal conjugate vaccination era.” International Journal of Pediatric Otorhinolaryngology 77 (8): 1231–6. https://doi.org/10.1016/j.ijporl.2013.04.002.

Quirk, Sigríður J, Gunnsteinn Haraldsson, Helga Erlendsdóttir, Martha Á Hjálmarsdóttir, Andries J van Tonder, Birgir Hrafnkelsson, Samuel Sigurdsson, et al. 2018. “Effect of Vaccination on Pneumococci Isolated from the Nasopharynx of Healthy Children and the Middle Ear of Children with Otitis Media in Iceland.” Edited by Daniel J. Diekema. Journal of Clinical Microbiology 56 (12): 1–14. https://doi.org/10.1128/JCM.01046-18.

Hadley, Jack. 2003. “Sicker and Poorer—The Consequences of Being Uninsured: A Review of the Research on the Relationship between Health Insurance, Medical Care Use, Health, Work, and Income.” Medical Care Research and Review 60 (2_suppl): 3S–75S. https://doi.org/10.1177/1077558703254101.

Smolderen, Kim G. 2010. “Health Care Insurance, Financial Concerns in Accessing Care, and Delays to Hospital Presentation in Acute Myocardial Infarction.” JAMA 303 (14): 1392. https://doi.org/10.1001/jama.2010.409.

Eurostat. 2018. “Employment statistics.” https://ec.europa.eu/eurostat/statistics-explained/index.php/Employment{\_}statistics.

OECD. 2019. “Employment rate (indicator).” https://doi.org/10.1787/1de68a9b-en.

“Enrolment in childcare and pre-schools.” 2013. Organisation for economic co-operation; development. http://www.oecd.org/social/soc/oecdfamilydatabase.htm.

Ramakrishnan, Kalyanakrishnan, Rhonda A Sparks, and Wayne E Berryhill. 2007. “Diagnosis and treatment of otitis media.” American Family Physician 76 (11): 1650–8. http://www.ncbi.nlm.nih.gov/pubmed/18092706.

Black, S, H Shinefield, Bruce Fireman, E Lewis, P Ray, J R Hansen, L Elvin, et al. 2000. “Efficacy, safety and immunogenicity of heptavalent pneumococcal conjugate vaccine in children. Northern California Kaiser Permanente Vaccine Study Center Group.” The Pediatric Infectious Disease Journal 19 (3): 187–95. http://www.ncbi.nlm.nih.gov/pubmed/10749457.

Dagan, Ron, M Sikuler-Cohen, O Zamir, J Janco, Noga Givon-Lavi, and D Fraser. 2001. “Effect of a conjugate pneumococcal vaccine on the occurrence of respiratory infections and antibiotic use in day-care center attendees.” Pediatr Infect Dis J 20 (10): 951–58. http://www.ncbi.nlm.nih.gov/pubmed/11642629.

Eskola, J, T. Kilpi, A. Palmu, J. Jokinen, J. Haapakoski, E. Herva, A. Takala, et al. 2001. “Efficacy of a pneumococcal conjugate vaccine against acute otitis media.” N Engl J Med 344 (6): 403–9. https://doi.org/10.1056/nejm200102083440602.

Fireman, Bruce, Steven B Black, Henry R Shinefield, Janelle Lee, Edwin Lewis, and Paula Ray. 2003. “Impact of the pneumococcal conjugate vaccine on otitis media.” 1. 2003/01/25. Vol. 22. The Pediatric infectious disease journal. https://doi.org/10.1097/00006454-200301000-00006.

Kilpi, T, Heidi Ahman, J Jokinen, K S Lankinen, A Palmu, H Savolainen, M Gronholm, et al. 2003. “Protective efficacy of a second pneumococcal conjugate vaccine against pneumococcal acute otitis media in infants and children: randomized, controlled trial of a 7-valent pneumococcal polysaccharide-meningococcal outer membrane protein complex conjugate v.” Clin Infect Dis 37 (9): 1155–64. https://doi.org/10.1086/378744.

O’Brien, Katherine L., Angeline B David, Aruna Chandran, Lawrence H Moulton, Raymond Reid, Robert Weatherholtz, and Mathuram Santosham. 2008. “RANDOMIZED, CONTROLLED TRIAL EFFICACY OF PNEUMOCOCCAL CONJUGATE VACCINE AGAINST OTITIS MEDIA AMONG NAVAJO AND WHITE MOUNTAIN APACHE INFANTS.” The Pediatric Infectious Disease Journal 27 (1): 71–73. https://doi.org/10.1097/INF.0b013e318159228f.

Palmu, Arto, Jukka Jokinen, Heta Nieminen, Hanna Rinta-Kokko, Esa Ruokokoski, Taneli Puumalainen, Magali Traskine, et al. 2015. “Effectiveness of the 10-Valent Pneumococcal Conjugate Vaccine Against Tympanostomy Tube Placements in a Cluster-Randomized Trial.” The Pediatric Infectious Disease Journal 34 (11): 1230–5. https://doi.org/http://dx.doi.org/10.1097/INF.0000000000000857.

Prymula, Roman, Pascal Peeters, Viktor Chrobok, Pavla Kriz, Elena Novakova, Eva Kaliskova, Igor Kohl, et al. 2006. “Pneumococcal capsular polysaccharides conjugated to protein D for prevention of acute otitis media caused by both Streptococcus pneumoniae and non-typable Haemophilus influenzae: A randomised double-blind efficacy study.” Lancet 367 (9512): 740–48. https://doi.org/10.1016/S0140-6736(06)68304-9.

Tregnaghi, Miguel W. Marcelo, Xavier Sáez-Llorens, Pio López, Hector Abate, Enrique Smith, Adriana Pósleman, Arlene Calvo, et al. 2014. “Efficacy of pneumococcal nontypable Haemophilus influenzae protein D conjugate vaccine (PHiD-CV) in young Latin American children: A double-blind randomized controlled trial.” Edited by Elizabeth Miller. PLoS Medicine 11 (6): e1001657. https://doi.org/10.1371/journal.pmed.1001657.

Vesikari, Timo, Aino Forsten, Ilkka Seppa, Tarja Kaijalainen, Taneli Puumalainen, Anu Soininen, Magali Traskine, et al. 2016. “Effectiveness of the 10-valent pneumococcal nontypeable Haemophilus influenzae protein D-conjugated vaccine (PHiD-CV) against carriage and acute otitis media-a double-blind randomized clinical trial in Finland.” Journal of the Pediatric Infectious Diseases Society 5 (3). https://doi.org/10.1093/jpids/piw010.

Ben-Shimol, Shalom, Noga Givon-Lavi, Eugene Leibovitz, Simon Raiz, David Greenberg, and Ron Dagan. 2014. “Near-Elimination of Otitis Media Caused by 13-Valent Pneumococcal Conjugate Vaccine (PCV) Serotypes in Southern Israel Shortly After Sequential Introduction of 7-Valent/13-Valent PCV.” Clinical Infectious Diseases 59 (12): 1724–32. https://doi.org/10.1093/cid/ciu683.

Poehling, Katherine A, Peter G Szilagyi, Carlos G Grijalva, Stacey W Martin, Bonnie LaFleur, Ed Mitchel, Richard D Barth, J Pekka Nuorti, and Marie R Griffin. 2007. “Reduction of Frequent Otitis Media and Pressure-Equalizing Tube Insertions in Children After Introduction of Pneumococcal Conjugate Vaccine.” PEDIATRICS 119 (4): 707–15. https://doi.org/10.1542/peds.2006-2138.

Magnus, Maria C, Didrik F Vestrheim, Wenche Nystad, Siri Eldevik Håberg, Hein Stigum, Stephanie J London, Marianne R A R Bergsaker, Dominique A Caugant, Ingeborg S Aaberge, and Per Nafstad. 2012. “Decline in early childhood respiratory tract infections in the Norwegian mother and child cohort study after introduction of pneumococcal conjugate vaccination.” The Pediatric Infectious Disease Journal 31 (9): 951–5. https://doi.org/10.1097/INF.0b013e31825d2f76.

Marom, Tal, Alai Tan, Gregg S. Wilkinson, Karen S. Pierson, Jean L. Freeman, and Tasnee Chonmaitree. 2014. “Trends in otitis media-related health care use in the United States, 2001-2011.” JAMA Pediatrics 168 (1): 68–75. https://doi.org/10.1001/jamapediatrics.2013.3924.

Singleton, Rosalyn J., Robert C. Holman, Randall Plant, Krista L. Yorita, Steve Holve, Edna L. Paisano, and James E. Cheek. 2009. “Trends in otitis media and myringtomy with tube placement among American Indian/Alaska native children and the US general population of children.” The Pediatric Infectious Disease Journal 28 (2): 102–7. https://doi.org/http://dx.doi.org/10.1097/INF.0b013e318188d079.

Bernal, James Lopez, Steven Cummins, and Antonio Gasparrini. 2016. “Interrupted time series regression for the evaluation of public health interventions: a tutorial.” International Journal of Epidemiology, 1–8. https://doi.org/10.1093/ije/dyw098.

Kontopantelis, Evangelos, Tim Doran, David A Springate, Iain Buchan, and David Reeves. 2015. “Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis.” BMJ (Clinical Research Ed.) 350 (January): h2750. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4460815{\&}tool=pmcentrez{\&}rendertype=abstract.

Bruhn, Christian A W, Stephen Hetterich, Cynthia Schuck-Paim, Esra Kürüm, Robert J Taylor, Roger Lustig, Eugene D Shapiro, Joshua L Warren, Lone Simonsen, and Daniel M Weinberger. 2017. “Estimating the population-level impact of vaccines using synthetic controls.” Proceedings of the National Academy of Sciences 114 (7): 1524–9. https://doi.org/10.1073/pnas.1612833114.

Shioda, Kayoko, Cynthia Schuck-Paim, Robert J. Taylor, Roger Lustig, Lone Simonsen, Joshua L Warren, and Daniel M Weinberger. 2018. “Challenges in estimating the impact of vaccination with sparse data.” Epidemiology (Cambridge, Mass.) 4 (5): 1. https://doi.org/10.1097/EDE.0000000000000938.

Ahmed, Sameer, Nina L Shapiro, and Neil Bhattacharyya. 2014. “Incremental health care utilization and costs for acute otitis media in children.” The Laryngoscope 124 (1): 301–5. https://doi.org/10.1002/lary.24190.

Cullen, Karen A, Margaret J Hall, and Aleksandr Golosinskiy. 2009. “Ambulatory surgery in the United States, 2006.” National Health Statistics Reports, no. 11 (January): 1–25. http://www.ncbi.nlm.nih.gov/pubmed/19294964.

Vlastarakos, Petros V., Thomas P. Nikolopoulos, Stavros Korres, Evangelia Tavoulari, Antonios Tzagaroulakis, and Eleftherios Ferekidis. 2007. “Grommets in otitis media with effusion: the most frequent operation in children. But is it associated with significant complications?” European Journal of Pediatrics 166 (5): 385–91. https://doi.org/10.1007/s00431-006-0367-x.

Finnbogadóttir, Anna Freyja, Hannes Petersen, Þröstur Laxdal, Fridrik Gudbrandsson, Þórólfur Gudnason, and Ásgeir Haraldsson. 2009. “An increasing incidence of mastoiditis in children in Iceland.” Scandinavian Journal of Infectious Diseases 41 (2): 95–98. https://doi.org/10.1080/00365540802593461.

Groth, Anita, Frida Enoksson, Ann Hermansson, Malou Hultcrantz, Joacim Stalfors, and Karin Stenfeldt. 2011. “Acute mastoiditis in children in Sweden 1993-2007-No increase after new guidelines.” International Journal of Pediatric Otorhinolaryngology 75 (12): 1496–1501. https://doi.org/10.1016/j.ijporl.2011.08.015.

Vojtek, Ivo, Marcus Nordgren, and Bernard Hoet. 2017. “Impact of pneumococcal conjugate vaccines on otitis media: A review of measurement and interpretation challenges.” International Journal of Pediatric Otorhinolaryngology 100 (September): 174–82. https://doi.org/10.1016/j.ijporl.2017.07.009.

Blommaert, A, C Marais, N Hens, S Coenen, A Muller, H Goossens, and P Beutels. 2014. “Determinants of between-country differences in ambulatory antibiotic use and antibiotic resistance in Europe: a longitudinal observational study.” The Journal of Antimicrobial Chemotherapy 69 (2): 535–47. https://doi.org/10.1093/jac/dkt377.

Bruyndonckx, R., N. Hens, M. Aerts, H. Goossens, J. Cortinas Abrahantes, and S. Coenen. 2015. “Exploring the association between resistance and outpatient antibiotic use expressed as DDDs or packages.” Journal of Antimicrobial Chemotherapy 70 (4): 1241–4. https://doi.org/10.1093/jac/dku525.

Costelloe, Céire, Chris Metcalfe, Andrew Lovering, David Mant, and Alastair D Hay. 2010. “Effect of antibiotic prescribing in primary care on antimicrobial resistance in individual patients: systematic review and meta-analysis.” BMJ 340. http://www.bmj.com/content/340/bmj.c2096.long.

Goossens, Herman, Matus Ferech, Robert Vander Stichele, Monique Elseviers, and ESAC Project Group. 2005. “Outpatient antibiotic use in Europe and association with resistance: a cross-national database study.” Lancet (London, England) 365 (9459): 579–87. https://doi.org/10.1016/S0140-6736(05)17907-0.

Austin, D J, K G Kristinsson, and R M Anderson. 1999. “The relationship between the volume of antimicrobial consumption in human communities and the frequency of resistance.” Proceedings of the National Academy of Sciences of the United States of America 96 (3): 1152–6. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=15366{\&}tool=pmcentrez{\&}rendertype=abstract.

Palmu, Arto A, Jukka Jokinen, Heta Nieminen, Hanna Rinta-Kokko, Esa Ruokokoski, Taneli Puumalainen, Dorota Borys, et al. 2014. “Effect of pneumococcal Haemophilus influenzae protein D conjugate vaccine (PHiD-CV10) on outpatient antimicrobial purchases: a double-blind, cluster randomised phase 3-4 trial.” The Lancet Infectious Diseases 14 (3): 205–12. https://doi.org/10.1016/S1473-3099(13)70338-4.

Gefenaite, G, M J Bijlsma, H J Bos, and E Hak. 2014. “Did introduction of pneumococcal vaccines in the Netherlands decrease the need for respiratory antibiotics in children ? Analysis of 2002 to 2013 data.” Eurosurveillance 19 (44): 20948. https://doi.org/10.2807/1560-7917.ES2014.19.44.20948.

Howitz, Michael Frantz, Zitta Barrella Harboe, Helene Ingels, Palle Valentiner-Branth, Kåre Mølbak, and Bjarki Ditlev Djurhuus. 2017. “A nationwide study on the impact of pneumococcal conjugate vaccination on antibiotic use and ventilation tube insertion in Denmark 2000-2014.” Vaccine 35 (43): 5858–63. https://doi.org/10.1016/j.vaccine.2017.09.006.

Tsaban, Gal, and Shalom Ben-Shimol. 2017. “Indirect (herd) protection, following pneumococcal conjugated vaccines introduction: A systematic review of the literature.” Vaccine 35 (22): 2882–91. https://doi.org/10.1016/j.vaccine.2017.04.032.

Loo, Jennifer D, Laura Conklin, Katherine E Fleming-Dutra, Maria Deloria Knoll, Daniel E Park, Jennifer Kirk, David Goldblatt, Katherine L O’Brien, and Cynthia G Whitney. 2014. “Systematic review of the indirect effect of pneumococcal conjugate vaccine dosing schedules on pneumococcal disease and colonization.” The Pediatric Infectious Disease Journal 33 Suppl 2 (1): S161–71. https://doi.org/10.1097/INF.0000000000000084.

Nicholls, Thomas Rodger, Amanda Jane Leach, and Peter Stanley Morris. 2016. “The short-term impact of each primary dose of pneumococcal conjugate vaccine on nasopharyngeal carriage: Systematic review and meta-analyses of randomised controlled trials.” Vaccine 34 (6): 703–13. https://doi.org/10.1016/j.vaccine.2015.12.048.

Ben-Shimol, Shalom, Noga Givon-Lavi, Eugene Leibovitz, Simon Raiz, David Greenberg, and Ron Dagan. 2016. “Impact of Widespread Introduction of Pneumococcal Conjugate Vaccines on Pneumococcal and Nonpneumococcal Otitis Media.” Clinical Infectious Diseases : An Official Publication of the Infectious Diseases Society of America 63 (5): 611–8. https://doi.org/10.1093/cid/ciw347.

Greenberg, David, Natasha Bilenko, Zvika Liss, Tomer Shagan, Orly Zamir, and Ron Dagan. 2003. “The burden of acute otitis media on the patient and the family.” European Journal of Pediatrics 162 (9): 576–81. https://doi.org/10.1007/s00431-003-1260-5.

Pichichero, Michael E., Janet R. Casey, Alejandro Hoberman, and Richard Schwartz. 2008. “Pathogens causing recurrent and difficult-to-treat acute otitis media, 2003-2006.” Clinical Pediatrics 47 (9): 901–6. https://doi.org/10.1177/0009922808319966.

Black, Nick. 1984. “Surgery for glue ear—A modern epidemic.” The Lancet 323 (8381): 835–37. https://doi.org/10.1016/S0140-6736(84)92280-3.

Venekamp, Roderick P, Paul Mick, Anne GM Schilder, and Desmond A Nunez. 2018. “Grommets (ventilation tubes) for recurrent acute otitis media in children.” Cochrane Database of Systematic Reviews (4):CD0047 (4): CD004741. https://doi.org/10.1002/14651858.CD012017.pub2.

Browning, George G, Maroeska M Rovers, Ian Williamson, Jørgen Lous, and Martin J Burton. 2010. “Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children.” Cochrane Database of Systematic Reviews, no. 10 (October). https://doi.org/10.1002/14651858.CD001801.pub3.

Jardine, Andrew, Robert I. Menzies, Shelley L. Deeks, Mahomed S. Patel, and Peter B. McIntyre. 2009. “The impact of pneumococcal conjugate vaccine on rates of myringotomy with ventilation tube insertion in Australia.” The Pediatric Infectious Disease Journal 28 (9): 761–5. https://doi.org/10.1097/INF.0b013e31819e9bc5.

Madhi, Shabir a, Philippe De Wals, Carlos G Grijalva, Keith Grimwood, Ronald Grossman, Naruhiko Ishiwada, Ping-Ing Lee, et al. 2012. “The Burden of Childhood Pneumonia in the Developed World.” The Pediatric Infectious Disease Journal 32 (3): 1. https://doi.org/10.1097/INF.0b013e3182784b26.

Torné, Adoración Navarro, Joana Gomes Dias, Chantal Quinten, Frantiska Hruba, Marta Cecilia Busana, Pier Luigi Lopalco, Andrew J Amato Gauci, and Lucia Pastore-Celentano. 2014. “European enhanced surveillance of invasive pneumococcal disease in 2010: Data from 26 European countries in the post-heptavalent conjugate vaccine era.” Vaccine 32 (29): 3644–50. https://doi.org/10.1016/j.vaccine.2014.04.066.

Durando, Paolo, Pietro Crovari, Filippo Ansaldi, Laura Sticchi, Camilla Sticchi, Valter Turello, Lorenzo Marensi, Raffaella Giacchino, Anna Timitilli, and Roberto Carloni. 2009. “Universal childhood immunisation against Streptococcus pneumoniae: The five-year experience of Liguria Region, Italy.” Vaccine 27 (25-26): 3459–62. https://doi.org/10.1016/j.vaccine.2009.01.052.

Marom, Tal, Ofer Israel, Haim Gavriel, Jacob Pitaro, Ali Abo Baker, and Ephraim Eviatar. 2017. “Comparison of first year of life acute otitis media admissions before and after the 13-valent pneumococcal conjugate vaccine.” International Journal of Pediatric Otorhinolaryngology 97 (June): 251–56. https://doi.org/10.1016/j.ijporl.2017.04.023.

Tawfik, Kareem O., Stacey L. Ishman, Mekibib Altaye, Jareen Meinzen-Derr, and Daniel I. Choo. 2017. “Pediatric Acute Otitis Media in the Era of Pneumococcal Vaccination.” Otolaryngology-Head and Neck Surgery 156 (5): 938–45. https://doi.org/10.1177/0194599817699599.

Black, Steven B, H R Shinefield, S Ling, J Hansen, Bruce Fireman, D Spring, J Noyes, et al. 2002. “Effectiveness of heptavalent pneumococcal conjugate vaccine in children younger than five years of age for prevention of pneumonia.” Pediatr Infect Dis J 21 (9): 810–15. https://doi.org/10.1097/01.inf.0000027926.99356.4c.

Cutts, Felicity T, S M Zaman, Godwin C Enwere, S Jaffar, O S Levine, J B Okoko, C Oluwalana, et al. 2005. “Efficacy of nine-valent pneumococcal conjugate vaccine against pneumonia and invasive pneumococcal disease in The Gambia: randomised, double-blind, placebo-controlled trial.” Lancet 365 (9465): 1139–46. https://doi.org/10.1016/s0140-6736(05)71876-6.

Kilpi, T. M., J. Jokinen, T. Puumalainen, H. Nieminen, E. Ruokokoski, H. Rinta-Kokko, M. Traskine, et al. 2018. “Effectiveness of pneumococcal Haemophilus influenzae protein D conjugate vaccine against pneumonia in children: A cluster-randomised trial.” Vaccine 36 (39): 5891–5901. https://doi.org/10.1016/j.vaccine.2018.08.020.

Alicino, Cristiano, Chiara Paganino, Andrea Orsi, Matteo Astengo, Cecilia Trucchi, Giancarlo Icardi, and Filippo Ansaldi. 2017. “The impact of 10-valent and 13-valent pneumococcal conjugate vaccines on hospitalization for pneumonia in children: A systematic review and meta-analysis.” Vaccine 35 (43): 5776–85. https://doi.org/10.1016/j.vaccine.2017.09.005.

Berglund, Anders, Mats Ekelund, Mark A. Fletcher, and Lars Nyman. 2014. “All-cause pneumonia hospitalizations in children <2 years old in sweden, 1998 to 2012: impact of pneumococcal conjugate vaccine introduction.” Edited by Bernard Beall. PloS One 9 (11): e112211. https://doi.org/10.1371/journal.pone.0112211.

Nair, Harish, Arun Thor Watts, Linda J. Williams, Saad B. Omer, Colin R. Simpson, Lorna J. Willocks, J. Claire Cameron, and Harry Campbell. 2016. “Pneumonia hospitalisations in Scotland following the introduction of pneumococcal conjugate vaccination in young children.” BMC Infectious Diseases 16 (1): 390. https://doi.org/10.1186/s12879-016-1693-x.

Saxena, Sonia, Christina Atchison, Elizabeth Cecil, Mike Sharland, Elizabeth Koshy, and Alex Bottle. 2015. “Additive impact of pneumococcal conjugate vaccines on pneumonia and empyema hospital admissions in England.” Journal of Infection 71 (4): 428–36. https://doi.org/10.1016/j.jinf.2015.06.011.

Sgambatti, Sabrina, Ruth Minamisava, Ana Luiza Bierrenbach, Cristiana Maria Toscano, Maria Aparecida Vieira, Gabriela Policena, and Ana Lucia Andrade. 2016. “Early impact of 10-valent pneumococcal conjugate vaccine in childhood pneumonia hospitalizations using primary data from an active population-based surveillance.” Vaccine 34 (5): 663–70. https://doi.org/10.1016/j.vaccine.2015.12.007.

Simonsen, Lone, Robert J Taylor, Cynthia Schuck-Paim, Roger Lustig, Michael Haber, and Keith P Klugman. 2014. “Effect of 13-valent pneumococcal conjugate vaccine on admissions to hospital 2 years after its introduction in the USA: a time series analysis.” The Lancet Respiratory Medicine 2 (5): 387–94. https://doi.org/10.1016/S2213-2600(14)70032-3.

O’Brien, Katherine L, Lawrence H Moulton, Raymond Reid, Robert Weatherholtz, Jane Oski, Laura Brown, Gaurav Kumar, et al. 2003. “Efficacy and safety of seven-valent conjugate pneumococcal vaccine in American Indian children: group randomised trial.” Lancet 362 (9381): 355–61. https://doi.org/10.1016/S0140-6736(03)14022-6.

Palmu, Arto A, Jukka Jokinen, Dorota Borys, Heta Nieminen, Esa Ruokokoski, Lotta Siira, Taneli Puumalainen, et al. 2013. “Effectiveness of the ten-valent pneumococcal Haemophilus influenzae protein D conjugate vaccine (PHiD-CV10) against invasive pneumococcal disease: a cluster randomised trial.” Lancet 381 (9862): 214–22. https://doi.org/10.1016/s0140-6736(12)61854-6.

Myint, Tin Tin Htar, Harish Madhava, Paul Balmer, Dina Christopoulou, Sepideh Attal, Damianos Menegas, Ralf Sprenger, and Eric Bonnet. 2013. “The impact of 7-valent pneumococcal conjugate vaccine on invasive pneumococcal disease: a literature review.” Advances in Therapy 30 (2): 127–51. https://doi.org/10.1007/s12325-013-0007-6.

Andrade, Ana Lucia, Eliane T Afonso, Ruth Minamisava, Ana Luiza Bierrenbach, Elier B Cristo, Otaliba L Morais-Neto, Gabriela M Policena, Carla M. A. S. Domingues, and Cristiana M Toscano. 2017. “Direct and indirect impact of 10-valent pneumococcal conjugate vaccine introduction on pneumonia hospitalizations and economic burden in all age-groups in Brazil: A time-series analysis.” Edited by Philip C Hill. PLOS ONE 12 (9): e0184204. https://doi.org/10.1371/journal.pone.0184204.

Griffin, Marie R., Yuwei Zhu, Matthew R. Moore, Cynthia G. Whitney, and Carlos G. Grijalva. 2013. “U.S. hospitalizations for pneumonia after a decade of pneumococcal vaccination.” The New England Journal of Medicine 369 (2): 155–63. https://doi.org/10.1056/NEJMoa1209165.

Grijalva, Carlos G., J. Pekka Nuorti, Patrick G. Arbogast, Stacey W. Martin, Kathryn M. Edwards, and Marie R. Griffin. 2007. “Decline in pneumonia admissions after routine childhood immunisation with pneumococcal conjugate vaccine in the USA: a time-series analysis.” Lancet 369 (9568): 1179–86. https://doi.org/10.1016/S0140-6736(07)60564-9.

Jardine, Andrew, Robert I. Menzies, and Peter B. McIntyre. 2010. “Reduction in Hospitalizations for Pneumonia Associated With the Introduction of a Pneumococcal Conjugate Vaccination Schedule Without a Booster Dose in Australia.” The Pediatric Infectious Disease Journal 29 (7): 607–12. https://doi.org/10.1097/INF.0b013e3181d7d09c.

Simonsen, Lone, Robert J Taylor, Yinong Young-Xu, Michael Haber, Larissa May, and Keith P Klugman. 2011. “Impact of Pneumococcal Conjugate Vaccination of Infants on Pneumonia and Influenza Hospitalization and Mortality in All Age Groups in the United States.” mBio 2 (1): e00309–10. https://doi.org/10.1128/mBio.00309-10.

Cilloniz, Catia, Ignacio Martin-Loeches, Carolina Garcia-Vidal, Alicia San Jose, and Antoni Torres. 2016. “Microbial etiology of pneumonia: Epidemiology, diagnosis and resistance patterns.” International Journal of Molecular Sciences 17 (12). https://doi.org/10.3390/ijms17122120.

Feikin, Daniel R., Laura L. Hammitt, David R. Murdoch, Katherine L. O’Brien, and J. Anthony G. Scott. 2017. “The Enduring Challenge of Determining Pneumonia Etiology in Children: Considerations for Future Research Priorities.” Clinical Infectious Diseases 64 (suppl_3): S188–S196. https://doi.org/10.1093/cid/cix143.

Bjarnason, Agnar, Johan Westin, Magnus Lindh, Lars-Magnus Andersson, Karl G. Kristinsson, Arthur Löve, Olafur Baldursson, and Magnus Gottfredsson. 2018. “Incidence, Etiology, and Outcomes of Community-Acquired Pneumonia: A Population-Based Study.” Open Forum Infectious Diseases 5 (2). https://doi.org/10.1093/ofid/ofy010.

McLaughlin, John M, Eric A Utt, Nina M Hill, Verna L Welch, Edward Power, and Gregg C Sylvester. 2016. “A current and historical perspective on disparities in US childhood pneumococcal conjugate vaccine adherence and in rates of invasive pneumococcal disease: Considerations for the routinely-recommended, pediatric PCV dosing schedule in the United States.” Human Vaccines & Immunotherapeutics 12 (1): 206–12. https://doi.org/10.1080/21645515.2015.1069452.

Oliveira, Lucia Helena de, Silas Pierson Trumbo, Cuauhtémoc Ruiz Matus, N. Jennifer Sanwogou, and Cristiana M. Toscano. 2016. “Pneumococcal conjugate vaccine introduction in Latin America and the Caribbean: progress and lessons learned.” Expert Review of Vaccines 15 (10): 1295–1304. https://doi.org/10.1586/14760584.2016.1166961.

Davis, Stephanie M., Maria Deloria-Knoll, Hilina T. Kassa, and Katherine L. O’Brien. 2013. “Impact of pneumococcal conjugate vaccines on nasopharyngeal carriage and invasive disease among unvaccinated people: Review of evidence on indirect effects.” Vaccine 32 (1): 133–45. https://doi.org/10.1016/j.vaccine.2013.05.005.

Shiri, Tinevimbo, Samik Datta, Jason Madan, Alexander Tsertsvadze, Pamela Royle, Matt J. Keeling, Noel D. McCarthy, and Stavros Petrou. 2017. “Indirect effects of childhood pneumococcal conjugate vaccination on invasive pneumococcal disease: a systematic review and meta-analysis.” The Lancet Global Health 5 (1): e51–e59. https://doi.org/10.1016/S2214-109X(16)30306-0.

Andrade, Ana Lucia, Ruth Minamisava, Gabriela Policena, Elier B. Cristo, Carla Magda S. Domingues, Maria Cristina de Cunto Brandileone, Samanta Cristine Grassi Almeida, Cristiana Maria Toscano, and Ana Luiza Bierrenbach. 2016. “Evaluating the impact of PCV-10 on invasive pneumococcal disease in Brazil: A time-series analysis.” Human Vaccines & Immunotherapeutics 12 (2): 285–92. https://doi.org/10.1080/21645515.2015.1117713.

Moore, Matthew R., Ruth Link-Gelles, William Schaffner, Ruth Lynfield, Catherine Lexau, Nancy M. Bennett, Susan Petit, et al. 2015. “Effect of use of 13-valent pneumococcal conjugate vaccine in children on invasive pneumococcal disease in children and adults in the USA: Analysis of multisite, population-based surveillance.” The Lancet Infectious Diseases 15 (3): 301–9. https://doi.org/10.1016/S1473-3099(14)71081-3.

Weinberger, Daniel M, Richard Malley, and Marc Lipsitch. 2011. “Serotype replacement in disease after pneumococcal vaccination.” Lancet 378 (9807): 1962–73. https://doi.org/10.1016/S0140-6736(10)62225-8.

Gray, Alastair M, Philip M Clarke, Jane L Wolstenholme, and Sarah Wordsworth. 2011. Applied Methods of Cost-effectiveness Analysis in Health Care. New York: Oxford University Press.

Saokaew, Surasak, Ajaree Rayanakorn, David Bin-Chia Wu, and Nathorn Chaiyakunapruk. 2016. “Cost Effectiveness of Pneumococcal Vaccination in Children in Low- and Middle-Income Countries: A Systematic Review.” PharmacoEconomics 34 (12): 1211–25. https://doi.org/10.1007/s40273-016-0439-3.

Vooren, K van de, S Duranti, A Curto, and L Garattini. 2014. “Cost effectiveness of the new pneumococcal vaccines: a systematic review of European studies.” Pharmacoeconomics 32: 29–45. https://doi.org/10.1007/s40273-013-0113-y.

Wu, David Bin-Chia, Nathorn Chaiyakunapruk, Huey-Yi Chong, and Philippe Beutels. 2015. “Choosing between 7-, 10- and 13-valent pneumococcal conjugate vaccines in childhood: A review of economic evaluations (2006–2014).” Vaccine 33 (14): 1633–58. https://doi.org/10.1016/j.vaccine.2015.01.081.

Blank, Patricia R., and Thomas D. Szucs. 2012. “Cost-effectiveness of 13-valent pneumococcal conjugate vaccine in Switzerland.” Vaccine 30 (28): 4267–75. https://doi.org/10.1016/j.vaccine.2012.04.028.

By, Åsa, Patrik Sobocki, Arne Forsgren, and Sven-Arne Silfverdal. 2012. “Comparing Health Outcomes and Costs of General Vaccination with Pneumococcal Conjugate Vaccines in Sweden: A Markov Model.” Clinical Therapeutics 34 (1): 177–89. https://doi.org/10.1016/j.clinthera.2011.12.007.

Castiglia, Paolo, Lorenzo Pradelli, Stefano Castagna, Veronica Freguglia, Giorgio Palù, and Susanna Esposito. 2017. “Overall effectiveness of pneumococcal conjugate vaccines: An economic analysis of PHiD-CV and PCV-13 in the immunization of infants in Italy.” Human Vaccines & Immunotherapeutics 13 (10): 2307–15. https://doi.org/10.1080/21645515.2017.1343773.

Chuck, Anderson W., Philip Jacobs, Gregory Tyrell, and James D. Kellner. 2010. “Pharmacoeconomic evaluation of 10- and 13-valent pneumococcal conjugate vaccines.” Vaccine 28: 5485–90. https://doi.org/10.1016/j.vaccine.2010.05.058.

Delgleize, Emmanuelle, Oscar Leeuwenkamp, Eleni Theodorou, and Nicolas Van de Velde. 2016. “Cost-effectiveness analysis of routine pneumococcal vaccination in the UK: a comparison of the PHiD-CV vaccine and the PCV-13 vaccine using a Markov model.” BMJ Open 6 (11): e010776. https://doi.org/10.1136/bmjopen-2015-010776.

Díez-Domingo, Javier, Manuel Ridao-López, M Victoria Gutiérrez-Gimeno, Joan Puig-Barberá, Jose a Lluch-Rodrigo, and Eliseo Pastor-Villalba. 2011. “Pharmacoeconomic assessment of implementing a universal PCV-13 vaccination programme in the Valencian public health system (Spain).” Vaccine 29 (52): 9640–8. https://doi.org/10.1016/j.vaccine.2011.10.038.

Earnshaw, Stephanie R, Cheryl L McDade, Giovanni Zanotti, Raymond A Farkouh, and David Strutton. 2012. “Cost-effectiveness of 2 + 1 dosing of 13-valent and 10-valent pneumococcal conjugate vaccines in Canada.” BMC Infectious Diseases 12 (1): 101. https://doi.org/10.1186/1471-2334-12-101.

Gouveia, Miguel, Francesca Fiorentino, Gonçalo Jesus, João Costa, and Margarida Borges. 2017. “Cost-effectiveness of the 13-valent Pneumococcal Conjugate Vaccine in Children in Portugal.” The Pediatric Infectious Disease Journal 36 (8): 782–87. https://doi.org/10.1097/INF.0000000000001587.

Klok, Rogier M, Rose-Marie Lindkvist, Mats Ekelund, Raymond a Farkouh, and David R Strutton. 2013. “Cost-Effectiveness of a 10- Versus 13-Valent Pneumococcal Conjugate Vaccine in Denmark and Sweden.” Clinical Therapeutics 35 (2): 119–34. https://doi.org/10.1016/j.clinthera.2012.12.006.

Knerer, Gerhart, Afisi Ismaila, and David Pearce. 2012. “Health and economic impact of PHiD-CV in Canada and the UK: a Markov modelling exercise.” Journal of Medical Economics 15 (1): 61–76. https://doi.org/10.3111/13696998.2011.622323.

Kuhlmann, Alexander, and J-Matthias Graf von der Schulenburg. 2017. “Modeling the cost-effectiveness of infant vaccination with pneumococcal conjugate vaccines in Germany.” The European Journal of Health Economics 18 (3): 273–92. https://doi.org/10.1007/s10198-016-0770-9.

Newall, Anthony T., Prudence Creighton, David J. Philp, James G. Wood, and C. Raina MacIntyre. 2011. “The potential cost-effectiveness of infant pneumococcal vaccines in Australia.” Vaccine 29: 8077–85. https://doi.org/10.1016/j.vaccine.2011.08.050.

Newall, A. T., J. F. Reyes, P. McIntyre, R. Menzies, P. Beutels, and J. G. Wood. 2016. “Retrospective economic evaluation of childhood 7-valent pneumococcal conjugate vaccination in Australia: Uncertain herd impact on pneumonia critical.” Vaccine 34 (3): 320–27. https://doi.org/10.1016/j.vaccine.2015.11.053.

O’Brien, Megan A, Lisa A Prosser, Jack L Paradise, G Thomas Ray, Martin Kulldorff, Marcia Kurs-Lasky, Virginia L Hinrichsen, Jyotsna Mehta, D Kathleen Colborn, and Tracy A Lieu. 2009. “New vaccines against otitis media: projected benefits and cost-effectiveness.” Pediatrics 123 (6): 1452–63. https://doi.org/10.1542/peds.2008-1482.

Robberstad, Bjarne, Carl R. Frostad, Per E. Akselsen, Kari J. Kværner, and Aud K H Berstad. 2011. “Economic evaluation of second generation pneumococcal conjugate vaccines in Norway.” Vaccine 29: 8564–74. https://doi.org/10.1016/j.vaccine.2011.09.025.

Rozenbaum, Mark H, Elisabeth a M Sanders, Albert Jan van Hoek, Angelique G S C Jansen, Arie van der Ende, Germie van den Dobbelsteen, Gerwin D Rodenburg, Eelko Hak, and Maarten J Postma. 2010. “Cost effectiveness of pneumococcal vaccination among Dutch infants: economic analysis of the seven valent pneumococcal conjugated vaccine and forecast for the 10 valent and 13 valent vaccines.” BMJ 340 (jun02 1): c2509–c2509. https://doi.org/10.1136/bmj.c2509.

Rubin, Jaime L., Lisa J. McGarry, David R. Strutton, Keith P. Klugman, Stephen I. Pelton, Kristen E. Gilmore, and Milton C. Weinstein. 2010. “Public health and economic impact of the 13-valent pneumococcal conjugate vaccine (PCV13) in the United States.” Vaccine 28 (48): 7634–43. https://doi.org/10.1016/j.vaccine.2010.09.049.

Strutton, David R, Raymond a Farkouh, Stephanie R Earnshaw, Sharon Hwang, Ulrike Theidel, Stathis Kontodimas, Rogier Klok, and Sotiria Papanicolaou. 2012. “Cost-effectiveness of 13-valent pneumococcal conjugate vaccine: Germany, Greece, and The Netherlands.” Journal of Infection 64 (1): 54–67. https://doi.org/10.1016/j.jinf.2011.10.015.

Talbird, Sandra E, Thomas N Taylor, Stefanie Knoll, Carl Richard Frostad, and Sebastián García Martí. 2010. “Outcomes and costs associated with PHiD-CV, a new protein D conjugate pneumococcal vaccine, in four countries.” Vaccine 28 Suppl 6 (November): G23–9. https://doi.org/10.1016/j.vaccine.2010.06.016.

Hoek, Albert Jan van, Yoon Hong Choi, Caroline Trotter, Elizabeth Miller, and Mark Jit. 2012. “The cost-effectiveness of a 13-valent pneumococcal conjugate vaccination for infants in England.” Vaccine 30 (50): 7205–13. https://doi.org/10.1016/j.vaccine.2012.10.017.

Zhou, F., Abigail Shefer, J. Wenger, M. Messonnier, L. Y. Wang, A. Lopez, M. Moore, T. V. Murphy, M. Cortese, and L. Rodewald. 2014. “Economic Evaluation of the Routine Childhood Immunization Program in the United States, 2009.” PEDIATRICS 133 (4): 577–85. https://doi.org/10.1542/peds.2013-0698.

Herdman, Michael, Amanda Cole, Christopher K Hoyle, Victoria Coles, Stuart Carroll, and Nancy Devlin. 2016. “Sources and Characteristics of Utility Weights for Economic Evaluation of Pediatric Vaccines: A Systematic Review.” Value in Health : The Journal of the International Society for Pharmacoeconomics and Outcomes Research 19 (2): 255–66. https://doi.org/10.1016/j.jval.2015.11.003.

Mauskopf, Josephine, Baudouin Standaert, Mark P. Connolly, Anthony J. Culyer, Louis P. Garrison, Raymond Hutubessy, Mark Jit, Richard Pitman, Paul Revill, and Johan L. Severens. 2018. “Economic Analysis of Vaccination Programs: An ISPOR Good Practices for Outcomes Research Task Force Report.” Value in Health 21 (10): 1133–49. https://doi.org/10.1016/j.jval.2018.08.005.

Walker, Damian G, Raymond Hutubessy, and Philippe Beutels. 2010. “WHO Guide for standardisation of economic evaluations of immunization programmes.” Vaccine 28 (11): 2356–9. https://doi.org/10.1016/j.vaccine.2009.06.035.