Using a socio-ecological model, the Alliance for HPV Free Colorado (Alliance) aims to address several individual, community, health system, and environmental barriers to increase vaccination rates among 11-17 year olds. As such, we have coordinated a multidisciplinary and multifactorial effort to promote evidence-based strategies, promising practices, and program specific roles to ensure a logical connection between population goals and program-level goals. The Alliance objectives are: 1. Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness, 2. Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers, 3. Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years, and 4. Utilize sustainable analytic infrastructure to measure HPV vaccination rates. An evaluation plan was developed in the first year of the grant cycle that assesses process and outcome measures from all aspects of the project. The Alliance is on track to meet all of its objectives; however, outcome metrics are not yet available.
With the socio-ecological model as our base, the Alliance uses dissemination and implementation (D&I) science grounded frameworks to plan, implement, and evaluate project outcomes. D&I helps bridge research and practice and is a study of methods to promote the adoption and integration of evidence-based intervention into routine health care to improve the impact on our population health outcomes. RE-AIM is one D&I tool the Alliance uses to ensure our evaluation plan measures appropriate outcomes for various stakeholders and audiences. RE-AIM stands for Reach, Effectiveness, Adoption, Implementation, and Maintenance, and aims to increase external validity and sustainability of programs. The following questions helped guide us through strategic planning, implementation, and evaluation phases of the project:
Reach: Who is intended to benefit and who actually participates or is exposed to the intervention?
Effectiveness: What are the most important benefits you are trying to achieve and what is the likelihood of negative outcomes?
Adoption: Where is the program or policy applied?
Implementation: How consistently is the program or policy delivered, what adaptations to the original plans were made, and how much did it cost?
Maintenance: When did the initiative become fully operational, how long do the results last and how long will the initiative be sustained?
Objective 1: Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness.
The Alliance hosted three Advisory Council meetings in the first fiscal year of the grant. Between 30 and 40 participants attended each of the three meetings online and in-person. In November 2018, the group brainstormed barriers to HPV vaccination that helped inform the strategic plan for the Alliance. During the January and May meetings, guest speakers provided educational sessions, participants engaged in networking activities, and updates were provided from each Committee and Workgroup.
At the end of each Advisory Council meeting, participants complete a survey to assess goals and productivity of the meeting, as well as to provide suggestions for future meeting topics and additional partners to recruit into the Advisory Council. The Alliance tracked three questions for qualitative trends during fiscal year 19: goals of the meeting, level of participation, and overall satisfaction with the meeting. Overall, an average of 98% of participants reported being satisfied or very satisfied with meetings, 90% reported excellent or good participation at meetings, and 97% reported that the goals of the meeting were excellent or good.
In June 2019, the Alliance created and distributed an engagement survey to send to the listserv. The survey was designed to understand the demographics, barriers, and motivators of both those who attended and those who were unable to attend quarterly meetings. Email addresses were removed from the listserv of those that could not be delivered and for those who were no longer involved in the project. In total, 120 individuals received the survey and 46 responded, giving a response rate of 38.3%. The most common professions among Advisory Council members are public health professionals, nurses, and researchers and 45% serve in the Denver Metro region. Survey respondents identified educational sessions and project updates as the preferred meeting content. They believe they can best support the Alliance by continuing to attend quarterly meetings to stay informed on the project.
Objective 2: Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers.
The first fiscal year of the grant was spent planning and developing the health systems approach to increasing vaccination rates. The first health system to agree to work with the Alliance signed a Memorandum of Collaboration (MOC) in September 2019. To date, six health systems have been engaged and are beginning the process of implementing a year-long quality improvement project, far exceeding our original goal to engage two health systems in the first year of implementation. The Alliance does not have process or outcome measures for work with health systems yet. A variety of data collection tools and methods to evaluate the intervention process and outcome metrics will be used. These include:
- A health system survey will be completed quarterly during the intervention period by leadership, HPV vaccine champions, implementers, and the Alliance points of contact. The survey assesses the usefulness of the evidence-based interventions (EBI), time spent implementing EBI, and attitudes among health system providers on value of HPV vaccine.
- Qualitative interviews will be conducted twice during the intervention period: once six months after baseline and again at project completion. The qualitative interviews collect feedback from at least one representative in leadership, HPV vaccine champions, implementers, and the Alliance points of contact to assess the use of the implementation plan, institutionalization of HPV vaccination best practices, and use of communications materials.
- Initiation and up-to-date (UTD) HPV vaccination rates for 9-17 year old boys and girls in the health system will be reported at baseline and quarterly throughout the intervention year.
Between July 2018 and December 2019, two train-the-trainer sessions, five HPV 101 training sessions, and seven Motivational Interviewing training sessions were completed. In the train-the-trainer sessions, participants were trained to lead HPV 101 and Motivational Interviewing trainings to others. In addition to training Alliance workgroup members, CDPHE and American Cancer Society representatives were trained so that we are able to conduct trainings outside of the 15 county Alliance region. Participants complete pre- and post-training surveys to assess knowledge change from the training sessions. Pre- and post-training survey results indicate that knowledge, skills, and attitudes toward HPV and HPV vaccination improved among trainees and that they felt more confident in understanding the importance of the vaccine. Among providers trained in the Metro Alliance period, there was a 20% improvement in confidence of using motivational interviewing skills when discussing HPV vaccination with hesitant parents. The Alliance will continue to analyze survey results from the HPV 101 and Motivational Interviewing training sessions to assess the knowledge, skill, and attitude change among trainees and improve the content of the training sessions.
The Alliance has engaged three dental practices for pilot projects in 2019. The dental pilot process is ongoing and the evaluations have not been completed. The pilot process facilitators will assess the following indicators through a combination of dental provider surveys and qualitative interviews:
- Percent of eligible patients in the clinic population prompted to get HPV vaccination as a result of the practices implemented.
- Value dental providers place on the HPV vaccine compared to other adolescent preventive dental health needs.
- Confidence among dental providers to recommend the HPV vaccine.
- Confidence among dental providers to discuss HPV infection as a risk factor for oropharyngeal cancer.
- Time spent implementing the protocol.
- Usefulness of the protocol.
- Dental professionals reporting using communication materials for HPV education and awareness.
- If and how HPV vaccine best practices are sustained.
Objective 3: Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years.
The Denver Metro Alliance for HPV Prevention (Metro Alliance) evaluated the communications campaign via 454 telephone surveys of parents with children living in their home ages 9-17 between September 28 and October 17, 2017, after that media buy had concluded. The survey asked parents to recall their knowledge, attitudes, and beliefs about the HPV vaccine from the fall prior to the campaign and compare them to the point in time after the campaign was complete. Among participants, 19% reported more positive attitudes toward HPV vaccination than the year prior. Of those who recalled the HPVFreeCO advertisements, 24% reported having more positive attitudes toward HPV vaccination and were 50% more likely to report that they planned to have their child vaccinated over the previous year.
A paid media campaign, conducted by Peri Marketing and Public Relations (Peri), totaling $295,000 was implemented in the 15 county region from January 29-May 31, 2019. The advertisements ran on Comcast Always On, radio, and digital (Pandora, Facebook, Mobile In-App, Mobile browser) outlets. Peri monitored the progress and metrics of the buy including impressions, click-through rates, and value added. The campaign planned for 34,000 Comcast spots and received over 10,000 additional spots, valuing an additional $20,539 in free media exposure. The online campaign delivered 4.4 million additional impressions, equating to $6,705 in additional value for the campaign. In total, the HPVFreeCO campaign served 90,982,355 impressions over the four month time period it was in market.
In the spring and fall of 2020, the Alliance will conduct a pre/post evaluation of the adapted campaign with parents of 11-17 year olds in the 15 county region to assess recall, demand, knowledge, and attitude change towards the HPV vaccine. Earned and owned social media metrics are reported by individual county health departments and rolled up into a single evaluation. Website metrics are gathered monthly from Google Analytics. During the time the HPVFreeCO campaign was in the market, the traffic to the HPVFreeCO landing page increased significantly. Individuals spent 20 seconds on average on the webpage and there were 413,681 total sessions among 198,406 unique users from January 29-June 1, 2019.
Objective 4: Utilize sustainable analytic infrastructure to measure HPV vaccination rates.
Using IBIS, the Alliance is able to see vaccination rate trends over time, which can be utilized to inform decision making. IBIS allows DPH to stratify vaccination data by age, sex, ethnicity, and location. In June 2018, the initiation rate for HPV vaccination was 59.6% for 11-17 year old boys and girls in the 15 county region and the UTD rate was 38.5%. After 15 months of interventions in September 2019, the initiation rate was 62.3% and the UTD rate was 41.5% in the region. Overall, since the project initiation there has been an 8% increase in UTD rates and a 4.5% increase in initiation rates. This equates to approximately 17,000 more adolescents who have initiated the HPV vaccine series and 31,000 more adolescents who are UTD.