JCPSLP Vol 18 no 2 2016

23 / 60

JCPSLP Volume 18, Number 2 2016 69
• focus of site content (population, age);
• credibility indicators (including the presence of citations,
date of posting / update, and domain type).
These metadata were only collected for the page directly
linked to the Google search result. The full coding schema
has been included in the Appendix. Findings underwent
descriptive statistical analysis in Excel. Inter-rater coding
was performed by the second author on 20% of the
sample. Cohen’s Kappa statistic was used to calculate
inter-rater reliability (Hallgren, 2012), with an average Kappa
score of 0.42 (moderate strength). Kappa values for each
criteron are presented in Table 2.
Search strategy
One factor that features heavily in a search engine’s
algorithms is the individual searcher’s prior search behaviours
and browsing history. As a consequence, a parent of a child
with newly diagnosed autism and an experienced AAC
clinician may in fact receive different results from the same
keyword search. To ensure results were not influenced by
the researcher’s own search history, searches were
conducted using an anonymous browser setting, and new
sessions were launched for each search. Given Spink and
Jansen’s 2004 findings that most searchers do not look
beyond the second page of results, we restricted the
number of harvested results to 20 (2–3 standard Google
results pages), which were then transferred as URLs to an
Excel spreadsheet for later analysis. Google-generated
definitions, sponsored websites, advertisements, and other
suggestions (e.g., “images” and “scholarly articles”) were
not included on this list.
Search result analysis
The first 20 results harvested for each term were rated in
Excel, according to the following criteria:
• purpose of the website;
• relevance to AAC and Australian location;
Table 1. AAC-related search terms containing the
words communication or communication device,
in order of search frequency
Search term Estimated
monthly
Estimated
daily
(Google)
Containing the term “communication”
augmentative communication 1560 43
facilitated communication 1560 43
picture exchange communication
system
1560 43
communication board 1200 33
communication devices 1200 33
Containing the term “communication device”
augmentative communication device 168 5
tech talk communication device 60 2
go talk communication device 60 2
dynavox communication device 60 2
springboard communication device 48 1
Diagnosis-specific terms
aphasia communication 24 1
autism communication 252 7
cerebral palsy communication 24 1
Additional terms
AAC 39 720 1 103
communication app No data
available
No data
available
Table 2. Inter-rater reliability for each criteria,
calculated using Cohen’s Kappa
Criteria Kappa value Strength
Relevance .42 Moderate
Purpose .23 Fair
Citations .73 Substantial
Recency .47 Moderate
Domain .96 Near perfect
Location .45 Moderate
Population .41 Moderate
Age .48 Moderate
Average .50 Moderate
Note: Cohen’s Kappa compares the observed agreement against
an agreement level that could be expected by chance. A score of 1
would indicate perfect agreement between raters, while a score of
–1 would indicate perfect disagreement. Any scores between 0 and
1 indicate agreement at better-than-chance levels (Hallgren, 2012).
Results
Search results were analysed as a group, and then in
separate groupings according to their keyword, purpose,
and domain types. The analyses are presented below.
Viability, relevance and format
The total search set was analysed according to link viability
and relevance of the page to AAC (see table 3). Nine of the
resulting links (3%) led to expired or password-protected
pages, and were excluded from analysis. Of the remaining
291 sites, 204 (70%) were judged “mostly relevant”, 32
(11%) were judged “somewhat relevant” (for example, listing
AAC strategies among other communication strategies or
interventions), and the remaining 55 were found to have no
relevance to AAC (19%). For four search terms (augmentative
communication, augmentative communication device, and
communication board ) all viable top-20 results were judged
“mostly relevant”; product-/technique-specific search terms
also yielded high numbers of mostly relevant results. By
comparison, the search terms communication app, AAC,
communication device resulted in high numbers of
non-relevant pages (12, 11 and 9, respectively), typically
concerning mainstream computing. Diagnosis-specific
searches tended to result in somewhat-relevant sites that
mentioned AAC among a range of other diagnosis-specific
intervention techniques and strategies, for example, the use
of gestures in combination with other language strategies
for adults with aphasia.