Improving Search with the Controversy Discovery Engine

<h2>Better Search Is Deep and Dark.</h2>

Could a web search be more productive by confronting a nasty fact or controversy? People don’t agree about very much. Wouldn’t it be odd if we could use disagreement as a factor in searching when we need details about a topic?

This post offers a simple HTML hack that helps analytic thinkers search deeper and darker. Bring on the Controversy Discovery Engine!!!

The WWW Is Organizational versus Analytical.

First, let’s re frame the World Wide Web. The search page universe is divided into two layers. Organizational pagesinterlink by importance, community affiliation, and loose linking. These pages have advertising value, i.e. potential for a click and a look and a purchase. Organizational pages mainly describe purpose, boards, mottos, good works, and willingness to play together. Surely these pages must get high ranks, leading to the power curve and long tail laws of the web graph.

But, people looking for detailed results must have the right terms to reach into the Analytical Web layer of academic publications, thoughts, position papers, government reports, etc. Often, an abstract is a search hit, leave the person to deal with pay walls, passwords, and other time consuming challenges.

Is it possible to force a search engine through the Organizational into the Analytic web where experts think, write, and talk? Can searches structure valuable facets of information, detail, explanation, science, and mathematics? Can a trail of searches reduce mis-information and trolling?

Maybe the vernacular of “controversy” is key to surfacing pages the Analyst really wants and needs.

My experiment, “Do search engines suppress controversy?”, raised these issues in 2004. We can answer politically, philosophically, socially, algorithmically, or linguistically. Let’s hypothesize that: Adding terms related to controversy force search engines to dig past the Organizational web into the Analytic web to find deeper, and sometimes darker, writing that will ask and answer questions of value to an Analyst.”

We Need a Controversy Discovery Engine.!

How do we manipulate a search engine to make its search’s more systematic and efficient and built around controversy? Well, duh, how about adding words from the vernacular of controversy to basic queries, i.e. extend our normal queries with terms people use to explain and support a position on an issue.

It turns out easy to add a preprocessor for queries thanks to a 2003 book, “Google Hacks”, by Tara Calishain and Rael Dornfest. We use a “hack” in the golden sense of a slick way to solve a problem, not violating privacy or property. We’re speaking good, old-fashioned hacking here.

Come Play With A Controversy Generator.

You can download the controversy Engine” from my Dropbox, save the file, launch it in your browser, and experiment away. See instructions below.

Here’s a tableau of “controversy vernacular” terms used:

terms related to “controversy”

“(controversy OR controversies OR controversial) | controversy ||
“(dispute OR disputes OR altercation) | dispute ||
“(argue OR argues OR argument) | argument ||
“(conflict OR conflicts OR conflicting) | conflict ||
“(disagreement OR disagree OR disagrees) | disagreement ||
“(debate OR debates OR debating) | debate ||
“(oppose OR opposes OR opposition OR opposing) | opposition ||
“(dissension OR dissent OR dissenting) | dissension ||
“(contention OR contend OR contentious) | contention ||
“(quarrel OR squabble OR fracas OR spat) | spat ||
” | NONE ||

terms related to “forms of support”

“(evidence OR data OR model ) | evidence ||
“(data OR table OR graph) | data ||
“(opinion OR assertion OR claim ) | opinion ||
“(proof OR logic OR reasoning) | reasoning ||
(test OR valid OR reliability OR bias) | testing ||
“(hypothesis OR hypotheses OR hypothetical) | hypothetical ||
“(research OR researcher OR investigation OR experiment) | research ||
“(bunk OR diatribe OR lie OR Trump) | bunk ||
“(authority OR authoritative OR dictatorial) | authority ||
” | NONE ||

The Controversy Discovery Engine contains the list of terms related to”controversy” and another list of “forms of support”. A search form contains a text box to enter the base term for search then 2 list boxes from which to select additional terms abbreviated by the word before the “||”.

Example query sent to Google:

“life on Mars” (controversy OR controversies OR controversial) (hypothesis OR hypotheses OR hypothetical)

So, what’s going on here? The additional terms force the query to reach into the Analytic layer of the WWW since few Organizational layer pages use the terms in the extended query.

Hone Your Skills and Build Your Trails

How do you really learn anything about a controversial subject? Just like any search, you scan the resulting pages and experiment with both ‘controversy’ and ‘forms of support’ terms, individually or together.

When you have performed several searches you may want to home in on some results in logical combinations. Maybe you want only the results in both Search A and Search B (AND) or in wither (OR) or A but not B (A and not B). Sorry, this capability will take some programming to strip out the results. I built this interface 15 years ago in Visual Basic then again in Java, but need to take another look at languages that support accessibility and handle parsing. Anybody want to team?

Another approach is to use ‘visited’ an ‘unvisited’ links available in browsers with hot keys or built into screen readers. That’s a good way to find fresh content by repeating an earlier search.

Example Controversies

Example Controversy: Exponential Growth”

Following our earlier discussion of Memex ‘, the term “exponential growth” is ripe for controversy, not to mention confusion. A brave analyst could look at many combinations of terms to fight deep and dark papers related to spread of the Corona Virus. A bad choice would be “spat + bunk”, which is mainly political, while ‘argument + reasoning” becomes technical, and “dispute + testing” looks for reasons to accept claims in the news. Search results have a variety of sources, from high data visualization projects to publication counting in Google Scholar.

Example Controversy: “Guide Dog”

The use of dogs for guiding people with severe vision loss is socially controversial. One issue is animal rights, looking in on the practice of using guide dogs. A broader issue is public confusion about “service dog” versus “guide dogs”, where services relate to many disabilities. And, practical issues trouble people traveling with animals, as on airplanes. The term “controversy” raises many of these issues in combination. “controversy + research” shows few results that qualify as research, leaving the issue in the social realm, versus science. “Controversial + opinion” raises the social issues with proper qualifications. I’ll use this topic for a later post when I discuss my own experience as a guide dog handler of less than a year after a decade getting around with a long white cane.

---

Download the Controversy Discovery Engine.

Save Controversy_Discovery_Engine.html from DropBox. Please credit Susan L. Gerhart and this article at twURL.com as you use the ideas and search form under free and open terms.