The thoughtful responses to yesterday’s post really got me thinking. Why don’t we see more of the sorts of front-end + back-end consortia that would clearly serve everyone – such as the ability to add external database operations to the results of a Google search?
Google presents its search engine only through a monolithic access point. Unless you make a different business arrangement with Google, Inc., you can only access those powerful search algorithms through a relatively simplistic interface. I assume that this is because Google wants to remain a vertical monopoly – just one carefully branded route to its underlying search engine, unless you pay to become a business client for other uses of that engine.
But perhaps it might be in Google’s long-term interest to “let a thousand flowers bloom” – to allow others to try their hand at building alternate interfaces on top of the core search engine.
I’m not suggesting here that everyone program in SQL – any more than we would expect every consumer to implement an IPhone app. The key is to allow a vibrant community of people to do so, in a way that is of use to many more people. If Google were to allow more access to the underlying engine, then some people would start to come up with uses for the engine that Google couldn’t possible have thought of – because Google, as big as it is, and as smart as its employees are, cannot replace an entire economy of implementers.
There seems to be an underlying question here: How does a profit-oriented corporation make the best win-win arrangement with a peripheral group of folks who inhabit a more freewheeling microeconomic style of creation? Consumers in general are best served when those two forces combine to create a synthesis whereby everybody wins: (i) The corporation wins because ever new markets are found for its core engine (its greatest strength); (ii) The individual implementer community is able to do things that nobody would have done otherwise; (iii) The larger community of users wins because all these innovative, more-focused, easier-to-use, tailor-made interfaces start appearing in their lives, or at least become available at on-line check-out counters.
Maybe one way for U.S. industry to work its way out of the current economic malaise is to figure out how to harness our collective brainpower to foster new information microeconomies.
That’s why we have Lucene 🙂
http://lucene.apache.org/
There is the issue of scale. Google invests an enormous amount of money for the vast distributed army of server machines that cache the results of its web crawling. That’s one advantage of including, somewhere in the ecosystem, the capital-intense investment model provided by a large corporation.
Just out of curiosity, do you use Lucene/Nutch/Solr/Compass, etc., or do you use Google for web searching?
Interestingly, both the Solr and Compass homepages have Google search boxes positioned prominently near the top.
Yes, Google does indeed have scalability on its side. However, I would argue that wishing to have access to the large proprietary database that is Google Search is akin to wishing to have access to Microsoft’s or Apple’s operating system internals. In both cases, it can be argued that it would benefit the monopolies owning the secrets. It can be said that Google is more open than most other companies and as such, it would be likely to allow access to its Search database with an “API”, but I won’t bet on it.
That is the beauty of free software, especially with regard to “cloud computing.” A one-time setup of a free-for-all Lucene, while perhaps not comparable to Google’s results, will indeed allow you the freedom to tinker.
I use Google, but only because a GNU version of it is not yet here. It is a shame that Solr and Compass do not trust their own selves, but it does not refute the argument for the freedom of software. Please also see the Affero GPL license.
I find it amazing that in a period of less than ten years, we have gone from a situation where our public libraries acted as the maintainer of the index of knowledge to one where a private corporation has arguably taken on that role.
Personally, I think the library of congress should start a project to create a public open-source search index and archiving operation, with a commitment to scalability, openness, transparency, and public debate, and the kind of open APIs you propose.
Critics might argue that this is a waste of tax money, but I think the benefits of the kinds of innovative uses such a public index would create go far beyond what Google offers us today.
The concept of a public library is steeped in the enlightenment and democracy (wild guess, but do liberty and library share as a common root the latin word Liber, as in free?) We all lose if Google becomes the new library of congress.
Am I mistaken, or is the kind of interface you are looking for already available?
http://code.google.com/apis/ajaxsearch/
You will find, if you really look at Google ajaxsearch – at its highly limited capabilities and at the severe usage restrictions in its license – that it is meant to be a toy, and nothing more, something that exists just to let you play around with the Google API.