Integrating Yahoo! BOSS with your ruby on rails application

Amit Solanki — Thu, 29 Oct 2009 08:08:05 +0000

What is Yahoo! BOSS? Yahoo developer website cites it as:

“Yahoo! Search BOSS (Build your Own Search Service) is an initiative in Yahoo! Search to open up Yahoo!’s search infrastructure and enable third parties to build revolutionary search products leveraging their own data, content, technology, social graph, or other assets. This release includes Web, News, and Image Search as well as Spelling Suggestions.”

Some of the possible implementation of Yahoo! Boss may be finding related posts for your article, suggested tag/category for an article/query, correcting misspelled words by providing suggestions, fetching latest news on a topic dynamically, search over delicious tags, customized language search. Some of these have been tried and successfully implemented. An example is the TechCrunch’s Search Engine.

Here is a concise guide on how to use this new service in your next rails application.

Obtain API key

Get an application key from http://developer.yahoo.com/search/boss/.

Install BOSSMan Gem

BOSSMan is a gem for interaction with the Yahoo BOSS web service written by John Pignata. Install it using the following commands:

gem install gemcutter
gem tumble
gem install bossman

Usage

Put this in your environment.rb

require 'bossman'
include BOSSMan
BOSSMan.application_id =

Web Search

Use the following code to execute search over the web:

boss = Search.web("Apple Pie", { :count => 2, :filter => "-hate" })
puts "Number of results: #{boss.totalhits}"

boss.results.each do |result|
  puts "#{result.title}"
  puts "#{result.dispurl}"
  puts "#{result.abstract}"
end

# => Number of results: 2618888
# => Apple pie - Wikipedia, the free encyclopedia
# => en.wikipedia.org/wiki/Apple_pie
# => English apple pie recipes go back to the time of Chaucer. ... The basis of Dutch apple pie is a crust on the bottom and around the edges. ...

boss = Search.web("Moon", { :count => 10, :type => "nonhtml,-pdf" })

boss.results.each do |result|
  puts "#{result.title}"
  puts "#{result.dispurl}"
  puts "#{result.abstract}"
end

# => BACKGROUND:
# => www.radford.edu/~rusmart/moon.doc
# => We noticed was that one of the scan lines took longer to scan the moon then any other. This occurred toward the middle of the moon. ...

boss = Search.web("Star", { :sites => "wwe.com", :style=>'raw' })
boss.results.each do |result|
  puts "#{result.title}"
  puts "#{result.dispurl}"
  puts "#{result.abstract}"
end

# => John Cena - WWE: Raw
# => www.wwe.com/superstars/raw/johncena
# => Official profile from WWE for the professional wrestler John Cena, who has been training since he was 15. Features lots of photos and video clips, including John ...

Apart from universal arguments mentioned at the end of this document, the following arguments can also be used:

Parameters	Values/Description
`:filter`	Filter out adult or hate content Syntax: `:filter => "-hate, -porn"`
`:type`	Specifies document formats (pdf, msoffice,etc)
:view	Syntax: `:view => "view1,view2"`, etc `:view => "keyterms"` will retrieve related words and phrases for each search result. `:view => "searchmonkey_feed"` will retrieve structured data markup, if available, for the search result in dataRSS format. `:view => "searchmonkey_rdf"` will retrieve structured data markup, if available, for the search result in rdf format. `:view => "delicious_toptags"` will retrieve the top public delicious tags for a document and the counts associated with each tag `:view => "delicious_saves"` will retrieve the number of times a document was saved in delicious `:view => "language"` identifies the language of the document
:abstract	`:abstract => "long"` will retrieve and display an abstract of a web document up to 300 characters. This expanded abstract provides the requestor with a larger piece of information to work from in a web search query. The default for abstract is an abbreviated description.

Click here to view list of response fields returned by web search.

Fields	Details
`count`	Indicates how many results to show per page. The value shown in results is the smaller of the default or explicit “count” request argument value or the actual number of results in the response. The result set will include the requested “count” unless there are fewer than that number of matching documents.
`totalhits`	A result count that reflects no duplicates. The totalhits value is an approximation, and its value may change depending on the requested “start” and “count” values, because the approximation is adjusted as more exact result URLs are processed. A normal use for totalhits is to determine how many pages of results to offer in search result navigation. Since totalhits is an approximation, and the value may change as “start” increases on successive result pages, the result page navigation may need to be adjusted as a user browses result pages.
`deephits`	It returns an approximate count that reflects duplicate documents and all documents from a host. deephits, therefore, is invariably equal to or larger than TotalHits. The deephits value is normally used as an information display on a search result page, reporting how many total documents matched the search terms.
`abstract`	Abstract with keywords highlighted with HTML tags. abstract=long will retrieve and display an abstract of a web document up to 300 characters. This expanded abstract provides the requester with a larger piece of information to work from in a web search query.
`title`	Title with keywords highlighted with HTML tags
`url`	URL of result
`clickurl`	Returns a navigation URL that leads to the target URL for each result. A clickurl might lead through a redirect server, which provides Yahoo! with important usage data from search result sets.
`dispurl`	Returns the URLs of documents matching the query result. Use this field only for display purposes on result pages. To direct search users to the target document, use the clickurl value
`size`	Returns the document’s size in bytes
`date`	Returns date in YYYY/MM/DD format
`keyterms`	When `:view => "keyterms"` is used in query string the keyterms section is included in the response. This includes a list of related words and phrases with the `term` subheadings.
`searchmonkey_feed`	Using this field will retrieve structured data markup, if available, for the search result in dataRSS format.
`searchmonkey_rdf`	Using this field will retrieve structured data markup, if available, for the search result in RDF format.
`delicious_toptags`	Retrieves the top public delicious tags for a document and the counts associated with each tag.
`delicious_saves`	The number of times a document was saved in delicious.
`language`	The language of the document.

Images Search

For searching images on web, use the following code:

boss = Search.images("snow hills", { :dimensions => "large" })

boss.results.each do |result|
  puts "#{result.url}: #{result.abstract}"
end

# => http://www.amanita-photolibrary.co.uk/photo_library/BI_habitats/gb96_chiltern_hills_from_ivinghoe_snow_std.jpg: Copyright © 2004 Chiltern hills east from near Ivinghoe snow habitats landscapes British Isles

Click here for list of additional arguments that can be provided with image search.

Parameters	Values	Description
`:filter`	`yes` (default) or `no`	Whether or not to activate the Offensive Content Reduction filter. If set to yes, content marked as offensive is omitted from results.
`:dimensions`	`all` (default), `small`, `medium`, `large`, `wallpaper`, `widewallpaper`	Small images are generally thumbnail or icon sized. Medium sized images are average sized; usually not exceeding an average screen size. Large images are screen size or larger. Size pairs other than those listed are ignored.
`:refererurl`	URL, no default	Search for this URL. Depending on other query restrictions, returns all image objects with this referring URL. Example: http://www.foo.com/~foo/images.htm
`:url`	URL, no default	Search for this URL. Returns this exact image result. Example: http://www.foo.com/~foo/images/mydog.gif

Click here to view list of response fields returned by image search.

Field Name	Description
`totalhits, deephits`	Number of listings that match the query parameters.
`abstract`	Description of image
`clickurl`	URL to click to view image
`filename`	Filename of image
`size`	Size of image file
`format`	Format of image
`height`	Height of full-size image
`date`	Last modification date of image YYYY/MM/DD
`mimetype`	MIME type of image
`refererclickurl`	Clickable link to page in which image was found
`refererurl`	Original link to page in which image was found
`title`	Title of image (usually the filename)
`url`	URL of image itself
`width`	Width of full-size image
`thumbnail_url`	Thumbnail image
`thumbnail_height`	Thumbnail height
`thumbnail_width`	Thumbnail width

News Search

Yahoo! BOSS also provides news search capabilties.

boss = BOSSMan::Search.news("free shipping", { :age => "1d-2w", :orderby => "date", :count => 1})

boss.results.each do |result|
  puts "#{result.title} [from #{result.source}]"
  puts "#{result.abstract}"
end
# => Wal-Mart announces weekly price cuts for holidays [from USA Today]
# => Wal-Mart said it will cut prices this holiday season for a week at a time on thousands of items, from bananas to board games. The first group of cuts hit Wednesday.

Click here for list of additional arguments that can be provided with news search.

Parameters	Values	Description
`:age`	`7d`	Use age= to retrieve documents by age: `:age => "[value]d"` NOTE: Maximum [value] cannot be greater than 1000. The default is 30 days, and valid values are `[1-30]d`. (News Search maintains a 30-day index.) The following example searches the full index: `:age => "7d"` Age can be specified using a string made an integer and one of the letters “`s`“, “`m`“, “`h`“, “`d`” and “`w`” representing seconds, minutes, hours, days and weeks. No spaces are allowed in the string. Some examples: `:age => "30d"` `:age => "1w"` `:age => "12h"` To specify an age range two values can be concatenated with a dash/hyphen for example: `:age => "5d-10d"` Between 5 and 10 days old `:age => "2w-30d"` Between 2 weeks and 30 days old Multiple ranges or more than one age parameter is not supported.
`:orderby`	relevance	To order by date add `:orderby => "date"` to the query string. Results will be returned with the most recent first.

Click here to view list of response fields returned by news search.

Field Name	Description
`totalhits, deephits`	Number of listings that match the query parameters.
`abstract`	Abstract of news story.
`clickurl`	Click this link to reach story, if present.
`language`	Language of story.
`date`	Last publication date of story.
`time`	Last publication time of story.
`source`	Source publication.
`sourceurl`	URL of source publication.
`title`	Title (or Headline) of the story.
`url`	Link to story.

Spelling Search

Correct your misspelled words using its spelling suggestion feature:

boss = BOSSMan::Search.spelling("acknowlegment")
puts boss.suggestion # => acknowledgment

Site Explorer

You can also search for the pages from other sites linking into your site pages.

links = Search.se_inlink("http://mail.yahoo.com", :count => 2)
links.results.map { |result| p result.url }
# => http://www.aol.com/
# => http://groups.yahoo.com/

For displays a list of all pages belonging to a domain in the Yahoo! index, use se_pagedata method provided.

links = Search.se_pagedata("twitter.com", :count => 1)
links.results.map { |result| p "#{result.url} - #{result.abstract}" }
# => http://twitter.com/ - Use Twitter to send status updates (tweets) through your cell phone, instant messenger, or via the Web, and notify friends and followers of the little things ...

Universal Arguments for Web, Images and News

Following is the list of most commonly used arguments with web, image and news search. For more comprehensive list click here

Argument	Options/Details
`:start`	Ordinal position of first result. First position is 0. Default sets start to 0.
`:count`	Total number of results to return. Maximum value is 50. Default sets count to 10.
`:format`	The data format of the response. Value can be set to either `"xml"` or `"json"`. Default sets format to `"json"`.
`:callback`	The name of the callback function to wrap the result. Parameter is valid only if format is set to `"json"`. No default value exists.
`:sites`	Restrict BOSS search results to a set of pre-defined sites. Multiple sites must be comma separated. Example: (`:sites => "abc.com,cnn.com"`). The Images service does not yet support multiple sites.
`:view`	Retrieve additional search data provided by the respective BOSS service. Please see individual chapters to see what view options are available.
`:style`	By default for web search result titles and abstracts contain bold HTML tags around the search term. Use `:style => "raw"` to remove the bold tags around the search terms in titles and abstracts.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Vinsol - Leading Ruby on Rails Development and Consulting Firm in India » yahoo boss

Integrating Yahoo! BOSS with your ruby on rails application