The value of click-stream data derived from toolbars

Reading Greg Lindens blog is always pleasent if you want to dive deeper in the knowledge of search engines. In his last post he refers to an article by Jan O. Pedersen et al at Yahoo! called “Making sense of search result pages” [PDF] (published in a academic digital archive, not a peer-reviewed journal).

“Search engine query logs only reflect a small slice of user behavior – actions taken on the search results page. A more complete picture would include the entire click stream; search result page clicks as well as offsite follow-on actions”.

Yes, I’ve always been critical to the importance of query log data and click-stream data is of course a good expansion to quantitative studies on search beahvior. But, I think the heavy dependency on this kind of quantitative data is just an expression of the laziness of search engines to not use more qualitative research methods. For example naturlistic inquiry. Yes, it’s not possible to get data from as many users as in query log analysis, but I think the mix is important. Actually, I’m not quite sure how much qualitative reasearch search engines do today but among usability engineers it’s a matter of course.

But Pedersen also comment eye tracking:

“Ultimately a direct measurement of what the user perceives on a search results page would be most useful. A closely proxy of this ideal is measuring eye movement and fixations across the page followed by user interviews. Aggregate measures reveal which page elements get the greatest attention, while tracking individual sessions is very revealing about scanning behavior”.

User interviews! Good qualitative research, but not always easy. But remember what Jared Spool said about eye tracking in: “Eyetracking: worth the expense?“.

More of mixing quantitative data with qualitative in web search behavior research. This scientific field is overwhelmed by query log analysis papers.