This section presents the overall conclusions drawn from the brief results analysis carried out.
Firstly, the analyses’ results show that thelocationsearch factor has the largest impact on the mutability of search results (as expressed by the values of both metrics calculated), followed by thetime,authentication status, andcookiesfactors. Thesafe searchandprivacyfactors seem to be the ones that have the lowest impact on search results volatility. To aid visualization, we present in Table5.1a concise summary of the results of both metrics for the experiments performed, for the top 10 results, for both Google and Bing. As for the results’ comparison between the Google and
5.9 Results Discussion 77
Figure 5.52: Graph Showing the Rank Movements Per Rank Position Metric Results for Each Query Topic, for the Top 10, for Google (authentication statusExp.)
Bing search engines, we observe that for some search factors the values of the metrics are larger for Bing, while for other factors the values of the metrics are larger for Google. This data indicates that the search engines influence the amount of search results’ mutability differently depending on the varying search factor.
Table 5.1: Results of Both Metrics for the Experiments Performed, for the Top 10 Results, for Both Search Engines
% New URL Rank Movements Per Rank Position
Google Bing Google Bing
-When it comes to the differences between the metric’s results obtained for the top 10, top 5, top 3 and top 1 search results, several conclusions can be drawn. First, when it comes to the% New URL, in some experiments (time,locationandsafe search) the results of the metric generally decrease as the top rankings analyzed become higher, while in some other experiments (privacy, user agent,cookies, andauthentication status) the results of the metric generally increase as the top rankings analyzed become higher. This indicates that the percentage of new URLs increasing
or decreasing for higher top rankings is dependant on the varying search factor. Secondly, for all experiments analyzed, the amount of rank movements per rank position decreases as the top rankings analyzed become higher. This shows that there are less rank shifts in the search results with higher ranking positions, regardless of the varying search factor, which means higher top results often maintain their ranking positions. Also, since in the time, location, and safe search experiments both metrics’ values decrease for higher top rankings, we can conclude that, when varying these search factors, search results in higher ranking positions are less volatile.
Furthermore, the search results’ mutability seems to be influenced by the query topics, as many of the experiments’ results show some variation between the values of the metrics for different query topics. The differences in the value of the % New URL metric for different query topics went as high as 15.4% for Google and 17.5% for Bing. As for the Rank Movements Per Rank Positionmetric, the largest difference was 0.33 for Google and 0.51 for Bing. We also notice that these differences have larger peaks for Bing than for Google, which may indicate that query topics have a larger influence on the mutability of search results when using Bing. Additionally, even though query topics seem to influence search result’s mutability, the analyses’ results don’t supply enough information to conclude if there are specific topics that always have more or less impact on results volatility.
One additional interesting fact is that, often, when a search engines’ results for one metric are larger than those of the other search engine, the inverse scenario can be seen when it comes to the other metric. This happens for the results of the time, location, privacy, anduser agent experiments. One explanation for this could be the fact that having a lower value for the% New URLmetric indicates there are less insertions/deletions of URLs in thetop kresults, which means there is a larger amount of common results in thetop kthat have a chance to shift ranking positions (contributing for the increase of theRank Movements Per Rank Positionmetric). By the same logic, having a larger value for the% New URLmetric leads to a smaller amount of common results in thetop kand thus to less possible ranking shifts.
Chapter 6
Conclusions and Future Work
This chapter presents the final conclusions obtained from this study, as well as some features that could be improved in future works. Section6.1 summarizes the results of this project and Section6.2explores possible enhancements to this work, as well as some next steps that could be taken by future projects.
6.1 Conclusions
Information on the web is unstable, due to the web’s dynamic characteristics. Web search engines reflect this instability, as the search results returned can change based on several factors, such as time or the search engine used. The personalization of search results (based on factors such as location or cookies), has also evolved significantly recently.
This work allowed us to obtain insights about how much several search factors — time, lo-cation, safe search, privacy, user agent, cookies, and authentication status — can impact the mutability of the search results obtained in web searches, in each of the search engines examined, for each of the top results analyzed (top 10, 5, 3, and 1), as well as the influence of query topics on results volatility.
The main contribution of this work was the production of structured datasets detailing the anal-ysis of the volatility of the obtained search results, for each search factor studied, and making these datasets available publicly so they may aid future investigations. Furthermore, a brief analysis of the data obtained was performed, serving as a proof of concept that the data created by this study could be used for further research, and revealing some conclusions. Firstly, the analysis performed reveals thelocationsearch factor to be the one with the largest impact on the mutability of search results, followed by thetimefactor. The factorssafe searchandprivacypresent the lowest impact.
Furthermore, the use of different search engines also influences the volatility of the search results.
Additionally, when varying some factors (time,locationandsafe search) the higher top results are less volatile (experiencing a lower percentage of insertions/deletions of URLs), while the inverse
79
scenario applies to theprivacy,user agent,cookies, andauthentication statusfactors. Still, when varying any of the factors studied, higher top results experience less rank shifts, meaning higher top results often maintain their ranking positions. Finally, the mutability of search results is also influenced by the query topics, even though the amount of impact each query topic has may not be consistent when varying different search factors.
The obtained insights about web search and search engines are relevant, not only to the general public, but also for developments in the area of information retrieval research. The results of this study may help shed more light on the subjects of search results volatility and personalization, by presenting an analysis of the variance of search results according to several search factors and making the obtained results available to the public.