When Everything Works But Still Fails: Federated Search Engines and Link Resolvers

As an Electronic Resources Librarian, I get to see all sorts of technical issues due to our various resources. Federated search engines and OpenURL link resolvers offer up a lot of interesting cases. But sometimes, it’s not their fault. Everything is actually working as intended, and yet the system still fails to deliver a full text article to our users.

An interesting case in point recently crossed my computer. The user had searched our federated search engine and found an article called “Music: Hip-Hop Nation” in Time magazine. TheĀ federated search engine identified that we should have full text access and provided a link. But when the user followed the link, it led to an error page.

So what happened?

Thinking that perhaps there was just an error in the link, I manually checked the source where the full text was supposed to be found. And there was indeed an entry for the article. Except, the title was “Hip-Hop Nation. (cover story).” The link resolver was trying to pass through the wrong title! But why?

That gets to the nature of how a federated search works. In order to find an article, it needs metadata about the article that tells it that it exists and where to find it. However, the companies that provide the full text don’t always share their metadata with the companies that make the federated search engines. They may be competitors, they may want money, they may want metadata in return; whatever the reason, they may not want to share their metadata.

Federated search engines get around this by, where possible, getting the metadata about an article from some third party that is more amenable to working with them. But, as in this case, just because two different places have created metadata for an article doesn’t mean they both did it in an identical way. Which leads to a metadata mismatch; the federated search engine knows about the article, and the full text source knows about the article, but they don’t agree on the details, and we end up with a frustrated user.

But in this case, that’s not the end of the story; there was yet another reason why the user couldn’t get the full text of the article; it didn’t actually exist in the full text source!

So why did the link resolver think that the full text of the article would be there?

In this case, the full text source was what’s known as an aggregator database; a resource, different from the original publisher, that aggregates content from thousands of different sources. The link resolver knows that full text exists by checking the year, volume, and issue of an article against an internal database of coverage dates for the magazine or journal where the article was published.

The problem arises, particularly with newspapers and magazines, when the original publisher may not have secured the rights to sell a particular article to the aggregator. This can mean that almost all of the content in a particular magazine issue may be available as full text, but one or two articles may be missing.

And that is likely what happened here.

Despite the federated search engine correctly identifying that the article existed, and the link resolverĀ  identifying the correct full text source, in the end, the user left empty-handed.

Of course, that’s not the end of the story, because libraries have other tools, and in this case the user was able to request the article through our document delivery/inter-library loan service, providing a (slightly delayed) happy ending.