Usage Statistics: What and Why

This article was originally published in the American Association of Law Librarians' "The CRIV Sheet", 31.2, February 2009.  Reporting methods for the vendors discussed below have doubtless changed since publication, but the need for reliable information on product use has not.

Usage Statistics: What and Why

Librarians are not generally considered the most mathematically inclined group, since our ranks feature a high percentage of liberal arts types.  However, we do have a very real interest in statistics, or at least the statistics describing our vendor-supplied database use.

To be clear, we are discussing the two main legal database vendors, Westlaw and LexisNexis.  While we have other vendors, the bulk of our spending is usually with either West or Lexis simply because of the sheer volume of material these giants bring together.

In all fairness, both vendors try to provide their clients with sufficient usage information.  However, the information they make available is geared to the billing process.  Invoices show how much time, and money, was spent on a particular client—essential for cost-recovery purposes, and something the academic environment where I spent the last four years simply does not require.  These numbers are for the accountants, and while we appreciate them, they do not really help us make decisions.

Making decisions is, after all, the reason we need information.  Which specific databases should we subscribe to?  Which ones actually get used, or get used enough to include in a package rather than accessing at full price when needed?  How can we save money without compromising our service?  As professionals active in our offices, we have an intuitive sense and anecdotal evidence to support our choices, but hard data is really what we need.  Statistical use analysis is an important part of collection development, and is not particularly well supported for law librarians now.  But take heart—it can be done.

While at the University of Michigan, I did statistical analysis and reporting for the Electronic Resource group.  The method varied by vendor, but always involved checking a handful of key indicators, reported either via a secure vendor website or delivered directly to us in spreadsheet form.  Every vendor made this information available, and it would be irresponsible for a subscriber not to monitor how the product was used.  What we wanted to know fell into two categories: how often was a database used, both the number of individual sessions and the total number of queries made, and how good were the results, both by number of results returned for the searches and, perhaps a better indicator, the number of results actually viewed or downloaded.  Based on these numbers, as well as coverage overlap or uniqueness analysis, we could then determine whether our community was deriving sufficient use from a particular database to justify continued subscription.

As law librarians, we naturally want access to the same kind of information our academic brethren have to help control our own costs.  Unfortunately, West and Lexis do not, yet, provide this kind of information routinely.  The Lexis PowerInvoice system only reports on two categories: LexisNexis Legal Services and Matthew Bender Treatise Services.  These categories are simply too broad to be meaningful.  A library might subscribe to a number of separate Lexis libraries, like Nimmer on Copyright, Chisum on Patents, and so on, but the PowerInvoice categories lump these together under Matthew Bender Treatise Services and tell us nothing about how much each is used, so we are in the dark as to how efficiently we are spending.  If no one is using one of those libraries, we could re-allocate the funds to address other information needs—if only we knew!

Westlaw similarly provides billing-oriented invoice information.  However, their system does allow further exploration of individual product use.  This isn’t especially easy, so we will walk through the process in the next section.

Usage Analysis for Westlaw

Now we're going to look more closely at the Westlaw reporting system, QuickView+.  This secure website (www.quickview.com) is a service for estimating usage charges (the results do not reflect discounts that may be part of the user's plan).  Access requires login with a Westlaw password, and must be requested from one's Westlaw representative by the account's administrator.

Once we've logged in, the first thing to note is that QuickView+ is primarily a tool for billing.  Since we are not concerned with that now, we can skip over most of what is here.  Instead, select "View Database Information" from the drop-down menu in the upper right corner.  This view is intended to let us verify what databases were used on a given day so we aren't surprised by our Westlaw invoice.  Our purpose is slightly different--we want to see everything we're using (and, by extension, everything we're paying for but not using).  To begin, let's examine the various report settings.

Account/ Account Group

If we have multiple billing groups, either physically separate offices or separate practice areas within a firm, this allows us to either view the entire organization's use, or the use by a specific subset.  Our concern is the overall use; this is reflected by "account group".  Select the "Account Group" button and ignore the “Select Account Group" drop-down menu.

Select Date Range Type

Our choices are Daily, Monthly, and Month-to-Date.  Westlaw makes their usage stats available back to January of the prior year.  Daily reports will let us see any use on a given day in that time; Monthly will show all use from a chosen month; Month-to-Date shows use for the current month.  For our purposes, select Monthly, then chose a month from the "Select Usage Date" drop-down menu.

Select Sort Options

Here, we can choose whether the results are displayed according to the client for whom the work was done, or according to the attorney who used the resources.  Select user; knowing who uses which material can be valuable to us (like when looking for acceptable alternative resources).  Clients come and go; database use patterns may be indicative of important tendencies.

Select Special Offer Charges

Finally, we choose what charges to display: only material included in our subscription; only that excluded; both, separately; or both, combined.

Since our underlying goal is to see how our subscription compares to actual use, we will need to see both included and excluded materials.  Letting the report separate them saves us a step later.

This report also offers to calculate tax on use for us, which doesn't matter for our project.

To recap: we've set our report to include the entire Account Group on a Monthly basis, sorted by User and separating Included and Excluded charges.  Choose the most recent completed month under Select Usage Date and click Submit.

The next screen presents a drop-down menu listing everyone who used Westlaw in this month: Westlaw is giving us the information we need, but they aren't making it easy to pull together.  Select the first name and click Submit.  Now a report for this user comes up, allowing us to see specifically which databases were used, as well as how much each was used on a given day, whether it is included in or excluded from any special pricing plan, and the total cost for use.

We will need to repeat this process for each user listed, so it's time to start recording information.  Westlaw does provide options for this, allowing a report to be downloaded or emailed, and also allowing us to view results as either a printable HTML page or Excel spreadsheet.

Creating a spreadsheet is a good plan, since a spreadsheet will allow us to manipulate the data and facilitates analysis.  However, we will be running this same report for every user in each month of the past year, so it will generate a large number of reports to manage and ultimately compile into a single file for analysis.  It may be more efficient to start from scratch, especially when we consider how much data in this report is not relevant to our analysis.

While it does require greater up-front effort (like typing), I prefer setting up my own spreadsheet (This may be easier if we select 'view full printable HTML report' and then print the results, instead of trying to copy them from the screen).  My column categories are User (a last name or initials--this is optional, but lets us see who uses what), In/Out of Contract, Number of Transactions (searches) for a particular database, and Number of Documents Returned for the search (Docs/Lines).  Each database this person used during the month gets its own line.  This means that a frequently-used database may have multiple entries in a given month because several people use it.

Having recorded this information for the first user on the list, use the report tool's Back button and repeat the process for each of the others.  We will then have a complete record of databases used during that month.  Now we're ready to do it again, for the month before.  Eventually, we will have a list of what is actually used, which can then be compared to, and used to bargain for, special pricing in our subscription negotiations.

Usage Analysis for LexisNexis

While it is not easy, Westlaw allows us to see which specific titles are being used, how much, and by whom.  For LexisNexis subscribers who want access to this information, I have good news and bad news.  The bad news: Lexis does not give us a way to retrieve usage stats.  The good news: this means it takes a lot  less work to get usage stats for Lexis than it does to get them from Westlaw.

Lexis has an online statistics site, PowerInvoice (www.lexisnexis.com/powerinvoice).  However, this site is only valuable for billing purposes: the full extent of product breakdown is two categories: Lexis Legal Services, and Matthew Bender Services.  Most of us subscribe to a number of separate Lexis 'libraries', such as LexisNexis, All Public Records, our different states' cases, and individual databases for various treatises.

Given that we generally subscribe to, and want information about, a number of separate titles, the two categories available are much too broad to be of any value in determining what our clientele is actually consulting.  What we need to see is a title-by-title account of what is accessed, by whom and how many times.   Instead, all we can see is a total number of transactions and their cost.  These are of course important for billing and cost recovery, but are not much help in determining what a user was doing to run up the charges: is she struggling to use the system properly (if so, we want to arrange training), or looking in the wrong place (we'd like to point her to the correct resources)?

This sort of information, of course, would also allow us to determine how to most effectively use our subscription resources.  By showing us which databases get heavy use, which do not get sufficient use, and which out-of-contract resources get used anyway, we can fine-tune our subscriptions and better provide for our attorneys.  I want to stress that the goal is not to reduce our spending with Lexis, but to increase our usage by targeting resources that are needed while reducing coverage of areas that are not accessed.

But if Lexis does not make this information available, where is the good news?  The good news is that, while Lexis doesn't allow us direct access to the statistics we need, the numbers are available--we need only ask for them.  A polite written request, made to our regular Lexis representative, should get us what we need for effective collection analysis.  The key point to remember in this request is that we need these numbers to plan appropriate database spending--not to cut spending, but to re-allocate it effectively, so we get our monies-worth, instead of spending wastefully.  Isn't that good news?  It should be much easier to write this letter than to compile comparable stats from Westlaw.

In Conclusion

It is important to remember that this is not an adversarial relationship.  We need what our vendors offer, and they obviously need us.  While statistical analysis has the potential to save us money, that should not be the only goal.  Instead, this is about spending our limited resources efficiently and appropriately, based on evidence from our actual product use.  This will let us see the value we get from our subscriptions as well as showing what, outside the subscription, gets used, whether we should subscribe to that as well (or instead), or if it is more cost-effective to access a product at full price because it is rarely used.  It is about truly partnering with the vendor to provide the best possible information access to our clients—the attorneys or students we serve.

Usage statistics are important to us as law librarians: knowing what vendor-supplied resources actually get used in our offices can allow us to better manage our subscription spending so we can tailor our subscriptions to match our needs.  This involves tracking the individual database titles accessed, and is much trickier than collecting the billing information that Westlaw or LexisNexis makes available for cost-recovery purposes.  However, both vendors recognize our need for this information and do make it available, through different methods, if we are willing do make an effort.  In this age of dwindling resources, the effort is worth making.