All over the web people are writing about Amazon.com’s seemingly positive holiday sales numbers in contrast to the overall retail market. At least one blog took some time to do some analysis between this year’s results and last year’s. After reviewing the analysis, however, it is clear to me that the data is inconclusive. Take the example, of comparing the results of this year’s peak sales day to last year’s:
The busiest day of the year was pushed back to Dec. 15th from Dec. 10th in 2007, and Dec. 11th in 2006. It could be that customers were apprehensive about making purchases this year, and in that case, did not make purchases in the preceding days of the month, waiting until they ascertained their comfort level this holiday season. That would be a negative. However, it could also mean that customers became comfortable with shipping times and continued to shop until that date. That would be a positive. Or the shift in the peak date could just reflect the fact that there were 5 fewer shopping days this holiday season.
Naturally, as I read this my mind recalled countless similar conversations with our clients in higher education. In higher education, we too have “shopping days” before enrollment or the drop-add period. Yet, we often have about as much insight into why there are differences in enrollment patterns from year to year as is evident in the quotation above.
There are two main reasons for this; first, we often use far too few data points for a statistically meaningful comparison. One, two, or even three years of comparison data does not necessarily constitute a trend. Secondly, when there are many more years of data available, the passage of time often distorts the clarity of the information.
So before jumping to conclusions when analyzing year over year data, consider the following:
- Do you have five or more years worth of data to provide for meaningful trend analysis?
- Have there been any major events (e.g. weather, emergency outages) during these years that might trigger an outlier?
- Has the data been consistently captured across all of the years? Perhaps the system was down for an extended period of time or the data entry person was a week behind on entry.
- Have new technologies been introduced or promoted? For example, the introduction of web registration or a new online application could have a major impact on enrollment patterns.
Without considering these possibilities the conclusions you draw could adversely impact the decisions that you make. Having said that, I feel compelled to caution that if you aren’t sure what you would do differently with this information, any analysis conducted in this area is purely academic and may not actually help you run your institution more effectively. Ultimately you may be better served to focus analysis on more aggregate year-over-year numbers such as for the term or for a given month then on particular days or weeks during an enrollment cycle.