Analyzing Model Effectiveness

  • Article

You can test the effectiveness of an analysis model by using it to predict new data. The Predictor resource provides "scores" that measure how accurately a model predicts new data. You can use these scores to measure the effectiveness of analysis models as you adjust the values used for the Sample Size and Input and Output Property Fraction parameters. You use the parameter Measured Accuracy Sample Fraction to set the number of cases used for testing. A effectiveness and accuracy of a model is directly proportional to the scores.

The Predictor resource includes two scores:

  • Recommendation Score, which is designed to measure the quality of your analysis model when used to recommend products for cross selling or cross browsing. It measures the quality of a ranked list of recommendations returned from the model.
  • Data Fit Score, which is designed to measure the quality of predictions made by an analysis model when filling in missing data, such as missing user properties.

Both the Recommendation score and the Data Fit score evaluate only the ability of the model to make predictions on the cases in the test data. Consequently, in order for either of the scores to be a significant indicator of the actual performance of an analysis model:

  • The test data needs to approximate the distribution of cases that are likely to be seen in practice.
  • The test data must not overlap with the data used to build the analysis model.

If the scores vary significantly depending on the particular test data sampled for scoring (for example, by changing the Measured Accuracy Sample Fraction parameter slightly, you observe a very different score), then your test data is likely to be inadequate.

You may also see a high degree of variability if you have an insufficient number of input or "training" cases.

You can view these scores in Commerce Server Manager under the Models node. The scores are stored in your Commerce Server Data Warehouse in the PredictorModels table under the columns DataFitScore and RecommendScore.

This section contains:

Copyright © 2005 Microsoft Corporation.
All rights reserved.