Showing posts with label Score Matching. Show all posts
Showing posts with label Score Matching. Show all posts

Wednesday, April 16, 2008

ADAPA's computed value is not the same as the expected value, why is that?

There are many reasons for the validation test to fail. I can think of two main reasons.

1) The model ADAPA loaded and executed may be different than the model you built in your development environment. This may reflect a problem with ADAPA or see below.

2) It may be the case that the PMML file you got out of your model development environment does not really represent all aspects of the model or is problematic semantically speaking.

In both cases, you can try to follow ADAPA's decisions by clicking on the computed value which is a hyperlink and follow through its log of computations which are displayed as a text file. This may be very helpful in determining why ADAPA generated the value(s) it did.

Also, the problem may have to do with your data validation file itself. It may be the case that you generated your model in SPSS, for example, exported it as a PMML, converted it using the iGoogle converter and uploaded it into ADAPA. So far so good, but how about the data? If you saved your data in SPSS as well, you have to make sure you saved the expected value or prediction with the correct name. SPSS usually calls this value "PRE_1." You will need to change the name of this variable to the name of the predicted variable defined in the PMML file. Also, if your data contains the original target used to build the model, you will need to rename it to something different than the predicted variable. Your new predicted variable now should be the predicted result you got out of SPSS or any model development environment you used to score the data in the first place.

Friday, February 8, 2008

How can I test my model once it is successfully uploaded into ADAPA?

This is a good question. Given that you built your model outside of ADAPA, you want to make sure that both ADAPA and your development environment produce exactly the same results.

ADAPA provides an integrated testing process to make sure your model was uploaded and works as expected. It allows for a test file containing from 1 to thousands of records with all the necessary input variables and the expected result for each record to be uploaded for score matching.

This can be done easily through its web management console. After processing the file, ADAPA returns statistics on total amount of matched and unmatched records, percentages, etc. If any records failed the matching test, a complete list of all failed records is displayed. One can then peer through computed information for each record to locate where expected and computed values differed and thus pinpoint the source of the problem.

PMML also offers a Model Verification element for similar testing purposes. In this way, verification records are part of the PMML file itself. As of now, ADAPA does not support the Model Verification PMML element.