JP5907649B2 - Method and apparatus for data analysis - Google Patents

Description

(Citation of related application)
This application is a continuation-in-part of US patent application Ser. No. 11 / 535,851 (filed Sep. 27, 2006) entitled “Methods and Apparatus for Hybrid Country Detection”, and “Methods and Apparatus”. US Patent Application No. 10 / 817,750 (Filing Date 2 April 2004) entitled “For Data Analysis” and “Advanced Automatic Parametric Detective Technology fort. Claimed priority of US Provisional Patent Application No. 60 / 774,682 (Filing Date: February 17, 2006), and disclosed each application. Which is incorporated by irradiation. However, if this disclosure conflicts with any of the referenced applications, this disclosure should prevail.

  Semiconductor companies test components to make sure that they work properly. Test data may come from a variety of sources, such as parametric electrical testing, optical inspection, scanning electron microscopy, energy dispersive x-ray spectroscopy, and focused ion beam processes for defect analysis and fault isolation . Testing is typically performed before device packaging (at the wafer level) and after assembly is complete (final testing).

  Collecting and analyzing test data is expensive and time consuming. An automatic tester applies a signal to a component and reads the corresponding output signal. The output signal can be analyzed to determine if the component is operating properly. Each tester generates a large amount of data. For example, each tester may perform 200 tests on a single component, and each of these tests may be repeated 10 times. As a result, testing a single component yields 2000 results. Since each tester tests more than 100 components per hour and multiple testers may be connected to the same server, the test process generates a huge amount of data.

  Furthermore, most of the data interpretation is done by engineers, who check the data and deduct the test and manufacturing process based on their experience and knowledge about the manufacturing and testing process. Although manual analysis is often effective, engineers tend to arrive at different subjective conclusions based on the same data because they understand manufacturing and test systems differently. Another problem arises when skilled personnel leave the company or are unavailable, because the knowledge and understanding of the personnel regarding manufacturing and test systems and the interpretation of test data cannot be easily communicated to other personnel.

The present invention will be described with respect to functional block components and various process steps. Such functional blocks and steps can be realized by a plurality of hardware or software components configured to perform a specific function. For example, the present invention can use various testers, processors, storage systems, processes, and algorithms such as statistical engines, memory elements, signal processing elements, neural networks, pattern analyzers, logic elements, programs, etc. Various functions can be performed under the control of one or more testers, microprocessors, or other control devices. In addition, the present invention can be implemented in conjunction with multiple test environments, and each described system is just one exemplary application of the present invention. Furthermore, the present invention can use a number of conventional techniques for data analysis, component interface, data processing, component handling, and the like.
(Item 1)
A semiconductor test data analysis system for identifying outliers in semiconductor test data,
A memory configured to store the test data;
A processor connected to the memory;
The processor comprises:
Retrieving the test data from memory;
Comparing the test data to an outlier upper threshold and an outlier lower threshold, the upper threshold and outlier lower threshold being derived from the median of the frequency distribution of the test data; ,
Determining whether the test data includes an outlier according to a comparison between the upper threshold of the outlier and the lower threshold of the outlier of the test data;
A semiconductor test data analysis system configured to perform:
(Item 2)
The upper threshold of the outlier is derived from the third quartile,
The semiconductor test data analysis system according to item 1, wherein the lower threshold of the outlier is derived from the first quartile.
(Item 3)
The semiconductor test data analysis system according to item 1, wherein the threshold value is asymmetric with respect to the median value.
(Item 4)
The semiconductor test data analysis system according to item 1, wherein the processor is configured to adjust the threshold value according to a slope of the frequency distribution.
(Item 5)
The semiconductor test data analysis system according to item 1, wherein the processor is configured to classify the outlier according to the magnitude of the outlier.
(Item 6)
The semiconductor test data analysis system of item 1, wherein the processor is configured to assign a category to the outlier according to a range between at least one of the thresholds and an edge of the distribution.
(Item 7)
The semiconductor test data analysis system of item 1, wherein the processor is configured to determine whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters.
(Item 8)
A method for identifying outliers in semiconductor test data, comprising:
Establishing an upper threshold for outliers and a lower threshold for outliers according to the test data, the upper threshold for outliers and the lower threshold for the outliers being derived from the median of the frequency distribution of the test data And
Comparing the test data to an upper threshold for the outliers and LCL;
Identifying outliers in the test data according to a comparison of the test data with UCL and LCL;
A method of identifying outliers, including
(Item 9)
Establishing the upper threshold for the outlier includes identifying a third quartile;
The method of identifying an outlier according to item 8, wherein establishing the upper threshold for the outlier includes identifying a first quartile.
(Item 10)
9. The method of identifying an outlier according to item 8, wherein the threshold is asymmetric with respect to the median.
(Item 11)
Item 9. The method of item 8, wherein establishing the outlier upper limit threshold and the outlier lower threshold includes adjusting the threshold according to the slope of the frequency distribution.
(Item 12)
9. The method of identifying an outlier according to item 8, further comprising classifying the outlier according to the magnitude of the outlier.
(Item 13)
9. The method of identifying outliers according to item 8, further comprising assigning a category to the outliers according to a range between at least one of the threshold values and an edge of the distribution.
(Item 14)
The method of identifying an outlier according to item 8, further comprising determining whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters.
(Item 15)
A medium storing instructions for causing a computer to execute a process, the process comprising:
Retrieving a series of semiconductor test data from memory;
Establishing an upper threshold for outliers and a lower threshold for outliers, the upper threshold for outliers and the lower threshold for outliers being derived from the median of the frequency distribution of the test data; ,
Comparing the test data to an upper threshold for the outlier and a lower threshold for the outlier;
Determining whether the test data includes an outlier according to a comparison between the upper threshold of the outlier and the lower threshold of the outlier of the test data;
Including medium.
(Item 16)
Establishing the upper threshold for the outlier includes identifying a third quartile;
16. The medium of item 15, wherein establishing the upper threshold for outliers includes identifying a first quartile.
(Item 17)
Item 16. The medium according to Item 15, wherein the threshold value is asymmetric with respect to the median value.
(Item 18)
Item 16. The medium according to Item 15, wherein establishing the upper threshold for outliers and the lower threshold for outliers includes adjusting the thresholds according to the slope of the frequency distribution.
(Item 19)
16. The medium of item 15, wherein the process further comprises classifying the outlier according to the magnitude of the outlier.
(Item 20)
16. The medium of item 15, wherein the process further comprises assigning a category to the outlier according to a range between at least one of the thresholds and an edge of the distribution.
(Item 21)
16. The medium of item 15, wherein the process further comprises determining whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters.

FIG. 1 is a block diagram of a test system in accordance with various aspects of the present invention and associated functional elements. FIG. 2 is a diagram showing test results including outliers and defects. FIG. 3 is a diagram illustrating a system for automatically selecting one or more outlier identification algorithms. FIG. 4 is a diagram showing test results including hidden outliers. FIG. 5 is a diagram showing a sample path along which the wafer and the test probe follow the wafer. FIG. 6 is a diagram illustrating changes in test data when the prober follows the path. FIG. 7 is a flowchart for identifying hybrid outliers. FIG. 8 shows a wafer and a spatial analysis window. FIG. 9 is a diagram showing potential filter types for proximity analysis. FIG. 10 is a diagram illustrating a representative example of an asymmetric distribution. 11A and 11B are correlation charts of test data from two different tests. 11A and 11B are correlation charts of test data from two different tests. FIG. 12 is a map of wafer test results.

  A more complete understanding of the invention can be derived by reference to the detailed description and claims when considered in conjunction with the following illustrative drawings, which may not be in proportion. Throughout the drawings, the same reference numerals are used for the same elements.

  Elements in the drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the relationship between some elements in the drawings and the steps they perform may be exaggerated or omitted with respect to other elements to make the embodiments of the invention easier to understand.

  Referring to FIG. 1, methods and apparatus in accordance with various aspects of the present invention operate in conjunction with a system 100 having a tester 102, such as an automatic test equipment (ATE) for test semiconductors. In the present embodiment, the test system 100 includes a tester 102 and a computer system 108. The test system 100 can be configured to test any component 106, such as a semiconductor device, circuit board, packaged device, or other electrical or optical system on a wafer. Note that various aspects of the present invention can be applied to many environments with multiple data points, such as credit card fraud detection, athlete performance analysis, fraud voting analysis, and advanced weather forecasting. . In this embodiment, component 106 comprises a plurality of integrated circuit dies formed on a wafer, or a packaged integrated circuit or device. Component 106 is created using a manufacturing process, which may include any suitable manufacturing process for creating component 106. This process can also include a test process that can include any suitable process for testing the operation of component 106.

  Tester 102 suitably comprises any testing device that tests component 106 and generates output data associated with the test, and may comprise multiple machines or other data sources. The tester 102 can comprise a conventional automatic tester such as a Teradyne tester and operates properly in conjunction with other equipment to facilitate testing. Tester 102 may be selected and configured according to the particular component 106 to be tested and / or any other suitable criteria.

  The tester 102 operates in conjunction with the computer system 108 to, for example, program the tester 102, load and / or execute a test program, collect data, provide instructions to the tester 102, analyze test data, Parameters can be controlled. In the present embodiment, the computer system 108 receives tester data from the tester 102 and executes various data analysis functions regardless of the tester 102. The computer system 108 comprises an independent computer having a processor 110 and a memory 112, such as a personal computer or workstation, connected to or networked with the tester 102 to exchange signals with the tester 102. be able to. In another embodiment, the computer system 108 can be omitted from or integrated with other components of the test system 100, and various functions can be performed on the tester 102 or network-connected elements. Can be executed by other components.

  The memory 112 suitably stores a component identifier for each component 106, such as xy coordinates corresponding to the position of the component 106 on the wafer map of the tested wafer. Each xy coordinate in memory 112 can be associated with a particular component 106 at a corresponding xy coordinate on the wafer map. Each component identifier has one or more fields, and each field can be, for example, a specific test performed on component 106 at a corresponding xy location on the wafer, statistics associated with the corresponding component 106, or other Corresponds to related data. The memory 112 can be configured to contain any data that the user has identified as needed, according to any criteria or rules.

  The computer 108 of this embodiment also suitably has access to a storage system, such as another memory (or a portion of the memory 112), a hard drive array, an optical storage system, or other suitable storage system. The storage system can be local, such as a hard drive dedicated to the computer 108 or tester 102, or remote, such as a hard drive array associated with the server to which the test system 100 is connected. The storage system can store programs and / or data for use by the computer 108 or other components of the test system 100. In this embodiment, the storage system comprises a database 114 that can be used via a remote server 116 comprising a main production server for manufacturing equipment, for example. Database 114 stores tester information such as tester data files, master data files for operating test system 100 and its components, test programs, downloadable instructions for test system 100, and the like. In addition, the storage system can include a complete tester data file, such as a past tester data file that is retained for analysis.

  The test system 100 can include additional equipment to facilitate testing of the component 106. For example, the test system 100 may process a device 106 and provide a device interface 104, such as a conventional device interface board and / or device handler or prober, to provide an interface between the component 106 and the tester 102. Is provided. Test system 100 comprises other components, devices, software, etc. to facilitate testing of component 106 according to the particular configuration, application, environment, or other relevant factors of test system 100, or You can connect to these. For example, in this embodiment, the test system 100 may use a suitable communication medium, such as a local area network, the Internet, or a global network such as the Internet, to send information to other systems such as the remote server 116. Connected to.

  The test system 100 can include one or more testers 102 and one or more computers 108. Further, the computer 108 can be separate from the tester 102 or integrated into the tester 102 using, for example, one or more processors, memory, clock circuits, etc. of the tester 102 itself. In addition, various functions can be performed on different computers.

  Test system 100 according to various aspects of the present invention tests component 106 to provide extended analysis and test results. For example, extended analysis can identify false, suspicious, or unusual results. The test system 100 can also analyze multiple sets of data, such as data acquired from multiple wafers and / or multiple wafers, to generate composite data based on multiple data sets. Further, the test data may include one or more process control or electrical test data (ET) related to electrical characteristics for various points on the wafer and / or the component 106, pass / fail binning classification for the component 106. Data from multiple sources, such as wafer bin map data, outlier data and outlier signature data, and outlier classification data such as outlier classification as SMALL, MEDIUM, or CRITICAL according to selected criteria Can be included. Various data may also be used by the test system 100 to produce, test, and / or other aspects such as problems, inefficiencies, potential hazards, instabilities, or other aspects that can be distinguished from test data. The characteristics in the process can be diagnosed. Operators such as product technicians, test engineers, manufacturing engineers, device engineers, or other personnel using test data and analysis can then use the results to test system 100 and / or manufacturing system. Validation and / or improvement and classification of component 106 can be performed.

  Test system 100 according to various aspects of the present invention performs an extended test process for testing component 106 and for collecting and analyzing test data. The test system 100 operates appropriately in conjunction with a software application executed by the computer 108. The software application of this embodiment includes a plurality of elements for performing an extended test process including components, supplemental data analysis elements, and output elements. The test system 100 can also include synthetic analysis elements for analyzing data from more than one data set. In addition, the test system can include a diagnostic system for identifying characteristics and potential problems using the test data.

  Each software element suitably comprises software modules that run on computer 108 to perform various tasks. In general, the components prepare the test system 100 for testing and analysis. In the supplemental data analysis element, output test data from tester 102 and / or other sources is analyzed to generate supplemental test data in an appropriate runtime and automatically in conjunction with or after the inline process. The supplemental test data is then transmitted to the operator or another system, such as a synthetic analysis element, diagnostic system, and / or output element.

  The test system 100 starts a test operation according to a test program, for example, in conjunction with a series of conventional tests. The tester 102 can perform multiple tests on each component 106 on the wafer or on the wafer itself, and each test can be repeated multiple times on the same component 106. Tests include (but are not limited to) any suitable test such as continuity, supply current, leakage current, parametric static, parametric dynamic, and functional stress tests. Test data from tester 102 is stored for quick access and supplemental analysis when the test data is acquired. Data can also be stored in long-term memory for further analysis and use.

  As tester 102 generates test results, output test data for each component, test, and iteration is stored by tester 102 in a tester data file. The output test data received from each component 106 is classified by the tester 102, classifying the performance of the component 106, for example by classifying it into a specific bin by comparing with the upper and lower test limits, and the result of the classification is also the tester Stored in the data file. The tester data file can also include additional information such as logistics data and test program identification data. The tester data file is then provided to the computer 108 in an output file, such as a standard tester data format (STDF) file, and stored in memory. The tester data file can also be stored in a storage system for long-term storage for later analysis, such as by a synthetic analysis element.

  When the computer 108 receives the tester data file, the supplemental data analysis element analyzes the data and provides an extended output result. The computer 108 can provide any suitable analysis of the tester data to achieve any suitable purpose. For example, the supplemental data analysis element 206 can implement a statistical engine to analyze the output test data and identify characteristics of the data and associated data at runtime or later. The identified data and characteristics can be stored, but the unidentified data can be stored or discarded for processing. The supplemental data analysis element 206 can perform supplemental data analysis at runtime as an inline process or as an offline process. This supplemental data analysis is performed as an inline process, i.e. as an automated function incorporated into the test process.

  The computer 108 can perform further analysis functions on the generated analysis and output test data. Each test produces at least one result for at least one component. Referring to FIG. 2, an exemplary set of test results for a single test of a plurality of components includes a first set of test results that have statistically similar values and values that are out of the first set. And a second set of test results characterized by: Each test result can be compared with an upper test limit and a lower test limit. A component is classified as “defective” or classified according to tests and / or test results if a particular result of the component exceeds any limit.

  Some of the second set of test results out of the first set may exceed the control limit, but some test results do not. For this purpose, test results that are out of the first set but not exceeding the control limit, or test results that fail to be detected, are referred to as “outliers”. Outliers are generally considered views that appear to be inconsistent with the rest of the set of data. Outliers in test results can be identified and analyzed for appropriate purposes, such as identifying potentially unreliable components. Outliers can also be used for various potential problems and / or improvements in testing and manufacturing processes.

  Analysis of each relevant data according to the selected algorithm appropriately identifies global and / or hybrid outliers. If a particular algorithm is inappropriate for a set of data, the computer 108 can select a different algorithm. The computer 108 can operate in any suitable manner to specify outliers, such as by comparison with preselected values or dynamic values. For example, an outlier identification system according to various aspects of the present invention first automatically calibrates sensitivity to outliers based on selected statistical relationships for each relevant data or other data. Some of these statistical relationships can then be used to define relative outlier threshold limits, other reference points such as thresholds or data modes, mean values, or medians, or combinations thereof. Compared with In this embodiment, the statistical relationship is scaled by, for example, the standard deviation of 1, 2, 3, and 6 data to define different outlier widths. The output test data can then be compared to outlier threshold limits to identify and classify the output test data.

  The computer 108 stores the obtained statistics and outliers and corresponding identifiers such as the coordinates of the xy wafer map. Selected statistics, outliers, and / or failures also trigger notification events, such as sending an electronic message to the operator, starting the light tower, shutting down the tester 102, or notifying the server.

  In the present embodiment, the supplemental data analysis element 206 includes a scaling element 210 and an outlier classification engine 212. The scaling element 210 is configured to dynamically scale selected coefficients and other values, eg, according to output test data. Outlier classification engine 212 is configured to identify and / or classify various outliers in the data according to a selected algorithm.

  More specifically, the scaling element of this embodiment uses various statistical relationships to dynamically scale outlier sensitivity. The scaling factor is calculated by the scaling factor and used to change the sensitivity value of the selected outlier. Any suitable criteria can be used for scaling, such as a suitable statistical relationship.

  Outlier classification engine 212 is suitably configured to identify and / or classify outliers in component 106 and output analysis results in accordance with test data and / or any suitable algorithm. In addition, the outlier classification engine 212 can be configured to identify one or more algorithms suitable for identifying outliers in the output test results using a plurality of candidates for outlier identification algorithms. . Different tests generate different populations so that an outlier identification algorithm that is appropriate for one test may be inappropriate for another test. Outlier classification engine 212 is suitably configured to distinguish differences between data populations based on the data population of the current data and automatically select one or more outlier identification algorithms. For automatic selection, any suitable set of outlier identification algorithm candidates can be selected, and the selection can be made according to any suitable criteria and analysis.

  For example, referring to FIG. 3, the outlier classification engine 212 can be configured to automatically perform an outlier identification algorithm selection process. One or more suitable algorithms can be selected from a plurality of algorithms to perform an outlier identification algorithm selection process to identify global and / or hybrid outliers. In one embodiment, the outlier classification engine 212 suitably comprises a preprocessing engine 310 and a classification engine 312. Pre-processing engine 310 generates data appropriately to facilitate selection of associated outlier identification algorithms. Classification engine 312 appropriately selects one or more associated outlier identification algorithms to identify global and / or hybrid outliers, respectively.

  Output test data (eg, data obtained from a particular test) is first provided to the outlier classification engine 212 to analyze the output test data for compatibility with various candidates for outlier identification algorithms. . The data can be analyzed in any suitable manner to identify an appropriate algorithm for identifying outliers in the output test data. For example, in this embodiment, the preprocessing engine 310 prepares an available outlier identification algorithm, such as by receiving output test data and retrieving an outlier identification algorithm from an outlier identification algorithm library stored in memory. To do. The preprocessing engine 3010 analyzes output test data for outliers using a plurality of available algorithms. In this embodiment, the pre-processing engine 310 analyzes the output test data using each of the algorithms specified by the user or other suitable algorithm selection to determine all algorithms, and the minimum, maximum, Pre-processing data such as outliers identified by various descriptive statistics such as mean, median, standard deviation, CPK, CPM are generated.

The algorithm can be based on industry standards (eg, IQR, median +/− N ^* sigma, etc.) and / or dedicated, special, or user-defined outlier identification techniques. The outlier identification algorithm library can be set appropriately by the user, for example, according to the characteristics of the particular product under test or the test being performed, for example to add, remove or edit outlier identification algorithms. Different algorithms may be appropriate for different statistical population types, such as normal, lognormal, bimodal, fixed, or low CPK data populations. Candidate outlier identification algorithms can include suitable algorithms for various types of data and distributions such as: inter-quartile range (IQR) normal distribution, 3 sigma; IQR normal distribution IQR lognormal, 3 sigma; IQR lognormal, 6 sigma; bimodal algorithm; fixed algorithm; low function algorithm; custom algorithm based on 3 sigma, 6 sigma, or n sigma; and various sensitivities Dedicated algorithm that has.

  The result of the preprocessing algorithm is dynamically selected for detection of global and / or hybrid outliers. In this embodiment, the outlier classification engine 212 analyzes the test results generated by the preprocessing engine 310 to identify the most useful or applicable outlier identification algorithm. Data from the selected outlier identification algorithm can be retained, but the remaining data is discarded. For example, in this embodiment, the classification engine 312 receives the result of the preprocessing analysis generated by each of the available outlier identification algorithms. The classification engine 3012 analyzes the preprocessing data according to any suitable criteria, such as predetermined and / or user-defined recipe-driven rules, to determine whether the preprocessing data meets various criteria. Judging.

  This rule can use statistical ratios or values, such as comparative statistics such as minimum, maximum, average, median, standard deviation, CPK, and CPM against thresholds or other criteria. For example, the classification engine 3012 can skip outlier detection processes under certain circumstances, such as test results that are too few, test results that are too narrow, or a bimodal distribution. This rule can be pre-selected by the user and / or adjusted or added to correspond to a specific state of the product and test environment. Further, the classification engine 312 can be configured to apply a particular algorithm to a particular type of test, for example when it is known that there is a particular distribution of test results. Other rules can determine whether a particular test is applicable. For example, the classification engine 312 can compare the CPK with a threshold value. If CPK is below the threshold, an IQR normal outlier identification algorithm can be used. In this system, the result from the algorithm that satisfies the rule is used to identify outliers. Other algorithm results for that test are ignored appropriately.

  Outlier classification engine 212 can also identify and classify selected global and / or hybrid outliers and components 106 according to test results and information generated by supplemental analysis element 206. For example, the outlier classification engine 212 is configured to classify the component 106 into a category of critical / marginal / good parts in conjunction with the following user-defined criteria: user-defined Good / bad spatial pattern recognition; classification of data related to tester data compression; qualification and analysis of test setup in situ sensitivity; leveling analysis provided by tester; dynamic for part placement and dynamic retesting Wafer map and / or specimen mapping; or analysis of test program optimization. Outlier classification engine 212 can classify the component 106 and associated data according to conventional SPC control rules, such as Western Electric rules or Nelson rules, to characterize the data.

  Outlier classification engine 112 appropriately classifies the data using a selected set of classification limit calculations. Any suitable classification method can be used to characterize the data according to the requirements of the operator. For example, the outlier classification engine 212 can be statistically determined from one, two, three, and six thresholds, such as, for example, test limits or data intermediate values, modes, and / or intermediate values. Outliers are classified by comparing a selected threshold, such as a value corresponding to the scaled standard deviation, with the output test data. Such outlier identification tends to normalize every identified outlier for any test, regardless of the width of the data and relative noise.

  In one embodiment, the outlier threshold may be defined asymmetrically with respect to a selected center point in the data distribution. By using an asymmetric threshold, the effects of non-Gaussian distributions and / or the presence of outliers on outlier identification can be reduced. Outlier thresholds can be selected based on the data that the outliers can affect without determining a particular position in the data, such as intermediate, or a scale, such as standard deviation. For example, the outlier threshold can be selected according to the data distribution, such as by deriving the threshold from the center point of the frequency distribution of the test data. In one embodiment, referring to FIG. 10, instead of the outlier threshold being based on the standard deviation from the center point, the outlier thresholds 1010 and 1012 are adjusted or selected according to the quartile values 1014 and 1016 for the series of data. be able to. The interquartile value corresponds to the median between the median of the complete data set and the edges of the distribution.

  In addition, thresholds 1010 and 1012 can be adjusted according to any relevant criteria, such as the distribution of data. In one embodiment, thresholds 1010 and 1012 are adjusted according to a generalized slope of the data distribution within the relevant region, such as first and third quartile points 1014 and 1016. For example, thresholds 1010 and 1012 can be adjusted by an amount that is inversely related to the general slope of the data in the relevant region. As a result, in regions with a small slope, thresholds 1010 and 1012 may be farther from the center point than regions with a large slope. Therefore, the threshold upper limit 1010 and the threshold lower limit 1012 can be selected according to the following equations.

UCL = Q3 + N * UQW
LCL = Q1-N * LQW
Where UCL is the upper threshold 1010 for outliers, LCL is the lower threshold 1012 for outliers, Q1 is the first quartile, Q2 is the third quartile, N is a constant, UQW and LQW are slopes Is the weight assigned according to the inverse correlation to.

  Additional threshold values can be defined as well, for example, to define SMALL, MEDIUM, or LARGE outliers. Thresholds and categories can be defined in any way. For example, the category threshold may be determined according to a statistical relationship scaled by, for example, one, two, three, and six standard deviations in the data from the median to define different outlier amplitudes. it can. The output test data can then be compared to outlier threshold limits to identify and classify the output test data.

  Alternatively, categories can be defined for outlier thresholds. The boundaries between the various categories can be selected according to any suitable criteria, such as according to outlier thresholds, test limits, and / or distribution edges. For example, the range of outlier thresholds and test limits can be divided into two or more categories 1020, 1022, 1024. The range of outlier thresholds and test limits can be divided into equal or different regions so as to define different magnitudes of outliers. In this embodiment, each side of the distribution between the outlier threshold and the edge of the distribution is divided into three or four equal categories 1020, 1022, and 1024, large, medium, small, and / or An outlier is defined as tiny. The size and number of the various categories can be the same on either side of the distribution or can vary for each side of the distribution.

Outlier classification engine 112 analyzes and associates normalized outliers and / or raw data points based on user-defined rules. The outlier classification engine 112 appropriately classifies according to each test, but the classification can be performed independently of other tests or in conjunction with relevant data from other tests. Any suitable criteria can be used to classify components based on test failures and outliers such as:
FAIL if the part fails in at least one test;
CRITICAL-if at least one test is identified as having a LARGE outlier, or two different tests for other acceptable parts are determined to have at least two MEDIUM outliers;
MARGINAL-if at least one test is identified as having a MEDIUM outlier, or four different tests on other passing parts are identified as having at least four SMALL outliers;
SMALL—when at least one test is identified as having a SMALL outlier with respect to other acceptable parts;
PASS-if other than SMALL, MEDIAUM, or LARGE outlier for other acceptable parts.

  The criteria for small, medium, or large outliers can be selected based on any suitable criteria, such as test limits and / or thresholds according to the characteristics of the data.

  Outlier classification engine 112 may also include a hybrid outlier system configured to identify hybrid outliers in the data, also referred to as local outliers or hidden outliers. A hybrid outlier is an inconsistent data point within a local group of data, but does not intersect the threshold that defines the “global” outlier boundary of the main population. Referring to FIG. 4, the global outlier is located outside the data distribution of the main data population. However, the hybrid outlier is not located outside the data distribution of the main data population, but may deviate significantly from the standard when multiple parameters are considered. Accordingly, the outlier classification system can analyze test data and identify outliers according to two or more parameters. The other parameters can be any suitable data, such as spatially related components, data obtained from correlated tests, or other relevant data. For example, using spatial information, it is possible to recognize a device whose parameters are all in the main distribution but do not match the data in a more local physical region. In addition, by using a high level of correlation between two parameters, it is possible to identify devices whose parameters individually match the main distribution but do not match in relation to another parameter.

  The degree of correlation between two parameters, such as two test results, can be evaluated in any way. For example, the correlation index can be calculated using linear or logarithmic regression methods that establish a best-fit line between two data sets. The closer the slope of this line is, the more unity and the higher the degree of correlation. The step of determining the degree of correlation includes any relevant considerations or processes, such as removing LARGE outliers or compensating for multisite changes.

  Once the expected degree of correlation is determined, the correlation of the test results can be determined to identify outliers that do not fit the underlying relationship. Referring to FIG. 11A, for a highly correlated test, all results are expected to be within a certain distance 1112 from the best-fit straight line 1110. Referring to FIG. 11B, if there is an out-of-range result 1114, the entire correlation index is slightly affected, but the corresponding data point is the best even though the test result remains within the main distribution for the relevant parameter. It exists outside the expected distance 1112 from the fitted straight line 1110. The computer 108 can also select a situation for performing a correlation analysis, for example, according to the degree of correlation between the two parameters, the usefulness of computational resources, and the requirements for optimal analysis.

  In one embodiment, in the light of component test data associated with spatially related component data, such as components from smaller local populations, such as temporal or spatial local populations. The value can be identified. For example, referring to FIG. 5, the tester generates data by sequentially testing the components on the wafer. The tester moves the prober between components and accumulates test data for each component. Due to changes in manufacturing processes and materials, and test processes, test data may vary periodically as the prober crosses the wafer (FIG. 6). Data points 610 that are well outside the variation for single or multiple wafers are typically classified as global outliers. Even if a data point 604 is spatially adjacent or significantly different from the data point of the corresponding component, a data point 604 that is within or slightly beyond its variation is typically Avoid classification as outliers.

  Hybrid outliers are properly identified by analyzing individual raw, normalized, or other processed data points against adjacent data points. The outlier classification engine 112 can be applied to hybrid outlier thresholds calculated using spatially related component data by comparing parametric or other data for individual components. Proximity analysis can also be weighted according to, for example, the distance between a central component and an adjacent component. Proximity analysis is a type of proximity analysis engine that is used to generate composite representations based on multiple data sets, such as a dedicated proximity analysis engine, or a proximity analysis engine associated with another task, Can be done by any suitable system.

  In one embodiment, when parametric data is processed, the outlier classification engine 112 can calculate a hybrid outlier threshold for a local data set, such as data generated just before a selected number. . The data to be analyzed may include any suitable data such as previous data points, subsequent data points, both previous and subsequent data points, temporally related data points, or spatially related data points. it can. In addition, the number of data points in the local data set can be any number, such as using a predetermined number of data points or selecting the number of data according to various criteria such as data variability or noisiness. Selection can be made according to suitable criteria. If the data is noisy, the number of data points can be adjusted automatically, for example by increasing the number of data points in the local data set, to reduce the effects of noise. The hybrid outlier threshold can be dynamically recalculated by accumulating new parametric data by using, for example, a first-in first-out (FIFO) calculation. Parametric data for components in the region can then be compared to a threshold value to identify hybrid outliers.

  Outlier classification engine 112 may be configured to identify hybrid outliers 604 according to any suitable mechanism or process. In this embodiment, the hybrid outlier 604 is identified in conjunction with the inline process. That is, automatically through a process of receiving data and providing results automatically without operator intervention or other action when the data is generated or after the test is completed and the data is stored To be identified. Alternatively, the hybrid outlier 604 can be identified at run time or after completing the analysis and storing the data. The outlier classification engine 112 can be incorporated into a test program executed by the tester 102. The test data can include any type of test data, such as bin results or parametric test data. In addition, by removing data for components categorized as global outliers and / or defects, or by normalizing data across various sections of the wafer, such as sections associated with multi-site testing or stepper fields, etc. Data pre-processing can be performed.

  Referring to FIG. 7, in this embodiment, the outlier classification engine 112 or other outlier identification system first filters the data to remove global outliers and processes the remaining data to mix outliers. Is identified (710). Outlier classification engine 112 may further process the filtered data to normalize the data (712), for example, against existing variation sources. Outlier classification engine 112 can then identify the hybrid outlier according to any suitable technique or criteria. For example, the outlier classification engine 112 may select and / or adjust a window containing a selected number and geometry of components to identify spatially related components (714). The outlier classification engine 112 can then determine 716 the component neighborhood values that correspond to the test data of the nearby components. The component test data can then be analyzed for neighborhood values to identify hybrid outliers.

  In addition, the data can be normalized, which includes adjusting the data values to correct for differences in the data that can be generated by known causes that adversely affect the consistency of the data. In this embodiment, the hybrid outlier system normalizes the test data for each component associated with the test data for all other components on the wafer. For example, when test data is generated using a multi-site test where multiple resources test a wafer or component.

  The data can be normalized in any suitable manner, such as addressing known variable sources. Components that share a known common conflict source can be normalized using the same normalization criteria. For example, in this embodiment, parametric test data generated in conjunction with a multi-site test environment can be normalized according to the following equation.

Here, in the case of site n, norm_site _n is the obtained normalized data, data_site _n is data normalized in advance, median_site _n is the median of statistics, and iqr_site _n is the four of the sites to be analyzed. Quantile range. The normalized data can be stored in any suitable manner, for example on a device map of normalized data having normalized data for each device.

  To detect hybrid outliers, the outlier classification engine 112 analyzes local data such as spatially close component data. For example, the outlier classification engine 112 can detect hybrid outliers according to the geometry of the wafer. Hybrid outliers can be identified according to any suitable technique or process. In this embodiment, the outlier classification engine 112 analyzes data of adjacent components.

  In one embodiment, the computer 108 can perform proximity analysis for each component on the wafer. For example, the computer 108 can be configured to identify hybrid outliers by identifying outliers in a subset of the entire data set, such as in data for a selected group of components on the wafer. In one embodiment, referring to FIG. 8, the computer 108 can build a pattern or window 810 of a selected size that can be used to select a plurality of components on the wafer 812. Pattern or window 810 includes boundaries that define associated spatial regions, such as a subset of components on wafer 812. The size of window 810 can be selected or adjusted according to any suitable criteria, such as the number of components adjacent to the central component, the type of test or component, or the desired sensitivity of outlier analysis. The shape of the window can also be selected according to any suitable criteria, such as the associated spatial region. For example, if it is not easy to use a rectangular window in the target area, such as near the edge of the curve of the wafer 812, other shapes can be created for the window 810 to correspond to the relevant area of the wafer 812. You can choose.

  Referring to FIG. 9, in one embodiment, the outlier classification engine 112 establishes an associated window using one or more predetermined windows or masks that can be applied to each component in the data set. Can do. The size and shape of the window can be selected according to any suitable criteria, such as following a preselected type, an operator designation, or an automatic selection process. The user can either predefine the type and size of the window to be used, or the outlier classification engine 112 can be located near the edge of the wafer, possible sampling factors, gaps on the wafer or lost data, etc. Depending on the nature of the area surrounding the associated component, the shape and / or size of the window can be automatically adopted.

  In this embodiment, the outlier classification engine 112 requires a selected amount of data, such as data for at least 4-8 (eg, 5) surrounding components, to perform the analysis. In one embodiment, a device is considered a minimum only if it meets one or more selected criteria, such as being classified as a “good” device. If the first area in the window is less than the other five “good” devices, the outlier classification engine 112 determines the size and / or shape of the window or pattern until the number of components reaches the required number. It can be adjusted to increase. In one embodiment, the outlier classification engine 112 adjusts the window or pattern by applying a sequence of predetermined patterns, the sequence of which includes increasing size patterns. In addition, the outlier classification engine 112 can abort the analysis if the minimum number of acceptable data is not available. For example, the outlier classification engine 112 may only analyze and / or adjust the size and / or shape of the window if the device under investigation has at least five “bin 1” devices in the first window. it can. Thus, data for a particular device can only be classified as a hybrid outlier if there is sufficient data for adjacent components in the first window. In other embodiments, alternative data can be used. For example, the lost data can be replaced with ideal data representing the ideal value of each component, or the components in the surrounding bands around the lost data, or the components at corresponding locations on other wafers in the lot. Such as data based on other components on the wafer.

  Window 810 is then applied to various regions of wafer 812 to identify hybrid outliers from the component data in window 810. Outlier classification engine 112 can identify hybrid outliers according to any suitable criteria and technique. For example, the outlier classification engine 112 may first analyze a starting component, such as the leftmost component in the top row of the wafer, and sequentially analyze the data for each component on the wafer. Outlier classification engine 112 can adjust the size and shape of the window as each component is analyzed. The component data in the window is then retrieved from memory, and outliers are analyzed using, for example, only the component data in the window.

  The area within the window can be characterized and / or analyzed according to any suitable criteria or technique. For example, the outlier classification engine 112 can calculate one or more representative neighborhood values for each component based on data of surrounding components. A window is applied to each component to generate a neighborhood value for each component of the associated test. The data for each component can then be analyzed and / or compared with corresponding neighborhood values to identify hybrid outliers.

  In this embodiment, when a window is applied to each component, the outlier classification engine 112 calculates a window neighborhood value corresponding to a derived value derived from the values of neighboring components. For example, the derived value can include an average value representing a median value of multiple values, such as an arithmetic average, a statistical median, a mode, a geometric average, a weighted average, and the like. The outlier classification engine 112 uses the median value as the neighborhood value because the median value can often act as a robust central estimate of the local neighborhood. The median value is then stored on a neighborhood value device map that stores, for example, neighborhood values calculated for each device. Outlier classification engine 112 moves the window to the next component. In addition, the outlier classification engine 112 can apply different weights to surrounding component data, such as according to the proximity of the device to the central device. For example, component data closer to the central component may have a greater influence in calculating neighborhood values than components farther away.

  Hybrid outliers can be identified by analyzing each component's data against the corresponding neighborhood values. Any suitable outlier detection system or technique can be used to identify hybrid outliers. For example, the outlier classification engine 112 can calculate a unique hybrid outlier threshold based on the neighborhood values of each component and compare the raw data with the calculated threshold. Alternatively, the outlier classification engine 112 can generate a value based on the raw data and the neighborhood value of each component and compare that value to a threshold or the like. The location of the identified hybrid outlier is stored along with other relevant information.

  In this embodiment, referring to FIG. 7, the outlier classification engine 112 generates a residual value for each component according to the raw data and the unique neighborhood value (718) and applies the conventional outlier identification technique to the residual. The hybrid outlier is identified (720). The residual component can be determined according to any suitable criteria or technique. In the present embodiment, the remainder is calculated using the following equation.

data _norm is the normalized raw test data value, data _smooth is the relevant neighborhood value, and RMSE is the mean square error of all relevant point values, such as all devices on the wafer. Define the square root of. In this embodiment, RMSE is the root mean square of the sum of the differences between the normalized and smoothed data for each device on the wafer. This formula produces a standardized error, which is a normal distribution with a mean of 0 and a variance of 1.

  The obtained residual data can be stored in a device map of residual data that stores the residual data value of each device, for example. Outlier detection techniques can be applied to residual values of various components to identify outliers. For example, the outlier classification engine 112 can calculate a control limit based on the residual value and identify the residual value that exceeds the limit as a hybrid outlier. For example, the outlier classification engine 112 can analyze the residual data according to one or more outlier identification algorithms to identify outliers in the residual data. The corresponding device can then be designated as the associated test outlier.

  The outlier classification engine 112 can also be configured to perform further analysis of the hybrid outlier data. For example, the outlier classification engine 112 can identify and classify selected hybrid outliers and components 106 that classify the components 106 into a plurality of categories, as described above for global outliers. In one embodiment, hybrid outliers are provided, for example, by user-defined criteria, user-defined spatial pattern recognition, classification of relevant data for tester data compression, qualification and analysis of test settings in situ sensitivity, tester Classified as SMALL, MEDIUM, LARGE hybrid outliers in conjunction with leveling analysis, dynamic wafer map and / or specimen mapping for part placement and dynamic retesting, analysis of test program optimization The

  The outlier classification engine 112 appropriately classifies the hybrid outlier data using a selected set of classification limit calculations. The hybrid outlier data can be characterized using any suitable classification method according to the operator's requirements. The outlier classification engine 112 may, for example, associate the associated test data or derived data with one, two, three, and six intermediate values, modes, and / or intermediates of the relevant geographic region data. Classify hybrid outliers by comparing to a selected threshold, such as a value corresponding to a standard deviation that is statistically scaled from a threshold, such as a value. Such hybrid outlier identification tends to normalize any identified hybrid outlier for any test, regardless of the width of the data and relative noise. In one embodiment, the outlier classification engine 112 may classify the hybrid outlier according to the magnitude of the difference between the central component test data and the derived values of the plurality of local components. The classification can be based on this difference or directly or indirectly on this difference, such as according to values derived therefrom.

  Outlier classification engine 112 analyzes and correlates normalized hybrid outliers and / or raw data points based on user-defined rules. The hybrid outlier classification engine 112 appropriately classifies according to each test, but can be performed independently of data from other tests or in conjunction with such data from other tests. The SMALL, MEDIUM, or LARGE outlier criteria can be selected according to any suitable criteria, such as test limits and / or thresholds based on data characteristics.

  In one embodiment, the outlier classification engine 112 avoids performing mixed outlier detection analysis or discards the results of such analysis if it appears that no useful information is likely to be obtained from the results. can do. For example, the outlier classification engine 112 can compare the number of test data points for a particular test with a threshold such as twenty. If there are fewer than the threshold number of test data points for analysis, the outlier classification engine 112 may not analyze the hybrid outlier. Similarly, if the number of detected hybrid outliers exceeds a threshold such as 10, the result of the hybrid outlier analysis can be discarded as too many results to be considered as a hybrid outlier. Alternatively, the outlier classification engine 112 can execute and / or hold the results, but can annotate the results to indicate the potentially suspicious nature of the data.

  The outlier classification engine 112 may also identify components that are close to clusters of bad and outliers. Such components in the same region can present problems similar to outliers and defects in that region, but the problem may not be identified during the initial analysis of the test data. Accordingly, the outlier classification engine 112 may use the bad data, global (or single parameter) outlier data, and / or hybrid (or multi-parameter) outlier data to “use “Die” may be identified.

  Outlier classification engine 112 can identify related neighborhoods and adjacent components in any suitable manner. For example, referring to FIG. 12, the outlier classification engine 112 first identifies related neighborhoods by analyzing the bad data 1210, the global outlier data 1212, and / or the hybrid outlier data 1212 to find adjacent neighbors. Or other spatially related groups of components 1214 can be identified. The neighborhood around group 1214 can be defined according to any criteria, such as the size, shape, or position of group 1214. In addition, the neighborhood can be defined according to one or more types of data, such as a defect 1210, a global outlier 1212, or a group 1214 of hybrid outliers 1212. In this embodiment, the outlier classification engine 112 identifies neighbors that contain the minimum number of test results for adjacent components 106 that have a defect 1210, a global outlier 1212, and a hybrid outlier 1212.

  Upon identifying related neighborhoods, the outlier classification engine 112 identifies spatially related test results that do not include the defect 1210, the global outlier 1212, and the hybrid outlier 1212. For example, designating test results for components 106 that are adjacent to at least one, two, or three components 106 in group 1214 or within a selected distance range away from group 1214 for further analysis. it can. The identified component 106 may be designated as, for example, a “suspicious” component 1216. The identified suspicious component 1216 and test results can be further analyzed, designated as outliers, and / or reported.

  Computer 108 suitably collects data from test system 100 at runtime or in conjunction with an inline process and provides output reports to a printer, database, operator interface, or other desired destination. Any form, such as a graphic, numerical, textual, printed, or electronic form can be used to indicate an output report for subsequent use or analysis. The output element 208 can provide selected content including selected output test data from the tester 102 and the results of supplemental data analysis.

  In this embodiment, the output element 208 suitably provides a selection of data from the output test data specified by the operator and supplemental data. Computer 108 may also be configured to include information related to outliers or other information generated or identified by supplemental data analysis elements. In such a configuration, an identifier such as an xy coordinate is also combined for each outlier. The coordinates and outliers of the component selected by the operator are merged into the output report, and in this embodiment is the output format of native tester data. Merging the resulting data into a dynamic data log facilitates compression of the original data into summary statistics and critical raw data files into a smaller native tester data file. Data storage capacity is reduced without compromising the integrity of the data for customer analysis.

  The retrieved information is then stored appropriately. Reports can be created in any suitable format or style. In this embodiment, the output report suitably includes a dynamic data log with a wafer map showing selected components on the wafer and their classification. Further, the output element 208 can superimpose wafer map data corresponding to outliers on a preselected component wafer map. In addition, the output element can include only outliers from the wafer map or batch as sampled output. The output report can also include a series of graphical representations of the data to highlight outbreak occurrences and correlations in the data.

  The output report may further include recommendations and auxiliary data for the recommendations. For example, if two tests are likely to produce the same set of failures and / or outliers, the output report should suggest that the test is redundant and that one of the tests should be omitted from the test program. Can include recommendations. The recommendation can include a graphic representation of the data indicating that the test is the same result.

  The output report may be provided in any suitable manner, such as, for example, output to a local workstation, sent to a server, alarm activation, or other suitable manner (step 712). In one embodiment, the output report can be provided offline so that the output does not affect system operation or transmission to the primary server. In this configuration, the computer 108 copies the data file, performs analysis, and generates a result for demonstration or verification, for example.

  The specific embodiments described with the drawings are merely illustrative of the invention and the best mode thereof, and are not intended to limit the scope of the invention in any way. For the sake of brevity, conventional signal processing, data transfer, and other functional aspects of the system (and components of the individual operating components of the system) may not be detailed. Further, the connecting lines shown in the various drawings are intended to illustrate exemplary functional relationships and / or physical connections between the various elements. In practical systems, there may be a number of alternative or additional functional relationships or physical connections. The invention has been described above with reference to preferred embodiments. However, changes and modifications can be made without departing from the scope of the invention. These and other changes or improvements are intended to be included within the scope of the present invention as expressed in the following claims.

Claims (21)

A semiconductor test data analysis system for identifying outliers in semiconductor test data,
A memory configured to store the test data;
A processor connected to the memory, the processor comprising:
Retrieving the test data from memory;
Comparing the test data with an upper threshold for outliers and a lower threshold for outliers, wherein the upper threshold for outliers and the lower threshold for outliers are derived from the median of the frequency distribution of the test data. And that
Determining whether the test data includes an outlier according to the comparison with the test data for an upper threshold for the outlier and a lower threshold for the outlier;
Classifying the outliers;
A semiconductor test data analysis system configured to classify components based on test failures and the classified outliers. The upper threshold for the outlier is derived from a third quartile,
The semiconductor test data analysis system according to claim 1, wherein the lower threshold of the outlier is derived from a first quartile.   The semiconductor test data analysis system according to claim 1, wherein the threshold value is asymmetric with respect to the median value.   The semiconductor test data analysis system according to claim 1, wherein the processor is configured to adjust the threshold according to a slope of the frequency distribution.   The semiconductor test data analysis system of claim 1, wherein the processor is configured to classify the outliers according to the magnitude of the outliers.   The semiconductor test data analysis system of claim 1, wherein the processor is configured to assign a category to the outlier according to a range between at least one of the thresholds and an edge of the distribution.   The semiconductor test data analysis system of claim 1, wherein the processor is configured to determine whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters. A computer-readable recording medium storing computer instructions for causing a computer processor to execute a method for identifying outliers in semiconductor test data, the method comprising:
Establishing an upper threshold for outliers and a lower threshold for outliers according to the test data, wherein the upper threshold for outliers and the lower threshold for the outliers are derived from the median of the frequency distribution of the test data. And that
Comparing the test data with an upper threshold for the outlier and a lower threshold for the outlier;
Identifying outliers in the test data according to the comparison with the test data against an upper threshold for outliers and a lower threshold for the outliers;
Classifying the outliers;
Classifying components based on test failures and the classified outliers. A computer readable recording medium . Establishing an upper threshold for the outlier includes identifying a third quartile;
9. Computer instructions for causing a computer processor to perform the method of identifying outliers as recited in claim 8, wherein establishing the upper threshold for the outlier includes identifying a first quartile. Computer-readable recording medium . The computer-readable recording medium storing computer instructions for causing a computer processor to execute the method of identifying outliers according to claim 8, wherein the threshold value is asymmetric with respect to the median value. 9. The method of identifying outliers according to claim 8, wherein establishing the upper threshold for outliers and the lower threshold for outliers includes adjusting the threshold according to a slope of the frequency distribution. A computer-readable recording medium storing computer instructions for execution. 9. A computer-readable recording medium storing computer instructions for causing a computer processor to execute the method of identifying outliers according to claim 8, further comprising classifying the outliers according to the magnitude of the outliers. . 9. A method for identifying an outlier according to claim 8, further comprising assigning a category to the outlier according to a range between at least one of the thresholds and an edge of the distribution. A computer- readable recording medium storing computer instructions. The method for identifying an outlier according to claim 8, further comprising determining whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters. A computer-readable recording medium storing computer instructions. A computer-readable recording medium storing instructions for causing a computer to execute a process, the process comprising:
Retrieving a series of semiconductor test data from memory;
Establishing an outlier upper limit threshold and an outlier lower limit threshold, wherein the outlier upper limit threshold and the outlier lower limit threshold are derived from a median of the frequency distribution of the test data; ,
Comparing the test data with an upper threshold for the outlier and a lower threshold for the outlier;
Determining whether the test data includes an outlier according to the comparison with the test data for an upper threshold for the outlier and a lower threshold for the outlier; and classifying the outlier;
Classifying components based on test failures and the classified outliers. A computer readable recording medium . Establishing an upper threshold for the outlier includes identifying a third quartile;
The computer-readable recording medium of claim 15, wherein establishing the upper threshold for outliers includes identifying a first quartile. The computer-readable recording medium according to claim 15, wherein the threshold value is asymmetric with respect to the median value. The computer-readable recording medium according to claim 15, wherein establishing the upper threshold value for the outlier and the lower threshold value for the outlier includes adjusting the threshold according to a slope of the frequency distribution. The computer-readable medium of claim 15, wherein the process further comprises classifying the outlier according to a magnitude of the outlier. The computer-readable medium of claim 15, wherein the process further comprises assigning a category to the outlier according to a range between at least one of the thresholds and an edge of the distribution. The computer-readable recording medium of claim 15, wherein the process further comprises determining whether the test data includes an outlier according to a correlation of data corresponding to at least two parameters.