Test Strategies¶

Test strategy defines the method of hypothesis testing. After collecting data, the Researcher runs a statistical test on the Experiment, and calculates appropriate statistics, and p-values for each outcome variables. Thereafter, based on the calculated p-value, and given ɑ, Test Strategy decides whether the test result is significant.

Similar to Data Strategy, we configure the Test Strategy by specifying the method name, follwing with its setting their specific parameters. Most tests should be accompanied by an ɑ level. This ɑ is going to be used throughout the rest of SAM to evaluate the significance level of an outcome. For instance, when a policy queies for significance of an outcome, SAM asks Test Strategy to see whether for a given outcome, the test is significant.

Configuration: Test Strategy

{
    "experiment_parameters": {
        ...,
        "data_strategy": {...},
        "test_strategy": {
            "name": "MethodName",
            "alpha": 0.5
        }
    }
}

While SAM offers a few test strategies out of the box, it is possible to extend this list by implementing your own methods using the provided API.

T-Test¶

TODO: Add descsription and reference

In order to configure a t-test, we need to set some general parameters, as follow:

T-Test Configurations

"test_strategy": {
    "name": "TTest",
    "alpha": 0.05,
    "alternative": "TwoSided",
    "var_equal": true
}

F-Test¶

TODO: Add descsription and reference

F-Test Configurations

"test_strategy": {
    "name": "TTest",
    "alpha": 0.05
}

Yuen T-Test¹¶

TODO: Add descsription and reference

Yuen Test Configurations

"test_strategy": {
    "name": "YuenTest",
    "alpha": 0.05,
    "alternative": "TwoSided",
    "trim": 0.2,
    "paired": true
}

Wilcoxon Test²¶

TODO: Add descsription and reference

Wilcoxon Test Configurations

"test_strategy": {
    "name": "WilcoxonTest",
    "alpha": 0.05,
    "use_continuity": true
}

KAREN K. YUEN. The two-sample trimmed t for unequal population variances. Biometrika, 61 $1$ :165–170, 04 1974. URL: https://doi.org/10.1093/biomet/61.1.165, doi:10.1093/biomet/61.1.165. ↩
Frank Wilcoxon. Individual Comparisons by Ranking Methods, pages 196–202. Springer New York, New York, NY, 1992. URL: https://doi.org/10.1007/978-1-4612-4380-9_16, doi:10.1007/978-1-4612-4380-9_16. ↩

Last update: 2021-09-18