Statistical Significance Tester 

Consider the predicament of a rifle shooter comparing two different shells: Shell A gives a group of 2.5", Shell B a group of 3". Shell A has done better in this test, but does that mean it will consistently give a tighter group? Applying statistical tests to the data can help answer this question. 

The Statistical Significance Tester gives an estimate of whether the difference between two measured averages is genuinely different, that is the difference is likely to be repeated in subsequent real physical tests. The higher the reported confidence level on a 0-100% scale the better. In general (and unsurprisingly), the larger the difference between two sets of data and the more samples that contributed to the data, the higher the confidence level in the data. 

Quick guide.

  

  

(Allow time to download - approx 20seconds with a 56K dial-up)

If the calculator does not appear you need to configure your browser and computer.

  

(c) Written by A C Jones http://www.shotgun-insight.com
    

 

Quick Guide.

Probably the best way to get a feel for the Tester is to plug in some data and see the results. The table below summarises what data from Shotgun-Insight and Rifle & Pistol can be used in which field of the Tester.

 

Results from Rifle & Pistol Insight and Shotgun-Insight that can be used in the Statistical Significance Tester.
 
"Average values"
Corresponding "Spread +/-"
Rifle & Pistol Insight
 Average point of impact - horiz Average spread - horiz
Average point of impact - vert Average spread - vert
Shotgun Insight (from an average panel)
 POA-x Corresponding "+/-" figure
POA-y "
Spread-x Shot to shot variation +/-
Spread-y "
75% Diameter "
Pellets 10" Dia, 10-20", 20-30" Corresponding "+/-" figure
All probabilities of hits (%) Corresponding "+/-" figure

Overview of what the "Confidence" reading means:

  • A reading of near 100% means that the two groups of data are genuinely different. If the physical test were to be repeated the same conclusion would be obtained.

  • A 95% confidence reading means there is a 95% chance (or 19 in 20 chance) that if the physical test were to be repeated the same conclusion would be obtained. There is a 5% chance (or 1 in 20) that the estimated difference between the two groups is a "one-off" and if more data was gathered it would cancel rather than reinforce the original finding. 

  • 95% confidence can be interpreted as "very confident". It is a widely used benchmark.

  • 90% confidence is getting borderline. Certainly much below this and one should consider testing more samples to confirm or deny the trend.

 

The phrase "conclusion" needs to be explained. It just means that the value of Set A is bigger (or smaller) than that of Set B. A high confidence value means that if the data was gathered again Set A would again be bigger (or smaller) than Set B. The magnitude of the difference might change in a subsequent test, but Set A should still have a higher value.