Overview of steps for analysing a pattern.

  1. Load in a JPEG (JPG) image from a digital camera.
  2. Set the area of the image to be analysed.
  3. Set the automatic pellet detection level.
  4. Hand-adjust the image by adding or deleting pellets manually.
  5. Save the analysis results.
  6. To assess the average of a number of shells add several saved result files into an "Average Panel".
  7. That's it!
  8. Tips for analysing a target.


Image & Analysis Screen Layout.

IMAGE CONTROLS - used to identify the pellet holes and set the scale of the image. See "Using the Controls & Buttons".
IMAGE - the original camera image. Superimposed on this are various symbols used to represent the pellets and scale the image. See "Symbols used on the Image Screen".
HEADER PANEL - enter the data relating to the shell under test. The drop-down lists speed data entry.
ANALYSIS GRAPHIC - shows the pellet positions in relation to the standard 30" and 20" diameter circles. The horizontal and vertical distribution graphics show the relative pellet density. The +/- number on each graphic shows the spread or "1-sigma" value of an ideal distribution based on the pellets measured. This is the distance that on average would account for 68% of the pellets. The histograms show the actual pellet distribution versus an ideal distribution. 
ANALYSIS PANEL - shows the numerical analysis of the distribution including the probability of an edge-on or face-on clay being hit. See "Analysis Panel - Interpreting the results!"

Using the Controls & Buttons.

Decrease / increase the size of the image. Does not affect pellet detection.
Block a small, medium or large area of the image from automatic pellet detection. Manually added pellets are unaffected. (see on-screen Blocked Area)

Set active area button. Drag the boxes at the top-left and bottom-right to set the area that will be scanned for pellet holes.
Adjust the slider to set the automatic pellet detection level and give the best balance between not detecting true pellet marks and false pellet detection when the contrast between the pellet marks and background is poor. False detection can be blocked manually. Undetected pellets can be added manually

Set the scale of the image. Drag the up/down arrows on the image to indicate a known vertical distance on the image and the left/right arrows to indicate a known horizontal distance. Move the graticule to indicate the point of aim.
"double" is used to detect near-coincident pellet strikes. (NB this feature works best with larger pellets that give a well defined and consistent impact mark.
"size" ignores features much larger than the typical pellet size. If a conventional pattern plate target with a bulls-eye and pre-marked segments is scanned-in, the "size" setting can suppress these features of the target.

Manually add or delete single pellets. Clicking will add a 'manual' pellet - shown as a blue marker. SHIFT+clicking over a manually generated pellet mark will delete it. SHIFT+click over an automatically generated pellet mark will exclude the pellet from the analysis. (see Manual Pellet and Deleted Pellet graphics)


Horizontal / Vertical fields to enter the true distance indicated by the scale setting arrows.
Update the analysis panel with the latest status of the image.
The distances may be input in imperial inches or metric cm. However, all outputs are displayed in inches and graphical output is shown with respect to a 30" and 20" circle.

Input Menus.

Open a camera image of a shot pattern. Note that the image must be in JPEG format and at least 200 x 200 pixels in size. The higher the contrast of the pellet marks against the background, the better the automatic pellet recognition will work. Remember the old computer adage of "Rubbish in, rubbish out!"

Open a previously saved analysis file. Note that only the Analysis Panel and Graphic will be reloaded to allow viewing of the Analysis Graphic or editing of the Header Panel.

Open a new window to allow multiple saved Analysis Files to be averaged. (see Averaging Data.)

Open a previously saved 'Average File'. (see Averaging Data.)

Closes all open windows and then quits the programme. Note that any windows that have been 'minimised' will remain open and the programme will not quit. These windows must be expanded and "Quit and close all" called again or each window should be closed individually.



Output Menus.

Print the analysis of the data. The print-out automatically resizes to fit a single page.

Export as comma separated variables (CSV). Saves a simple text file to disk with all fields separated by commas. The "Export as CSV" of an Analysis File includes the results of the Analysis Panel and also the original pellet x-y coordinates in cm.

Save the analysis of the data with the option of renaming the file. Please note that only the Header Panel, Analysis Graphic and Analysis Panel are saved. The original image file from which the data are drawn is not saved.

Save an analysis file using its current name without a prompt to rename the file.

Symbols Used on the Image Screen.

Automatically detected pellet.

Manually added pellet.

Automatically detected pellet that has been deleted manually.
Define the top and bottom of a known vertical distance. Click-and-hold inside an arrow with the mouse and drag the arrow around the image.

(Note: The vertical arrows allow a small misalignment to correct for any rotational errors incurred when taking the picture.)

Define the left and right points of a known horizontal distance. Click-and-hold inside an arrow with the mouse and drag the arrow around the image.

(Note: The horizontal arrows allow a small misalignment to correct for any rotational errors incurred when taking the picture.)

Indicate the point of aim. Click-and-hold inside the graticule with the mouse and drag the centre of the graticule to the original point of aim.
Area that blocks automatic pellet detection.

Analysis Panel - Interpreting the results!

Pattern statistics.

POA x/y.
spread x/y.
75% shot diameter.
corr / sig.

Pellet Distribution:

10" Dia., 10-20", 20-30", Total.
Oberfell field.

Probability of hit.

10" Dia., 20" Dia., 30" Dia.
Centre weighted.

Pattern statistics.

"POA x/y" are the 'Point of Aim' of the mean of the pellets compared with the point of aim indicator as positioned on the original image. x= horizontal placement, y= vertical.

"spread x/y" are the distances from the centre of the spread that would account for 68% of the shot (the "1-sigma" value) if the distribution was a perfect normal. 

"75% shot diameter" should be read as the diameter inside which on average approximately 75% of the shot would fall. It is a convenient way of expressing the width and height in a single number that has some practical significance. (For the statistically inclined it assumes that the x and y spreads are uncorrelated. In practice this is not unreasonable since if the correlation is pronounced that is the bigger problem, not the accuracy of this radius estimate!)

"corr / sig" are the Correlation and Significance respectively. The correlation (rs) is based on the Spearman Rank Correlation algorithm. If these terms are not familiar it is suggested that they are either ignored (recommended!) or the guidance of a statistics book is sought. However, in simple terms, the correlation relates to how closely tied the 'y' vertical position is to the 'x' horizontal. For example, the diagram below shows a correlation close to 1. That is, there is a high degree of correlation between the x and y position of the pellets. A correlation of -1 would be a an equally pronounced correlation but sloping downwards. Ideally, the shotgun pellets will have a correlation of zero. The correlation analysis is included for completeness. Referring again to the picture below, the distributions in the vertical and horizontal would be shown by the simple graphics in the Analysis Panel/Graphic as being quite good: similar vertical and horizontal; no gaps in the centre; symmetric about the centre. However, clearly something is wrong! The correlation figure would show this. In practice any correlation will be less obvious to the eye, but the figures should reveal if there is any systematic error. 

The statistical significance (i.e. the confidence one has that the result is real and not the result of chance) of the correlation is given by: t = rs * sqrt( (N-2) / (1 - rs * rs) ). For a large number of pellets ( > ~100) a significance above approximately 4.0 indicates a high level of confidence that the correlation rs is significant and not just a random event.

You are very, very unlikely to ever need this.

Pellet Distribution.


10" Dia., 10-20", 20-30". This is simply the number of pellets falling within the regions indicated. Please note that 10-20" and 20-30" are annuli! The number of pellets inside the 30" circle is the 10" Dia. value plus the 10-20" value plus the 20-30" value. If the number of pellets in the shell has been entered the percentages are also shown.

The 'total' is all pellets detected. This may include outliers that are not shown on the analysis panel graphic.

'Oberfell'. This is the number of non-overlapping 5" diameter discs that can be positioned in the 30" circle without falling on a pellet. Please note that the algorithm used under-reads if there are very few pellets within the circle. According to Oberfell p47, an empty 30" circle can accommodate 27 non-overlapping 5" discs. The algorithm as implemented records approximately 22. It was beyond this author to come-up with a fast algorithm for automatically replicating this result. However, for more realistic results the algorithm is reliable. This can be shown by manipulating pellets manually. It is also worth bearing in mind that the method of drawing in circles by hand is also non-perfect!


Probability of hit:

An edge-on clay is represented by a disc with the same area as an edge on standard sized UIT clay as used in the UK (25mm high x 110mm diameter - Laporte. 26mm high x 110mm diameter - CCI ). A "full-on" clay is represented by a disc of diameter 110mm (Laport, CCI data). By passing these discs across the pattern of pellets, the number of times the disc is 'hit' by at least one pellet versus the number of times it is completely free of pellets gives an estimate of the likelihood of the clay being hit.

The first three columns titled 10" Dia, 20" Dia and 30" Dia are simply estimates of the likelihood that the full-on or edge-on clay will be broken averaged over the given circle. This likelihood is estimated by passing the 'clay' across the target area and noting how many times the clay covers a pellet and how many times it does not.

Although this seems seductively simple, rating all areas of the 20" or 30" area equally may not be the best way to rate the pattern. What happens if there is a 'blown' pattern? That is one where a clay (or bird) could pass through the middle of the pattern without being hit even though there are plenty of pellets in the circle as a whole. Intuitively one can appreciate that the centre of the pattern is far more important than the fringe. If the shooter has done his part and centred the pattern on the target he should expect a kill. This is critical to trap and skeet shooters where top scores can approach 100%. The question is how to give the centre of the pattern greater importance when rating the pattern.

The columns titled 'Centre weighted sum' and 'Centre weighted norm' offer two alternatives.

The pattern is divided into the centre 10" diameter circle, an annulus 10"-20" diameter and an outer annulus 20"-30" diameter. Note, the relative areas of these regions are 1 : 3 : 5.

The 'Centre weighted sum' is simply the average of the hit probabilities of each of the three regions. Note that by using this regime, the centre 10" diameter disc is weighted at 1/3rd of the total compared with 1/9th using uniform weighting.

'Centre weighted norm' weights the centre of the pattern even more heavily. The weighting is based on a Normal distribution. The rationale for this is based on the premise that the shooter will centre his shots about the target in much the same way as a rifle or pistol shooter. Over a large number of shots the variation about the target will probably follow a Normal or Gaussian distribution. (See 'Background' for a brief explanation of the Normal distribution.) Assume the shooter's variation is contained within '3-sigma' of variation (~99.7%) and that this equals the 30" diameter circle (sadly, better than me!). Using this regime the centre 10" diameter disc is weighted 0.68, the middle annulus 0.27 and the outer 0.05. This weights the hit probability with the same weighting as the probability of the shooter's ability to place the shot, thus giving a truer estimate of the practical likelihood of a hit. Clearly this is open to debate. One could argue about how accurate the shooter really is - should we assume instead that the 30" diameter circle is the '2-sigma' level, that is ~5% of the time the shooter misses the 30" circle? It is probably best to simply say there are a number of ways to justify the relative weighting! However, as presented, the simple hit probability over a 30" circle would be applicable to a beginner or field shooter, the 'centre weighted sum' method would be for an intermediate shooter and the 'normal weighted distribution' for an expert.

Average Panel.

Menu Functions of the Average Panel.

Interpreting the Average Panel Results.

The HEADER PANEL is the same as in the Analysis Panel.
If no data is entered into the panel, it will take the data of the first Analysis File or Average File added.
As Analysis Files are added to the average panel, their name and some key features from their header panels are shown. This allows a quick visual check that all data to be averaged came from the same gun or cartridge.
The AVERAGE RESULTS are similar to the Analysis Panel except that now the shot to shot variation for key parameters is also calculated.
  1. 'POA' is the 'point of aim' as entered by the user on the target compared to the mean of the pellet distribution.
  2. '+/-' is the shot to shot variation of the POA. The figure generated gives the bound about the mean that accounts for approximately 68% of the targets.
  3. 'Spread x / y' are the average width (x) of the pattern and average height (y) of the pattern that account for 68% of the pellets.
  4. 'shot to shot variation +/-' gives the variation of the width and height about their averages. Again, the figure generated gives the bound that accounts for approximately 68% of the cases, i.e. the "1-sigma" value.
  5. '75% diameter' should be read as the diameter inside which on average approximately 75% of the shot would fall. It is a convenient way of expressing the width and height in a single number that has some practical significance. (For the statistically inclined it assumes that the x and y positions are uncorrelated. In practice this is not unreasonable since if the correlation is pronounced that is the bigger problem, not the accuracy of this quasi radius!)
  6. This is the shot to shot variation of the '75% diameter' figure 'E'.  
  7. This is simply the shot to shot variation of the combined spread 'F' expressed as a percentage of the average '75% diameter' figure given in 'E'.
  8. The 95% confidence reflects the accuracy with which the estimated spread or shot to shot variation are estimated. This figure is the range within which we are 95% confident the estimate of the shot to shot variation and average spread genuinely lie. That is, if the experiment was repeated, 95% of the re-runs would fall within this range. The importance of this figure is that if another group of shells is tested and their average and/or shot to variation fall beyond this range, we can say we are 95% confident this second group is genuinely different to the first and we have found a real difference in performance between the two groups of shells. Otherwise we conclude the two groups are not significantly different. The 95% bound is reduced by averaging more shells. It is calculated as twice the shot to shot variation divided by the square root of the number of shells. (Check using the above figure: 2 x 1.53 / 3.162 = 0.97. Correct!) 
  9. This is simply 'H' above expressed as a percentage of the shot to shot variation. Using the figure above, if a subsequent set of shells had a shot to shot variation of less than 3% or greater than 13% we would say the second set of shells was significantly more / less repeatable than the example above. If, however, a second group has a shot to shot variation within this range we could not say with any great confidence that one shell was better or worse than the other.

Menu Functions of the Average Panel.

Add an earlier Analysis File to this Average Panel.

Add an earlier Average File to this Average Panel.

Delete the file(s) highlighted in the list of the Average Panel.

Interpreting the Average Panel Results.

For a pistol or rifle shooter it is simple to establish what constitutes a 'good' cartridge and gun combination. It is simply the combination that gives the tightest group. Those who have experimented with different cartridges and in particular home loading will know what a wide range of results can be had from a single gun. The key concept to grasp is that what is sought is repeatability. With rifles and pistols one may get a 'tight' group with the occasional wild flier, or a consistent middling sized group, or best of all the consistently tight group. With a shotgun, all of these phenomena are probably happening, but difficult to 'see' given the hundreds of pellets.

One clear lesson that can be drawn from the rifle and pistol world is that one shot tells nothing! A number of shots must be made to show the average.

Given a number of test patterns, what constitutes a good shotgun and cartridge performance is still not clear. Using the rifle and pistol example, one should be trying to achieve consistent results. Consistent bias can be adjusted out. If the spread is a little wide, the choke can be adjusted in the same way as the rifle shooter can adjust the sights to correct for an off-centre group. However, if chokes are fixed, knowing the relative spread of various shells is in itself useful information. Consistency is measured by the variation of the pattern widths. For example, if a shell has a pattern with a reported spread of 8" (the 'E' field in the above figure), and every shell measured achieves this, it would be considered very repeatable (and almost impossible!), i.e. the spread on the 8" would be zero (reported in 'F' and 'G' fields).

The averages can be a little misleading. Think back again to the rifle and pistol scenario where the group was generally good, but there was one flier where clearly something 'one-off' had happened, perhaps a short charge of powder or bad crimp. The shotgun equivalent would be the 'blown' pattern. There is no easy way to analyse this by looking at averages with a shotgun. One must look at the probabilities of kills versus position in the pattern. If a miss is found in the centre of the pattern that is normally quite dense, then this would indicate a blown pattern. The analysis software allows the user to look at how reliably 'edge-on' or 'face-on' clays are hit. (A 'face-on' clay is also close to the 5" disc typically used to describe game.)

In summary, the analysis software allows the user to look at each pattern in detail to confirm an even spread (i.e. no 'fliers' or blown patterns) and then look at the average results to compare the consistency and average spread of different shells. A measure of consistency might be the "+/-" value of the spread horizontally and vertically and if the gun is also shot from a rest, the variation in the point of aim relative to the mean of the pattern.

Tips for Analysing a Target.

Dealing with 'fliers' or outlier pellets.

How to deal with 'fliers' or outlier pellets is a difficult topic to resolve. In earlier versions of the software the advice given here was to ignore obvious 'fliers'. However, I'm now inclined to include all pellet marks. The reasons for this are:

  1. If the number of outliers is low then their effect on the overall results will be small. The random nature of the Normal (Gaussian) distribution allows for a small number of outliers - in this case they are in fact a genuine contributor to the statistics of the pattern.
  2. If the number of outliers is large (say 5-10 pellets or upwards) this is indicative of something real happening with the shell and it is wholly correct that these outliers start to influence the measurement results - both in terms of pattern width and repeatability.
  3. If in the future this advice proves to be incorrect, it is easy for me to rewrite the software to automatically remove the outliers! It is easier for me to do this than for the users to re-scan all the targets.

Recording information on the target.

Make sure that all pertinent information is marked on the target and captured by the digital camera. Notes can always be blocked from the analysis area. Make sure the reference distance is marked on the target for setting the image scale.



Brief overview of the Normal Distribution.

The Normal distribution features strongly in this analysis of shotgun patterns. The picture below illustrates its main features:

The graph shows the relative probability of an 'event' happening versus its distance from the mean (average) which in the picture above is shown as zero. It will be seen that the likelihood of an event occurring is greatest nearest the mean. The distance +/- "1-sigma" accounts for approximately 68% of events, the distance +/- 2-sigma for 95% of events and the +/- 3-sigma range accounts for 99.7% of all events. This type of distribution model fits many naturally occurring observations, for example, heights of people, test scores, examination marks etc. Oberfell and Thompson also found that the average pattern distribution from a shotgun could be modelled by the Normal distribution. It is important to note that the Normal distribution does not define the pellet distribution, it is just a model that fits the observed distributions quite well. The closeness of the distributions (shown by the histograms in the Average Results Panel and Analysis Graphic) and the ideal Normal distribution should be convincing enough that the Normal distribution is a useful tool for characterising the distributions. (It is a guess on my part that rifle, pistol and shotgun users will tend to aim about the target with a variance that is also approximated by a Normal distribution.)

The value 'sigma' is used to indicate the width of the distribution. The value of sigma can be used to compare one shell against another for a given choke to see which holds the tightest pattern and hence which will be able to carry a useful pattern the longest distance. Alternatively, skeet shooters may be interested in obtaining the widest spread possible.

To demonstrate that the shell and gun combination work well together, the patterns should be as repeatable as possible in much the same way as a rifle shooter looks for minimum variation shot to shot. The big difference is that rather than measuring a single distance (the maximum spread), the shotgunner's measure of repeatability is how all the pellets behave together. One way of doing this is to fit all the pellet positions to a model and then look at how the parameters of the model vary. In this case the Normal distribution is employed as the model.

A measure of repeatability of the pattern is given by how sigma varies from shot to shot. Perhaps surprisingly to the uninitiated, the distribution of the value of the sigmas is itself a Normal! Thus we can have a 'sigma' value for the variation of the sigma of the spread of the pellets distributions as measured shot to shot!


The Mysteries of Shotgun Patterns.

The main source of reference (and part inspiration) for this work is a wonderful little book by the title of, "The Mysteries of Shotgun Patterns" written by George G. Oberfell and Charles E. Thompson, published by Oklahoma State University Press, 1957.

As well as many, many test targets, this book in turn cites work done by others in their efforts to deduce how shotguns perform.

One over-arching finding of Oberfell and Thompson (henceforth O&T) is that on average, over a large-number of shots, the pellet distribution follows that of a 'Normal' (also known as Gaussian) distribution.

The method of rating a shotgun developed by O&T is based on counting the gaps in a pattern. To ensure there is a reasonable probability of a gap, the test distance is 40 yards and No.6 shot recommended. As well as ensuring a reasonable likelihood of having gaps to measure, the number of pellets is low enough to be counted by hand. Based on the performance at 40 yards, performance at closer distances can be deduced.

With the availability of fast, low-cost computers the analysis can be more sophisticated. Performance at closer ranges can be confirmed and repeatability shot-to-shot can be investigated. However, generating more data then presents the problem of how to interpret the results. One of the great strengths of the Oberfell & Thompson method is that it rates the shotgun in simple terms - normal, poor, excellent etc.

Numerical Recipes in C.

The "Numerical Recipes" series is a great source for algorithms across a whole range of science, engineering and maths. The Spearman Rank-Order Correlation algorithm came from Numerical Recipes in C. Authors: William H. Press, Brian P. Flannery, Saul A. Teukolsky, William T. Vetterling. Cambridge University Press, ISBN 0-521-35465-X.