In a recent publication, Peter Bönisch and Roman Inderst tackle the delicate issue of the evaluation of seemingly contradictory econometric evidence. Introducing the concept of severity measures, they propose a method to avoid the common obstacles plaguing the interpretation of seemingly conflicting empirical evidence through the practical example of financial damage estimation in follow-on cases. The topic discussed by Bönisch and Inderst is particularly important considering the practical aspects of litigation cases, as judges typically do not have an in-depth knowledge of econometrics. This may result sometimes in the unwarranted situation that they blindly disregard contradictory evidence by party experts as assumably non-liquet,¹See practical examples of national judgments on competition private enforcement cases in which the judges rejected party-expert econometric evidence on inconsistency grounds: Bundesgerichtshof, 9 October 2018, KRB 10/17; Tribunal Administratif de Paris, 9 Janvier 2009, Requête N°9800111. and instead opt for other solutions such as their own independent estimation of damages.²See CJEU, Request for a preliminary ruling from the Juzgado de lo Mercantil n.o 3 de Valencia (Spain) lodged on 19 May 2021 — Tráficos Manuel Ferrer, S.L. and Other v Daimler AG, C-312/21, where the judge asked whether the power of the National Court to estimate the amount of damages enables those damages to be quantified independently on the grounds of a situation of information asymmetry or insoluble difficulties regarding quantification. Opinion of Advocate General Kokott on this question is set to be delivered on 22 September 2022.

How can expert estimation results be contradictory?

According to the authors, divergent empirical findings in estimating financial damages can typically be explained by multiple reasons. In the context of private damage litigation for competition law infringement, the main source of uncertainty often comes from the asymmetric information availability between the parties, which leads to divergent statistical data sets. Indeed, in such cases, the parties simply do not have the same level of access to the different sets of data available, under which circumstances it is difficult to reach true equality of arms between the claimant and the defendant. Thus, a strong asymmetry of information remains in favour of the defendant. Furthermore, conflicting estimated results can be explained by the differences in the underlying econometric model of each party, which is commonly based on conflicting assumptions, such as, for example, the structure of the model and the set of variables used in the estimation.

These differences lead, expectedly, to contradictory results. Because of these discrepancies, the opposing statistical estimates put forward by the two parties can be interpreted to be incompatible while, in essence, both pieces of evidence describe the same empirical phenomenon and reach opposite conclusions because of the uncertainty risks described above. Hence, Bönisch and Inderst screen the compatibility level of seemingly contradictory empirical results through the concept of the statistical severity of the furnished evidence.

Estimating infringement effects in private enforcement: the common procedure

In their publication, Bönisch and Inderst take a numerical example of a follow-on private action for damages. In this context, both parties submit expert testimony based on econometric estimations. In their hypothetical example, both the claimants and the defendants perform market assessments to quantify the damage per item resulting from the infringement (hereinafter, “the infringement effect”), yet the experts reach opposite conclusions: statistically insignificant damages of 1€ for the defendant versus statistically significant damages of 8€ for the claimant.

The authors point out that practitioners must be careful with the interpretation of statistical significance. In econometrics, a result is statistically significant if, with a “high enough” probability, the measured effect is real. In other words, it is highly probable that the impact does not occur only by chance (aka Type I error). There is no rule of thumb as to what probability is “high enough”, but empirical best practices use a cut-off point of 90% or 95%. Note that the usual statistical significance test is assessing the reliability of the results in comparison to a zero-effect (“null-hypothesis”), so we can tell by looking at the probability values whether the result is different from zero. However, could there be any other cutoff points? The article argues that exactly the testing vis-à-vis other cutoff points is the key to smoothing the differences between diametrically opposing results as in the example above.

Measuring severity levels of possible infringement effects

The statistical notion of “severity” was first introduced by Mayo and Spanos in 2006. The general idea behind a severity assessment is to test the statistical evidence at hand against an alternative hypothesis beyond the “general” null-hypothesis. Let’s take the authors’ example in which the plaintiff’s statistical evidence proved an 8€ damage estimation being significantly different from zero. Using the similar standard statistical calculations as before, the goal is now to measure the level of certainty of this evidence for values of other possible infringement effects x different than zero, such as 2€ or 6€. So, the general question on null-hypothesis testing whether the measured impact is significantly different from zero is replaced with the question whether the measured impact is significantly different from 2€ or 6€. Naturally, the further away we define (positive) x from zero, the lower the probability that the estimated damage of 8€ is different from x.

A similar severity testing can also be performed on the defendant’s statistical evidence that led to a statistically insignificant 1€ damage estimation. Here, the further away we define (positive) x from zero, the higher the probability that the estimated damage of 1€ is different from x. The two opposing results with severity testing at different values of the alternative hypothesis is depicted in the figure below.