Logistic Regression and Normal Distribution

Full Text

Introduction

Logistic regression (LogReg) is one of the methods for establishing statistical relationships between quantitative and dichotomous (taking two values Y = 0 and Y = 1) variables: the response variable Y is dichotomous, and the predictor X is a quantitative variable. The task is to relate the probability of the value Y = 1 taking place to the values of X [1].

Specifically, in a logistic regression model with one predictor X, the statistical relationship between Y and X is assumed to be [2-4]:

 

$\mathrm{W}(\mathrm{Y}=1|X=x)=\frac{exp(b_0+b_1·x)}{1+exp(b_0+b_1·x)}$

(1)

where $W(Y=1|X=x)$ is the probability of the value Y = 1 for a given x [2].

Using relationship (1), we have

$ln\left(\frac{w}{1-W}\right)=b_0+b_1·x$

(2)

Thus, we have a linear relationship between the predictor X and the complex $\ln \left(\frac{W}{1-W}\right)$, which is called logit (W) of the probability of occurrence of Y = 1 with Х.

Relationship (2) is the condition of logistic regression applicability: only if condition (2) is met do we have the right to apply logistic regression to analyze specific factual (epidemiological) material. This condition can be realized in a variety of cases, so it is essential to check that it is satisfied in each specific case before applying LogReg. There are, however, cases where condition (2) is automatically satisfied. This article examines some of these cases related to the normal distribution of the predictor X.

Logistic regression is used in scientific research as a method for studying dependencies and as a classification method [2]:

• as a method for studying dependencies, the LogReg model is characterized by the coefficients of regression b₀ and b₁ in formulas (1)-(2), and by the odds ratio OR = exp(b₁) with confidence intervals;
• as a method of classification, the LogReg model is characterized by a classification table, sensitivity (Se) and specificity (Sp) plots as functions of the cut-off point Wcut-off-point, and by an ROC-curve and area under it (AUC).

Table 1 presents an example of classification table.

In Table 1, $n_0=n_{00}+n_{01}, n_1=n_{10}+n_{11}, N=n_{00}+n_{01}+n_{10}+n_{11}$. We calculate (the proportion of correctly classified objects for Y_obs = 1),  (the proportion of correctly classified objects for Y_obs = 0), and the probability of misclassification [2]):

$\mathrm{PMC}=\frac{n_{01}+n_{10}}{N}$

(3)

1.1 Situation 1: Two normal distributions with equal variances

One of the sufficient, but not necessarily essential conditions for the fulfillment of relationship (2) is the following: the values of the quantitative predictor X in the two groups Y = 0 and Y = 1 are normally distributed with mathematical expectations µ₀ and µ₁ and with equal variances σ²: N(µ₀, σ²) in the group Y = 0 and N(µ₁, σ²) in the group Y = 1 [2-3; 5]. According to [2-3, 5], in the case of two normal distributions with equal variances, relationship (2) is satisfied, and for the coefficients of relationship (2) we have the formulas ($\pi =\frac{n_1}{N}$ is the proportion of values Y = 1 in the sample)

$b_0=-\ln \left(\frac{1-\pi }{\pi }\right)-\frac{1}{2·{\sigma }^2}·({{\mu }^2}_1-{{\mu }^2}_0)$	(4)
$b_1=\frac{1}{{\sigma }^2}·({\mu }_1-{\mu }_0)$	(5)

 

Hence it follows that in the case of two normal distributions for X, the odds ratio OR = exp(b₁), defined by the coefficient b₁, is determined by the difference between the mathematical expectations normalized for variance, and does not depend on the proportion π of Y = 1 units in the sample; the proportion π affects only the coefficient b₀.

For formulas (4)-(5), there is a relationship between b₀ and b₁:

$b_0=-\ln \left(\frac{1-\pi }{\pi }\right)-\frac{1}{2}·b_1·({\mu }_0+{\mu }_1)$

(6)

 

For two normal distributions with equal variances, there is a formula for calculating PMC (Posterior Mean Criterion) (3) based on the coefficients (4) and (5) [2]

 

$PMC=\frac{n_{01}+n_{10}}{N}=\pi ·\Phi ·[\frac{1}{b_1}·\ln (\frac{1-\pi }{\pi })-\frac{b_1}{2}]+(1-\pi )·\Phi ·[\frac{1}{b_1}·\ln (\frac{\pi }{1-\pi })-\frac{b_1}{2}]$

(7)

 

where Ф is the cumulative normal distribution function N(0, 1).

1.2 Situation 2: Two normal distributions with different variances

If we apply expression [3] to perform calculations for distributions with different variances ${{\sigma }^2}_0$ and ${{\sigma }^2}_1$ in the groups Y = 0 and Y = 1 similar to the calculations that led to relationships (3)-(4), we obtain expressions for probability W(Y=1|X=x) and for logit (W), containing a linear and X-quadratic term:

 

$W(Y=1|X=x)=\frac{exp(b_0+b_1·x+b_2·x^2)}{1+exp(b_0+b_1·x+b_2·x^2)}$	(8)
$\log it\left(W\right)=b_0+b_1·x+b_2·x^2$	(9)

 

The coefficients of relationships (8) and (9) are expressed through mathematical expectations and variances as follows:

 

$b_0=-\ln \left(\frac{1-\pi }{\pi }\right)-\ln \left(\frac{{\sigma }_1}{{\sigma }_0}\right)-0.5·(\frac{{{\mu }^2}_1}{{{\sigma }^2}_1}-\frac{{{\mu }^2}_0}{{{\sigma }^2}_0})$	(10)
$b_1=(\frac{{\mu }_1}{{{\sigma }^2}_1}-\frac{{\mu }_0}{{{\sigma }^2}_0})$	(11)
$b_2=0.5·(\frac{1}{{{\sigma }^2}_0}-\frac{1}{{{\sigma }^2}_1})$	(12)

 

The formula for the coefficient b₀ in (10) is similar to formula (3) taking into account possible differences in variances. It follows from formula (11) that the coefficient b₁, which describes the linear part of the relationship between the logit and X, is nonzero in the case of different variances even for equal mathematical expectations µ₀ and µ₁. The coefficient b₂ in formula (12), which describes the quadratic part of the relationship, is determined by the difference between the variances and is nonzero only in the case of unequal variances: the greater the difference between the variances, the greater the quadratic term in (8) and (9). The quadratic term is positive when  is less than  and negative otherwise.

 

1.3 Situation 3: Two lognormal distributions

Let us assume that we have a quantitative predictor Z, which in the groups Y = 0 and Y = 1 is distributed according to the lognormal law. Then the indicator X = ln(Z) in Y = 0 and Y = 1 will be normally distributed. If in the groups Y = 0 and Y = 1 the distributions of X have equal variances, we deal with Situation 1 described above. In this case, the logit is given by

 

$\log it\left(W\right)=b_0+b_1·X=b_0+b_1·\ln \left(Z\right)$

(13)

 

Thus, the statistical relationship of logit(W) with the predictor Z will be logarithmic rather than linear in Z.

The expression for the probability is:

 

$W(Y=1|Z=z)=\frac{exp(b_0+b_1·\ln (z\left)\right)}{1+exp(b_0+b_1·\ln (z\left)\right)}$

(14)

 

Let us transform the expression for probability (14). Let us denote the complex exp(b₀) = a₀. Then for the predictor Z, distributed lognormally, the probability W(Y = 1|Z = z) will have a form that is not at all similar to expression (1) as is usual for logistic regression:

 

$W=\frac{a_0·Z^{b1}}{(1+a_0·Z^{b1})}$

(15)

 

If we apply, without checking the conditions for logistic regression applicability, formulas (1)-(2) to the predictor Z itself distributed lognormally, we obtain the logit in an incorrect form (2) instead of the correct expression (13).

If in the groups Y = 0 and Y = 1 the distributions of X have unequal variances, we deal with Situation 2 described above:

 

$\log it\left(W\right)=b_0+b_1·X+b_2·X^2=b_0+b_1·\ln \left(Z\right)+b_2·{\ln }^2\left(Z\right)$

(16)

2 Examples of the Situations Considered

The details of the relationship between logistic regression and normal distributions described above will be explained using examples.

2.1 Example 1: Samples from two populations with normal distributions and equal variances (Situation 1)

To perform specific calculations in Situation 1 (two normal distributions for predictor X with equal variances in the groups Y = 0 and Y = 1), we generated 200 numbers of X from the sample N(µ₀, σ²) with µ₀ = 0 and σ² = 1, and 200 numbers of X from the sample N(µ₁, σ²) with µ₁ = 1 and σ² = 1. The values of X were assigned Y = 0 for the first sample and Y = 1 for the second. Thus, we deal with an imitation of Situation 1 described above.

For the mathematical expectations and the variances chosen for them μ₀ = 0, μ₁ = 1 and σ² = 1 by formulas (3)-(4) applicable to the populations N(µ₀, σ²) and N(µ₁, σ²), we obtain

 

$b_0=-0.5; b_1=+1; b_0=-\frac{1}{2}·b_1; OR=exp\left(1\right)=2.7183$

(17)

 

Calculations carried out with the samples from the populations, in which the mean values of X are <x₀> = 0 and <x₁> = 1 and the sample variance is σ² = 1, provide

 

$b_0=-0.50218; b_1=+1.00436; b_0=-\frac{1}{2}·b_1; OR=2.7301 (CI=2.1393-3.4842)$

(18)

 

The results for (17) and (18) are very close, which indicates a good approximation of our sample distributions to normal distributions in the populations. However, in relationship (18), the numerical values of the parameters are somewhat different from (17), although the mean values and variances in the samples are equal to the specified parameters of the populations with very high accuracy. Apparently, the matter is not only in the values of the mean and variance, but also in the shape of the distribution, which in the sample distribution may differ from that of the normal distribution in the populations.

Below are shown plots illustrating the processing of sample data.

Fig. 1 shows the logistic regression line calculated using formula (1) with coefficients (18).

LogReg models need to be tested for fit to factual material, i.e. that the factual material really agrees with condition (2). Various goodness-of-fit tests are used to test LogReg models; in this paper we use the Hosmer and Lemeshow chi-square test [2], obtained by stratifying the predictor X:

 

${\chi }^2(H-L)=\underset{}{\sum _i{n}_i·\frac{{(W_{obs,i}-W_{calc,i})}^2}{W_{calc,i}·(1-W_{calc,i})}}$

(19)

 

where summation (index i) is over strata; ni is the number of observations in a stratum; W_obs,i, W_calc,i are the observed and calculated probabilities, respectively; the probability W_obs,i is the mean of the dichotomous response Y in the i-stratum containing ni observations; and W_calc,i is the result of calculations by formula (1) for the mean value of Х in the i-stratum.

If the Hosmer and Lemeshow value of ${\chi }^2$ calculated from the primary data as a result of their stratification does not exceed the critical value χ²_crit(α, ν), the model is recognized as fit to the primary data at a given significance level α (ν is the number of degrees of freedom for the criterion ${\chi }^2$).

Hosmer and Lemeshow [2] described various ways of dividing the observation array into strata, including dividing into 10 strata with an equal number of observations in the strata. Figure 2a shows the result of stratification in model (1)-(2) with coefficients (18). The results for logit(W_obs) and logit(W_calc) are shown in Figure 2b.

In Fig. 2a, we see a very good fit between W_obs,i and W_calc,i . In Fig. 2b the values of logit(W_obs) show minimal differences from logit(W_calc). The points of logit(W_calc) lie strictly on a straight line as per formula (2).

The Hosmer and Lemeshow goodness-of-fit test calculated using the data from [2] as shown in Fig. 2a (χ² = 0.168) is much less than the critical value χ²_crit = 15.51 for ν = 8 degrees of freedom and significance level α = 0.05. Consequently, the hypothesis of logit linearity is not rejected as expected from the results shown in Fig. 2b.

For a more detailed (than with stratification) presentation of the comparisons between calculations and observations, we applied the moving average procedure as described in various publications (see, for example, [6-8]), using a computer program [9]. The moving average results for probability (1) with parameters (18) for an averaging window of n_W = 10 observations are shown in Fig. 3a. Taking the moving average window to be equal to 40 (as for stratification) provides Fig. 3b.

For n_W =10 (Fig. 3a), the average probabilities <W>_obs can take values from zero to one with a step of 0.1 (in reality, <W>_obs were found in the range from 0.1 to 0.9). For each value of <W>_obs, we have a series of average values for X.

For instance, for <W>_obs =0.1 the value of <Х> falls within the interval from – 2.1 to  – 0.9.

In Fig. 3b, the averaging window is wider. <W>_obs are found in the interval from 0.12 to 0.85. The observed values of <Х> for each <W>_obs occur in a narrower interval than in Fig. 3a. 

Formula (18) and Fig. 1-3 represent logistic regression as a method for studying dependencies. Next, we will consider LogReg as a classification method.

Table 2 shows the results of classification of observation objects Y = 0 and Y = 1 obtained in model (1) with coefficients (18) for the probability cutoff point (or ‘cutpoint’ according to Hosmer and Lemeshow [2]) W_cutpoint = 0.5.

 

$Se=\frac{n_{11}}{n_1}=0.695; Sp=\frac{n_{00}}{n_0}=0.695; PMC=\frac{n_{01}+n_{10}}{N}=0.305$

(20)

 

Fig. 4 shows Se and Sp plots for arbitrary cutpoint values.

For cutpoint = 0.5, we obtain Se = Sp = 69.5% as it follows from (20). Using the plots in Fig. 4, we can determine Se and Sp for any cutpoint value. If, for example, it were required to increase the model’s Se, it would be necessary to decrease the cutpoint; thus, for cutpoint = 0.4 we have Se = 82.0% and Sp = 53.5% (increasing Se obviously decreases Sp and vice versa).

There are two more characteristics of the LogReg model as a classification method: the Youden index J = Se + Sp - 1 (the plot is shown in Fig. 5) and the ROC curve (Fig. 6). The Youden index is often used to find the “optimal” cutpoint value [10]. The ROC curve allows one to analyze the quality of classifiers, particularly in binary classification, i.e. using this approach one can evaluate how accurately the elements of a given set were classified into two groups [11]. ROC curves are also used to visualize the relationship between Sp and Se, and to compare classification models.

According to Fig. 5, the index J = 0.39 is maximal for cutpoint = 0.5; Fig. 6 provides the area under the ROC-curve AUC = 0.762.

The value of PMC = 0.305, calculated by (20) through n₀₁ and n₁₀, matches that computed by formula (7) for two normal distributions with equal variances.

Let us continue our consideration of LogReg models in the case of two normal distributions for X with equal variances. In Example 1 described above, the proportion of Y = 1 values in the sample was π = 0.5 (equal number of zeros and ones among Y values). Consider a situation with an unequal number of zeros and ones in the response Y. To this end, we use the database of Example 1, in which the number of X values for Y= 0 is doubled, i.e. π = 0.3333. For this database, we obtain

 

$b_0=-1.1963; b_1=+1.0071; OR=2.7376 (CI=2.2226-3.3720)$

(21)

 

Here, the value of the coefficient b₀ compared with (18) has increased 2.5 times, which is in agreement with formula (4); the values of the coefficient b₁ and the odds-ratio OR are practically the same as in (18), but the confidence interval for OR has become narrower as a result of increasing the number of observations from 400 to 600. Fig. 7 shows the Se and Sp plots.

At the point where the lines “intersect” and where Se = Sp and Se and Sp are still equal to 69.5%; this equality is achieved, unlike in the previous case, for cutpoint = 0.3333. At cutpoint = 0.5 we have Se = 42.5 and Sp = 88.5 %.

Graphically, the Youden plot looks similar to that in Fig. 5; the difference from the latter being that J is maximal at cutpoint = 0.3333. The ROC curve is the same as in Fig. 6, and the area under it has not changed – AUC = 0.762.

Now consider the case where the proportion of Y = 1 values in the sample is greater than π = 0.5. For this purpose, let us use the database of Example 1, in which the number of X values for Y = 1 is doubled, i.e. π = 0.667. Given this database, we obtain b₀= + 0.1892;  b₁= + 1.0071;  OR = 2.7376 (CI 2.2225 – 3.3720).

The Se and Sp plots are similar to those shown in Fig. 7 with the only differences that now the equality Se = Sp is achieved for cutpoint = 0.6667.

2.2 Example 2: Samples from two populations with normal distributions and different variances (Situation 2А)

To perform these calculations, two samples of 200 observations each were drawn from the populations with normal distributions and unequal variances N (8, 1) and N (10, 4) for Y = 0 and Y = 1, respectively. The values of the mathematical expectations and variances were chosen so that all values of X in the samples should be positive. According to the above (Situation 2), the variances being unequal, the probability W(Y = 1|X = x) is described by a shape that is quadratic in X (8)-(9) with parameters (10)-(12).

For the sample distributions N(8, 1) and N(10, 4), the LogReg model with a quadratic term is given by

 

$\log it\left(W\right)=18.34+5.401·X+0.3699·X^2$

(22)

 

The coefficients for Х and Х² in model (22) are highly significant, p<10^–6. If calculations were performed for the populations N (8, 1) and N (10, 4), rather than for the samples from them, model (22), by virtue of formulas (8)-(9), would be given by

 

$\log it\left(W\right)=18.5-5.5·X+0.375·X^2$

(23)

 

It is evident that the differences between model (23) based on the data of the populations, and the real model (22) based on samples from these populations are minimal.

Let us perform a graphical analysis of the calculations by formula (22). Fig. 8 shows the logistic regression line constructed by model (22), containing a linear X and quadratic X² term.

It can be seen that the logistic curve in Fig. 8 for the quadratic model is radically different from that for the linear model (18) shown in Fig. 1.

To assess the quadratic model (22) for goodness of fit, perform stratification of the predictor X into 10 strata with 40 observations per stratum. The result is shown in Fig. 9.

In Fig. 9 we see a very good fit between <W>_obs and W_calc.

Based on the stratification results, calculate the Hosmer and Lemeshow chi-square goodness-of-fit test. We obtain χ² = 0.532. This value is much less than the critical value χ²_crit = 15.1; this means that the model is fit, which is obvious from Fig. 9.

For a more detailed description of the observed probability values, perform the moving average procedure (Fig. 10).

In Fig. 10a, B, the moving average results (the observed probabilities W_obs) are show a good fit with the probabilities W_calc calculated by the logistic regression method with coefficients (22) and confirm the stratification results in Fig. 9.

Now consider the LogReg model (22) as a method of classification. Fig. 11 shows the Se and Sp plots as a function of cutpoint.

It is evident from Fig. 11 that the shapes of the Se and Sp plots are clearly different from those in Fig. 4 for the distributions of X with equal variances. In the logistic regression (22) with cutpoint = 0.5, Se = 66.0%, and Sp = 89.0%. Fig. 11 shows that the equality of Se and Sp (Se = Sp = 75.0%) in model (22) is achieved for a cutpoint of 0.3525; the Youden index for this cutpoint is not maximal, being equal to J = 0.5.

The Youden index is found to be maximal (J = 0.545) for a cutpoint in the range from 0.463 to 0.574; this range does not include cutpoint = 0.3525, for which Se = Sp. For cutpoint = 0.463, we have Se = 68.5% and Sp = 86.0%, J = 0.545. For cutpoint = 0.574, the values of Se and Sp are Se = 62.5% and Sp = 92.0%, the value of J = 0.545 being the same. The Youden index is often used to find the "optimal" cutpoint based on the maximum value of J [10]. If there are several "optimal" values with the maximum J, the cutpoint is calculated as the arithmetic mean.

The shape of the ROC curve for model (22) is similar to that for the linear model (14). At the same time, the area under the ROC curve (AUC=0.8) is slightly larger than in model (14) in Fig. 6.

Addendum to Example 2

For the two samples N (8, 1) and N (10, 4) described in Example 2, which lead to the X-quadratic logistic regression model (22), construct an incorrect LogReg model with one linear predictor X. This is a situation that often occurs in reality when those who develop LogReg models do not test the LogReg applicability conditions and construct a “standard” LogReg model with the only predictor X available [15]. The calculations by the LogReg method for two samples N (8, 1) and N (10, 4) under the assumption of logit linearity are represented by formula (24):

 

$\log it\left(W\right)=7.66+0.864·X; OR=2.373 (CI=1.968-2.862)$

(24)

 

The coefficient b₁ = 0.864 for the predictor Х of model (24) is extremely highly significant, p<10^–17! Such a high statistical significance of the coefficient b1 may suggest that model (24) fits the factual material, i.e. the relationship of logit (W) with X is indeed linear, not quadratic. The fallacy of this assumption can be shown by performing the Hosmer-Lemeshow goodness-of-fit test: for model (24) we have χ² = 31.25, which exceeds significantly the critical χ²_crit = 15.5; therefore, model (24) does not fit the factual material of N (8, 1) and N (10, 4). The correct logistic regression model for these distributions is the X-quadratic model (22).

2.3 Example 3: Samples from two populations with normal distributions, different variances and equal mathematical expectations (Situation 2b)

In this example, we consider two samples from the populations with normal distributions N (10, 1) and N (10, 9), in which the variances are different (as in Example 2), but the mathematical expectations are equal. In this case, there are some peculiarities compared to Example 2.

Different variances in the two samples for the predictor X lead to the LogReg model being quadratic in X even for equal mathematical expectations. The logistic regression with two predictors X and X² for the samples from N (10, 1) and N (10, 9) is given by:

 

$\log it\left(W\right)=43.333-8.8844·X+0.44422·X^2$

(25)

 

The coefficients of X and X² in model (25) are highly significant, p<10^–6. The LogReg model for the populations N (10, 1) and N (10, 9), according to (10)-(12), takes the form:

 

$\log it\left(W\right)=43.335-8.8848·X+0.44424·X^2$

(26)

 

which is almost the same as (25).

Fig. 12 shows a regression line constructed according to model (25).

The figure demonstrates a logistic curve which is unusual for this quadratic model when compared with that in Fig. 1; this curve appears to be similar to the plot in Fig. 8.

Let us test model (24) for goodness of fit to factual data using stratification, i.e. verify that the factual data are indeed described by the X-quadratic function (25). The results of the stratification are shown in Fig. 13.

Fig. 13 shows a very good fit between the observed and calculated probabilities Wobs and Wcalc in all strata. Consequently, model (25) should fit the factual data, i.e. the two samples from the populations N (10, 1) and N (10, 9). However, checking the model for goodness off fit using the Hosmer and Lemeshow test gives an unexpected result: the criterion’s value χ² = 37.45 exceeds significantly the critical value χ²_crit = 15.1, i.e. formally model (25) does not fit the available data N (10, 1) and N (10, 9). This result is unexpected because the graphical representation of the stratification results in Fig. 13 displays a virtually complete fit between theory and factual material.

The unexpectedly high value of χ² = 37.45 is explained by the peculiarities of calculating the Hosmer and Lemeshow goodness-of-fit test. The numerator of the test is the difference between the observed and calculated probabilities W_obs and W_calc. As stated above, the differences between them are very small. The denominator of the test is the product W_calc∙(1 – W_calc), which tends to zero as the calculated probability W_calc approaches zero or one. Thus, if in any strata in the calculation of the H-L test the probability W_calc approaches zero or one, these strata make a very large contribution to χ².

As a result of the stratification that we used in constructing the Hosmer and Lemeshow test for model (25), we discover two strata (the first and the last) in which the W_calc values are practically equal to one (see Fig. 13). These two strata provide the maximum contribution to the total χ², although the differences between W_calc and W_obs in these strata are quite small. If we remove these two strata from the calculation of the Hosmer and Lemeshow chi-square test, the chi-square becomes equal to just χ² = 0.1048 (practically zero), i.e. model (25) fits the factual material consisting of two normal distributions N (10, 1) and N (10, 9) for the predictor X. This once again emphasizes the importance of graphical representation of LogReg results for assessing models for goodness of fit.

In addition to stratification, we calculated the moving average with an averaging window of 40. The results are shown in Fig. 14a for the probability, and in Fig. 14b for the logit.

Here we come across another surprise. In Fig. 14a, like in Fig. 13, we see for a good fit between the calculated and observed values of W_calc and W_obs for probabilities. At the same time, for the logit (Fig. 14b) there are significant differences between the results of calculation by the logistic regression method logit, W_calc, and the moving average logit W_obs for both small and large values of X. The reason for this discrepancy is clear: when the probability approaches one (Fig. 14a), small differences between the calculated and observed probabilities of W lead to large differences between the calculated and observed values of logits (Fig. 14b) due to the denominator in the expression for the calculated logit $\ln \left(\frac{W}{1-W}\right)$, which contains the complex (1 – W_calc) that we have already discussed.

Fig. 15-16 show an analysis of the LogReg model (25) as a classification method.

Fig. 15 demonstrates that the equality of Se and Sp (Se = Sp = 72.0%) is achieved for cutpoint = 0.361, the Youden index being J = 0.445. For cutpoint = 0.5, we obtain Se = 60.0% and Sp = 89.0%; and the Youden index takes the maximum value J = 0.490. The area under the ROC-curve is equal to 0.679.

Addendum to Example 3

In Example 3, we considered two samples from normally distributed N (10, 1) and N (10, 9) with different variances but equal mathematical expectations. According to Situation 2, in this case the X-quadratic model is correct (25). Now let us attempt to construct a logistic regression model with one predictor Х (linear model of type (2)). Obviously, such a model would be unfit, but it does occur in actual studies where the LogReg model is not tested for applicability. According to the “theoretical” formulas (7)-(9), the model for the distributions N (10, 1) and N (10, 9) should have a nonzero linear term. The coefficient of this linear term should equal b₁ = – 8.8848 and be (according to formula (25)) highly significant. Let us construct a LogReg model with one linear term in X for the distributions N (10, 1) and N (10, 9). Contrary to expectations, the coefficient b₁ of this linear term is practically equal to zero, p>0.999. An argument in favor of b₁ = 0 is available in Fig. 13 (except for calculations by the LogReg method), in which the logistic regression line is symmetrical relative to the center X = 10.

Since the logistic curve in Fig. 12 in Example 3 resembles a parabola, we will attempt to construct a LogReg model with one X-quadratic term

 

$\log it\left(W\right)=b_0+b_2·X^2$

(27)

 

Calculations using the logistic regression method with the sample distributions N (10, 1) and N (10, 9) give, as per (27), the value of b₂ = 0.00389, which is 100 times less than the value of the coefficient b₂ = 0.44422 in model (25), which contains both terms, linear and X-quadratic.

To be fair, formula (11) predicts a non-zero linear coefficient b₁, just as formula (12) predicts a non-zero quadratic coefficient b₂ when they are calculated as part of the full quadratic model (9) and (25). These data once again emphasize the need to check the conditions of LogReg model applicability to factual material before it is used for calculations and analysis of the results.

2.4 Example 4: Predictor Z is distributed lognormally (Situation 3)

To obtain lognormal distributions for Z, we used two normal distributions with equal variances N (µ₀, σ²) and N (µ₁, σ²) for the predictor X. After raising the exponent to the power of X, we have two lognormal distributions for the predictor Z = exp(X).

The calculations involved two samples from normally distributed populations N (3, 1) and N (4, 1). The values of the mathematical expectations were chosen so that all values of X were positive.

Constructing a logistic regression model for the predictor X = ln(Z) for the data of the distributions N (3, 1) and N (4, 1) gives

 

$\log it\left(W\right)=-0.5040+1.0079·X$

(28)

 

Since X = ln(Z) are distributed normally with equal variances, we come to all the conclusions described above in Example 1 (see Figures 1-6).

In this example, it is of greater interest to analyze the situation where logistic regression is constructed for the predictor Z itself, distributed lognormally. We realize that in this case the applicability conditions for LogReg model (2) will be clearly violated. Previously we mentioned that in the literature there are cases where the applicability of the LogReg model is not verified, and LogReg is constructed depending on the predictor that is available irrespective of whether LogReg is applicable or not [15].

So, let us construct, based on this factual material, a LogReg model for the predictor Z:

 

$\log it\left(W\right)=-0.8824+0.01775·Z$

(29)

 

The observed values of Y=0.1 and probability W(Y=1|Z=z) in the data N(3, 1) and N(4, 1) calculated by the LogReg method using formula (29) are shown in Fig. 16.

The coefficient for Z in formula (29) is statistically highly significant, p<10^-7. The high statistical significance of the predictor Z in the LogReg model (29) may suggest that logit(W) is indeed a linear function of Z, although this is obviously not the case. We can see such an incorrect conclusion, for instance, in [16]. Let us check the assumption of linearity of logit(W) in formula (29) applying the Hosmer and Lemeshow goodness-of-fit test to the data divided into 10 strata by Z, with 40 observations in each stratum. The result is shown in Fig. 17a. We obtained χ² = 31.32, which is much greater than the critical value χ² = 15.5; therefore, model (29) is recognized as unfit for the factual material. Fig. 17b shows the moving average with a window of 40 observations.

 

That the model is unfit is clearly obvious if we plot the stratification results (Fig. 17a) or the moving average with the window of 40 observations (Fig. 17b).

As expected, according to the logistic regression calculations, logit W_calc in Fig. 17 is a linear function of the argument Z, while in reality (logit W_obs) has a logarithmic relationship between the logit and the predictor Z, which is distributed lognormally.

The results of using model (29) as a classification method are presented below. The Se and Sp plots are shown in Fig. 18.

For cutpoint = 0.5, we have Se = 53.5% and Sp = 82.5%; the equality Se = Sp = 69.4% is observed at cutpoint = 0.427. Note that the values of Se = Sp = 69.4% obtained for the unfit model (29) are almost identical to those for the fit model (28) only at different cutpoint values. It appears that models (28) and (29) are completely different if used as a method for studying dependencies, but possess almost identical classification properties.

3 Comparison of “perfect” situations 1-3 with data from real epidemiological studies

The situations 1-3 described above and the corresponding examples 1-4 can be called "perfect" in the sense that they are based on artificially created data that exactly match the conditions specified for populations with preset normal distributions. All of these "perfect" situations are encountered in real epidemiological studies. Below, some LogReg models built on real data are compared with the models from Examples 1-4.

Example 1 (two normal distributions with equal variances) took place in a real study of the relationship between cardiovascular diseases (CVD) and age. In the example considered in [2], the average age in the groups is <Age₀>=39.2 years and <Age₁>=51.3 years (the differences are statistically significant), and the variance ratio is 1.045 (the differences are not significant). The results of comparing the real and perfect data are shown in Figures 3a (from Example 1) and 19 (from [17]).

The difference between the results obtained on the real data [2, 17] and the “perfect” ones is that the real data, with stratification and moving average, are “irregularly” scattered around the logistic regression lines.

Figures 4a and 20 display a comparison of the Se and Sp plots for the perfect data of Example 1 and for the LogReg model built on real data [2, 17].

Previously we showed for two samples from populations with normal distribution and equal variances at cutpoint = 0.5 that Se = Sp = 69.5% (Fig. 4a and Fig. 20). The maximum value of the Youden index is J = 0.39. For real data (Fig. 20) at cutpoint = 0.5, we have Se = 67.4%; Sp = 80.7%. The Youden index is J = 0.481. For real data, the equality Se = Sp = 74.0% takes place at W_cutpoint = 0.421. For this cutpoint, we have the Youden index J = 0.480.

Example 2a (two normal distributions with different variances) occurred in a study of the relationship between cardiovascular diseases in industrial workers in the Urals (Russia) and the body mass index (BMI) as predictor. In the example considered in [17], the average BMI values in the groups are <BMI₀> = 25.5 kg/m² and <BMI₁> = 28.2 kg/m² (the differences are statistically significant), the variance ratio is 1.344 (the differences are significant). Fig. 9 presents the “perfect” example 2. The probability plot constructed using the data from [17] is shown in Fig. 21.

The plots in Figures 9 and 21 have common features. At the beginning we see a plateau, and then a systematic increase in probability.

Example 3 took place in a real epidemiological study investigating the statistical relationship between the prevalence of thyroid disease and BMI in older women (Russia). The primary data were taken from [18], and the calculation results were published in [17]. In the example considered in [17], the average BMI values in the groups of women with and without thyroid disease were <BMI₀> = 27.0 kg/m² and <BMI₁> = 26.9 kg/m² (the differences are not statistically significant), the variance ratio was 1.341 (the differences are significant). The results of comparing the perfect and real data are shown in Fig. 22a repeats Fig. 14a from Example 3 without probability estimates (a perfect option); Fig. 22b for real data was taken from [17].

The relationship in Fig. 22 is not monotonic. We can see a dip for in perfect data (Fig. 22a) and real data (Fig. 22b). In the range of BMI values of 22-26 kg/m² the probability of W(TG) decreases on average with increasing the BMI, while in the range of BMI of 26-30 kg/m² W(TG) increases.

Example 4 (lognormal distribution of predictor Z). In epidemiological studies, a typical Z indicator which is often distributed lognormally is radon [19]. We have a small database of information on radon levels and possible lung cancer cases in the population of the city of Lermontov, Russia [20]. For this database, we checked the radon distribution shape in the groups of control and cases, and tested the logistic regression models for the relationship between radon and lung cancer for goodness of fit. For testing, we used formula (2) in which the relationship of the logit with radon Z is assumed to be linear, and formula (13), in which the logit is related to the logarithm of Z. The radon distribution histograms for the controls and cases show a clear difference in distribution pattern from the normal one (Fig. 23).

However, taking the logarithm for radon does not lead to normal distributions; these differ from normal, especially for the group of cases, Fig. 24.

According to the criteria for testing the normality of distribution, for the control group we have: Liliefors, p<0.10; Shapiro-Wilk, p = 0.048; therefore, the distribution of ln(Rn) for the controls barely passes the normality test. For the group of cases, we have Liliefors, p<0.05; Shapiro-Wilk, p = 0.00012, i.e. the distribution clearly differs from normal. Thus, the data from [20] do not fully fit Situation 3 described above, but are close to it.

In this context, it is interesting to compare the logistic regression models constructed on data for Rn itself and for its logarithm ln(Rn), which take the form

 

$\log it\left(W\right)=-0.270+0.0106·Rn$	(30)
$\log it\left(W\right)=-0.760+0.302·\ln \left(Rn\right)$	(31)

 

Let us check the logistic regression models for Rn and for ln(Rn) for goodness of fit using stratification into 10 strata and the Hosmer and Lemeshow test. The result is shown in Fig. 25.

A visual inspection of the plots in Fig. 25 shows that the actual data obtained by stratification are significantly different from the LogReg calculation results. However, given such differences, the Hosmer and Lemeshow test confirms that both models are fit, since the resulting values of χ² = 4.73 and χ² = 6.92, respectively, are much less than the critical value χ² = 15.5 for 8 degrees of freedom and α = 0.05 (obviously, because of a small number of observations).

Interestingly, for model (30) with a linear radon predictor (as in Fig. 25a), the value of χ² turned out to be smaller than for the ln(Rn) model (31) (see Fig. 25b) despite the fact that the distributions from the normal are greater for Rn itself than for ln(Rn). Probably, here we deal with a situation where condition (2) is satisfied better for Rn itself than for its logarithm, which is quite possible regardless of the shape of the Rn or ln(Rn) distribution.

Supporting information

Digression 1

For any logistic regression model with one quantitative predictor X in a situation with two normal distributions with equal variances, we have Se and Sp values at the 70% level. If a real study yields higher (or lower) Se and Sp values, this could mean two things:

1 – the factual data are consistent with condition (2), but the distributions of X are not normal with equal variances; in this case, it is desirable to test the relationship of the logit with X for linearity either by stratification and performing  goodness-of-fit tests or by using moving average procedures;

2 – the factual data are not consistent with condition (2); in this case, it is especially desirable to show the actual shape of the logit’s relationship with X using moving average procedures.

Digression 2

Assume that in an epidemiological study the value Y = 0 corresponds to healthy individuals, while Y = 1 to sick ones [12]. By selecting a certain cutpoint value = W^* in the logistic regression model, we can state that for W>W^* the LogReg model predicts sick individuals, and for W<W^* it predicts healthy ones. In this case, each probability value W^* can be associated with a predictor value X^*, whereby disease is predicted for X>X^*, and absence of disease for X<X^*. The value of X^* is usually called X cutpoint value [13]. For instance, in this example for W^* = 0.5 from Fig. 1 we obtain X^* = 0.50. For X>X^* = 0.5 disease is predicted, and for X<X^* the model predicts absence of disease.

Digression 3

In Example 1 (see above), for W^* = 0.5 there was one cutpoint for the predictor X^* = 0.50 (Fig. 1). Then, for X>X^* = 0.50, disease was predicted, and no disease for X<X^*. In this case, for the selected cutpoint = W^*, according to Fig. 12, we obtain two values of X^*. For example, for W^* = 0.361 (when Se = Sp), we obtain X₁^* = 8.92 and X₂^* = 11.08. Sick people would have values of X < 8.92 and X > 11.08; healthy people would have values of X in the range from 8.92 to 11.08. Cases where both small and large values of some health indicator are considered pathological do occur in medical practice. Almost all general blood test indices can take values below and above the norm.

For instance, low platelet levels are associated with thrombocytopenia, while high levels with thrombocytosis [14].

Conclusions

In this paper, we investigated the shape of the relationship between the logit of the probability of Y = 1 with predictor X occurring in logistic regression models for different variants of normal distribution of X in the groups Y = 0 and Y = 1. In the case of normal distribution with equal variances, the theory predicts a linear relationship between logit(W) and the predictor X, which is confirmed by numerical calculations. In the case of normal distribution with unequal variances, the relationship between logit(W) and the predictor X turns out to be quadratic. Numerical calculations show that an attempt to describe it by a linear function leads to results that do not fit primary data. This disagreement is clearly revealed by graphical representation of stratification and moving average results.

Examples are provided of how different variants of logistic regression model with a normal distribution of the predictor X occur in actual epidemiological studies, and these are compared with “perfect” models.

Testing of LogReg models revealed cases of discrepancy between results obtained by the goodness-of-fit criteria and by the stratification and moving average methods and presented in graphical form. It is shown that graphical methods provide a more detailed evaluation of the results of model fitness than goodness-of-fit criteria. Moreover, where the model is unfit, graphical representation of stratification and moving average results can be used to establish the real shape of the relationship between the response Y and the predictor X in primary data.

It is shown for the variants considered how calculations by the logistic regression method compare with observed data, and what happens when the condition of logistic regression applicability is violated.

Author contributions

• Varaksin A. N. – conceptualized the methodology of this research, wrote the original draft and conducted the investigation, analysis and visualizations, worked on the research framework and discussion, wrote the original draft of the study.
• Konstantinova E. D. – worked on the research framework and discussion, writing and editing all the main sections of the study.
• Maslakova T. A. – worked on the research framework and discussion, writing and editing all the main sections of the study.
• Ogorodnikova S. Yu. – worked on the research framework and discussion, writing and editing all the main sections of the study.
• Kabakova E. A. – worked on the research framework and discussion, writing and editing all the main sections of the study.

All authors contributed significantly to the review and editing of the paper. The authors have accepted responsibility for the entire content of this manuscript and approved its submission.

Consent to publication

Not applicable.

Research funding

This research was carried out using subsidies from the Ministry of Education and Science of the Russian Federation for the implementation of the scientific topic FUMN-2024-0002.

Conflict of interest

All authors state no conflict of interest.

Statement of originality

Not applicable.

Data availability

The journal's editorial policy on data access issues does not apply to this work.

Generative artificial intelligence

Not applicable.

Review and consideration

This manuscript was sent to the editorial board of the journal on its own initiative.

About the authors

Anatoly N. Varaksin

Institute of Industrial Ecology, Ural Branch of the Russian Academy of Sciences

Email: varaksin@ecko.uran.ru
ORCID iD: 0000-0003-2689-3006
SPIN-code: 9910-2326

Doctor of Physics and Mathematics, Professor, chief Researcher at the Biostatistics Laboratory

Russian Federation, 620066, Ekaterinburg, St. Sofya Kovalevskaya, 20

Ekaterina D. Konstantinova

Institute of Industrial Ecology, Ural Branch of the Russian Academy of Sciences

Author for correspondence.
Email: K_konst@ecko.uran.ru
ORCID iD: 0000-0002-2260-744X
SPIN-code: 1887-7612

Ph. D., head of the Biostatistics Laboratory

Russian Federation, 620066, Ekaterinburg, St. Sofya Kovalevskaya, 20

Tatyana A. Maslakova

Institute of Industrial Ecology, Ural Branch of the Russian Academy of Sciences

Email: masta@ecko.uran.ru
ORCID iD: 0000-0001-6642-9027
SPIN-code: 3233-7652

Ph.D., Senior Researcher at the Biostatistics Laboratory

Russian Federation, 620066, Ekaterinburg, St. Sofya Kovalevskaya, 20

Svetlana Yu. Ogorodnikova

Institute of Industrial Ecology, Ural Branch of the Russian Academy of Sciences

Email: ogor.svetlana@ecko.uran.ru
ORCID iD: 0000-0001-7193-3607
SPIN-code: 7818-4401

Russian Federation, 620066, Ekaterinburg, St. Sofya Kovalevskaya, 20

Elizaveta A. Kabakova

Institute of Industrial Ecology, Ural Branch of the Russian Academy of Sciences

Email: Liza.Kabakowa@yandex.ru
ORCID iD: 0009-0005-2173-1202
SPIN-code: 3652-0321

Research Engineer at the Biostatistics Laboratory

Russian Federation, 620066, Ekaterinburg, St. Sofya Kovalevskaya, 20

References

Supplementary files