Paper/sections/06_Results.tex

@@ -100,15 +100,15 @@ keeping enrollment open.
 
 \begin{table}[H]
     \centering
-    \caption{Boxplot Summary Statistics: percentage point due to intervention}
+    \caption{Boxplot Summary Statistics}
     \label{table:boxplotsummary}
     \begin{tabular}{ | c c c c c c c c | }
         \hline
         5th & 10th & 25th & median & 
         75th & 90th & 95th & mean \\
         \hline
-        -2.1 & -0.8 & 0.0 & 1.2 &
+        -0.376 & -0.264 & -0.129 & -0.023 &
-         4.2 &  8.2 & 11.0  & 2.5 \\
+         0.145 &  0.925 & 0.982  & 0.096 \\
         \hline
     \end{tabular}
 \end{table}
@@ -119,16 +119,66 @@ The key figures from the boxplot in figure
 \ref{fig:pred_dist_diff_delay}
 are sumarized in table \ref{table:boxplotsummary}
 There are a few interesting things to point out here. 
-First, over 75\% of the probability mass is equal to or above zero, suggesting
+Let's start by getting aquainted with the details of the distribution above.
-that most trials will experience some harm from a delay in closing enrollment.
+A couple more points
-Seconds, about 39.1\% of the probability mass is contained within the interval 
+First, 63\% of the probability mass is equal to or below zero.
+Seconds, about 13\% of the probability mass is contained within the interval 
 [-0.01,0.01].
 The full 5\% percentile table can be found in  table
 \ref{TABLE:PercentilesOfDistributionOfDifferences}
-\todo{fix table}
 in appendix
 \ref{Appendix:Results}
 
+It can also be devided into a few different regimes.
+% - spike at 0
+% - the boxplot
+% - 63% of mass below 0 : find better way to say that
+%   - For a random trial, there is a 63% chance that the impact is to reduce the probability of a termination.
+% - 2 pctg-point wide band centered on 0 has ~13% of the masss
+% - mean represents 9.x% increase in probability of termination. A quick simulation gives about the same pctg-point increase in terminated trials.
+% - there are 3 regimes: low impact (near zero), medium impact (concentrated in decreased probability of termination), and high impact (concentrated in increased probability of termination). 
+The first regime consists of the low impact results, i.e. those values of $\delta_p$ 
+near zero. 
+About 13\% of trials lie within a single percentage point change of zero, 
+suggesting that there is a reasonable chance that delaying 
+a close of enrollment has no impact. 
+The second regime consists of the moderate impact on clinical trials'
+probabilities of termination, say values in the interval $[-0.5, 0.5]$ 
+on the graph.
+Most of this probability mass is represents a decrease in the probability of 
+a termination, some of it rather large decreases.
+The third regime consists of the high impact region, 
+almost exclusively concentrated above increases in the probability of 
+termination $\delta_p > 0.75$. 
+These represent cases where delaying the close of enrollemnt changes a trial
+from a case where they were highly likely to complete their primary objectives to 
+a case where they were likely or almost certain to terminate the trial early.
+%   - the high impact regime is strange because it consists of trials that moved from unlikely (<20% chance) of termination to a high chance (>80% chance) of termination. Something like 5% of all trials have a greater than 98 percentage point increase in termination. Not sure what this is doing. 
+
+% Looking at the spike around zero, we find that 13.09% of the probability mass 
+% is contained within the band from [-1,1]. 
+% Additionally, there was 33.4282738% of the probability above that 
+% – representing those with a general increase in the 
+% probability of termination – and 53.4817262% of the probability mass 
+% below the band – representing a decrease in the probability of termination. 
+% On average, if you keep the trial open instead of closing it, 0.6337363% of
+% trials will see a decrease in the probability of termination, but, due to 
+% the high increase in probability of termination given termination was 
+% increased, the mean probability of termination increases by 0.0964726. 
+
+
+% Pulled the data from the report
+% ```{r}
+% summary(pddf_ib$value) 
+% Min.      1st Qu.   Median   Mean    3rd Qu. Max. 
+% -0.99850 -0.12919 -0.02259 0.09647 0.14531 1.00000 
+% quants <- quantile(pddf_ib$value, probs = seq(0,1,0.05), type=4) 
+%    # Convert to a data frame 
+%    quant_df <- data.frame( Percentile = names(quants),  Value = quants ) 
+%    kable(quant_df)
+%    Percentile Value 
+%   SEE TABLE IN APPENDIX
+%```
 
 Figure \ref{fig:pred_dist_dif_delay2} shows how the different disease categories
 tend to have a similar results:
@@ -137,81 +187,85 @@ tend to have a similar results:
     \caption{Distribution of Predicted differences by Disease Group}
     \label{fig:pred_dist_dif_delay2}
 \end{figure}
+Again, note the high mass near zero, the general decrease in the probability
+of termination, and then the strong upper tails.
+
+Continuing to the $\beta$ parameters in figure 
+\ref{fig:parameters_ANR_by_group}, 
+we can see the estimated distributions 
+the status: \textbf{Active, not recruiting}.
+The prior distributions were centered on zero, but we can see that the 
+pooled learning has moved the mean
+values negative, representing reductions in the probability of termination 
+across the board. 
+This decrease in the probability of termination is strongest in the categories of Neoplasms ($n=49$),
+Musculoskeletal diseases ($n=17$), and Infections and Parasites ($n=20$), the three categories with the most data.
+As this is a comparison against the trial status XXX, we note that YYY.
+\todo{The natural comparison I want to make is against the Recruting status. Do I want to redo this so that I can read that directly?It shouldn't affect the $\delta_p$ analysis, but this could probably use it. YES, THIS UPDATE NEEDS TO HAPPEN. The base needs to be ``active not recruiting.''}
+Overall, this is consistent with the result that extending a clinical trial's enrollment period will reduce the probability of termination.
 
-% Continuing to the $\beta$ parameters in figure 
+\begin{figure}[H]
-% \ref{fig:parameters_ANR_by_group}, 
+    \includegraphics[width=\textwidth]{../assets/img/betas/parameter_across_groups/parameters_12_status_ANR}
-% we can see the estimated distributions 
+    \caption{Distribution of parameters associated with ``Active, not recruiting'' status, by ICD-10 Category}
-% the status: \textbf{Recruiting}.
+    \label{fig:parameters_ANR_by_group}
-% %TOFIX: Discuss how this is a fixed effect with no comparator,i.e. compared 
+\end{figure}
-% %to the "average" conditions it is an "increase/decrease" in the probability of termination.
+% - 
-% This xxx in the probability of termination is strongest in the categories of Neoplasms ($n=49$),
-% Musculoskeletal diseases ($n=17$), and Infections and Parasites ($n=20$), the three categories with the most data.
-% % As this is a comparison against the trial status XXX, we note that YYY.
-% % \todo{The natural comparison I want to make is against the Recruting status. Do I want to redo this so that I can read that directly?It shouldn't affect the $\delta_p$ analysis, but this could probably use it. YES, THIS UPDATE NEEDS TO HAPPEN. The base needs to be ``active not recruiting.''}
-% Overall, this is consistent with the result that extending a clinical trial's enrollment period will reduce the probability of termination.
-%
-% \begin{figure}[H]
-%     \includegraphics[width=\textwidth]{../assets/img/betas/parameter_across_groups/parameters_12_status_ANR}
-%     \caption{Distribution of parameters associated with ``Active, not recruiting'' status, by ICD-10 Category}
-%     \label{fig:parameters_ANR_by_group}
-% \end{figure}
-% % - 
 
 
 %   - Potential Explanations for high impact regime:
-% This leads to the question:
+This leads to the question:
-% ``How could this intervention have such a wide range in the intensity 
+``How could this intervention have such a wide range in the intensity 
-% and direction of impacts?''
+and direction of impacts?''
-% The most likely explanations in my mind are that either
+The most likely explanations in my mind are that either
-% some trials are highly suceptable to enrollment struggles or that this is a 
+some trials are highly suceptable to enrollment struggles or that this is a 
-% modelling artifact.
+modelling artifact.
-% %       - Some trials are highly suceptable. This is the face value effect
+%       - Some trials are highly suceptable. This is the face value effect
-% The first option -- that some trials are more suceptable to 
+The first option -- that some trials are more suceptable to 
-% issues with participant enrollment -- should allow us to 
+issues with participant enrollment -- should allow us to 
-% isolate categories or trials that contribute the most to this effect.
+isolate categories or trials that contribute the most to this effect.
-% This is not what we find when we inspect the categories
+This is not what we find when we inspect the categories
-% in figure 
+in figure 
-% \ref{fig:pred_dist_dif_delay2}.
+\ref{fig:pred_dist_dif_delay2}.
-% Instead it appears that most of the categories have this high 
+Instead it appears that most of the categories have this high 
-% impact regime when $\delta_p > 0.75$, although the maximum value
+impact regime when $\delta_p > 0.75$, although the maximum value
-% of this regime varies considerably.
+of this regime varies considerably.
-%
+
-% Another explanation is that this is a modelling artefact due to priors 
+Another explanation is that this is a modelling artefact due to priors 
-% with strong tails and the relatively low number of trials in 
+with strong tails and the relatively low number of trials in 
-% each ICD-10 categories.
+each ICD-10 categories.
-% In short, there might be high levels of uncertanty in some parameter values,
+In short, there might be high levels of uncertanty in some parameter values,
-% which manifest as fat tails in the distributions of the $\beta$ parameters. 
+which manifest as fat tails in the distributions of the $\beta$ parameters. 
-% Because of the logistic format of the model, these fat tails lead to 
+Because of the logistic format of the model, these fat tails lead to 
-% extreme values of $p$, and potentally large changes $\delta_p$. 
+extreme values of $p$, and potentally large changes $\delta_p$. 
-% I believe that this second explanation -- a model artifact due to uncertanty --
+I believe that this second explanation -- a model artifact due to uncertanty --
-% is likely to be the cause. 
+is likely to be the cause. 
-% A few things lead me to believe this:
+A few things lead me to believe this:
-% \begin{itemize}
+\begin{itemize}
-%     \item The low fractions of E-BFMI suggest that the sampler is struggling 
+    \item The low fractions of E-BFMI suggest that the sampler is struggling 
-%         to explore some regions of the posterior. 
+        to explore some regions of the posterior. 
-%         According to 
+        According to 
-%         \cite{standevelopmentteam_runtimewarningsconvergence_2022}
+        \cite{standevelopmentteam_runtimewarningsconvergence_2022}
-%         this is 
+        this is 
-%         often due to thick tails of posterior distributions. 
+        often due to thick tails of posterior distributions. 
-%         During earlier analysis, when I had about 100 trials, the number of 
+        During earlier analysis, when I had about 100 trials, the number of 
-%         warnings was significantly higher.
+        warnings was significantly higher.
-%     \item When we examine the results across different ICD-10 category, 
+    \item When we examine the results across different ICD-10 category, 
-%         \ref{fig:pred_dist_dif_delay2}
+        \ref{fig:pred_dist_dif_delay2}
-%         we note that most categories have the same upper tail spike.
+        we note that most categories have the same upper tail spike.
-%     \item In Figure 
+    \item In Figure 
-%         % \ref{fig:betas_delay}, 
+        % \ref{fig:betas_delay}, 
-%         \ref{fig:parameters_ANR_by_group},
+        \ref{fig:parameters_ANR_by_group},
-%         we see that most ICD-10 categories
+        we see that most ICD-10 categories
-%         have fat tails in the $\beta$s, even among the categories 
+        have fat tails in the $\beta$s, even among the categories 
-%         relatively larger sample sizes.
+        relatively larger sample sizes.
-% \end{itemize}
+\end{itemize}
-%
+
-% Overally it is hard to escape the conclusion that more data is needed across
+Overally it is hard to escape the conclusion that more data is needed across
-% many -- if not all -- of the disease categories.
+many -- if not all -- of the disease categories.
-% At the same time, the median result is a decrease in the probability 
+At the same time, the median result is a decrease in the probability 
-% of termination when the enrollment period is held open.
+of termination when the enrollment period is held open.
-% My inclination is to believe that the overall effect is to reduce the 
+My inclination is to believe that the overall effect is to reduce the 
-% probability of termination.
+probability of termination.
 
 
 \end{document}