JobMarketPaper/Paper/sections/06_Results.tex

\documentclass[../Main.tex]{subfiles}

\begin{document}

In this section 
I describe the model fitting, the posteriors of the parameters of interest,
and intepret the results.


\subsection{Data Summaries and Estimation Procedure}

% Data Summaries
Overall, I successfully processed 162 trials, with 1,347 snapshots between them.
Figure \ref{fig:snapshot_counts} shows the histogram of snapshots per trial.
Most trials lasted less than 1,500 days, as can be seen in 
\ref{fig:trial_durations}. 
Although there are a large number of snapshots that will be used to fit the 
model, the number of trials -- the unit of observation -- are quite low. 
Add to the  fact that these are spread over multiple ICD-10 categories
and the overall quantity of trials is quite low. 

To continue, we can use a scatterplot to get a rough idea of the observed
relationship between the number of snapshots and the duration of trials. 
We can see this in Figure \ref{fig:snapshot_duration_scatter}, where
the correlation (measured at $0.34$) is apparent.


\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/trials_details/HistTrialDurations_Faceted}
    \caption{Histograms of Trial Durations}
    \label{fig:trial_durations}
\end{figure}

\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/trials_details/HistSnapshots}
    \caption{Histogram of the count of Snapshots}
    \label{fig:snapshot_counts}
\end{figure}

\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/trials_details/SnapshotsVsDurationVsTermination}
    \caption{Scatterplot comparing the Count of Snapshots and Trial Duration}
    \label{fig:snapshot_counts}
\end{figure}

% Estimation Procedure
I fit the econometric model using mc-stan 
\cite{standevelopmentteam_StanModelling_2022}
through the rstan 
\cite{standevelopmentteam_RStanInterface_2023}
interface using 4 chains with 
%describe  
2,500
warmup iterations and
2,500
sampling iterations each.

Two of the chains experienced a low 
Estimated Baysian Fraction of Missing Information (E-BFMI) ,
suggesting that there are some parts of the posterior distribution
that were not explored well during the model fitting. 
I presume this is due to the low number of trials in some of the 
ICD-10 categories.
We can see in Figure \ref{fig:barchart_idc_categories} that some of these 
disease categories had a single trial represented while others were 
not represented at all.

\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/trials_details/CategoryCounts}
    \caption{Bar chart of trials by ICD-10 categories}
    \label{fig:barchart_idc_categories}
\end{figure}


\subsection{Primary Results}

The primary, causally-identified value we can estimate is the change in 
the probability of termination caused by (counterfactually) keeping enrollment
open instead of closing enrollment when observed. 
In figure \ref{fig:pred_dist_diff_delay} below, we see this impact of 
keeping enrollment open.


\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_boxplot}
    \small{
        Values near 1 indicate a near perfect increase in the probability 
        of termination. 
        Values near 0 indicate little change in probability,
        while values near -1, represent a decrease in the probability
        of termination. 
        The scale is in probability points, thus a value near 1 is a change 
        from unlikely to terminate under control, to highly likely to 
        terminate.
    }
    \caption{Histogram of the Distribution of Predicted Differences}
    \label{fig:pred_dist_diff_delay}
\end{figure}

There are a few interesting things to point out here. 
Let's start by getting aquainted with the details of the distribution above.
% - spike at 0
% - the boxplot
% - 63% of mass below 0 : find better way to say that
%   - For a random trial, there is a 63% chance that the impact is to reduce the probability of a termination.
% - 2 pctg-point wide band centered on 0 has ~13% of the masss
% - mean represents 9.x% increase in probability of termination. A quick simulation gives about the same pctg-point increase in terminated trials.

A few interesting interpretation bits come out of this.
% - there are 3 regimes: low impact (near zero), medium impact (concentrated in decreased probability of termination), and high impact (concentrated in increased probability of termination). 
The first this that there appear to be three different regimes. 
The first regime consists of the low impact results, i.e. those values of $\delta_p$ 
near zero. 
About 13\% of trials lie within a single percentage point change of zero, 
suggesting that there is a reasonable chance that delaying 
a close of enrollment has no impact. 
The second regime consists of the moderate impact on clinical trials'
probabilities of termination, say values in the interval $[-0.5, 0.5]$ 
on the graph.
Most of this probability mass is represents a decrease in the probability of 
a termination, some of it rather large.
Finally, there exists the high impact region, almost exclusively concentrated 
around increases in the probability of termination at $\delta_p > 0.75$. 
These represent cases where delaying the close of enrollemnt changes a trial
from a case where they were highly likely to complete their primary objectives to 
a case where they were likely or almost certain to terminate the trial early.
%   - the high impact regime is strange because it consists of trials that moved from unlikely (<20% chance) of termination to a high chance (>80% chance) of termination. Something like 5% of all trials have a greater than 98 percentage point increase in termination. Not sure what this is doing. 

%   - Potential Explanations for high impact regime:
How could this intervention have such a wide range in the intensity 
and direction of impacts?
A few explanations include that some trials are suceptable or that this is a 
result of too little data.
%       - Some trials are highly suceptable. This is the face value effect
One option is that some categories are more suceptable to 
issues with participant enrollment. 
If this is the case, we should be able to isolate categories that contribute
the most to this effect.
Another is that this might be a modelling artefact, due to the relatively
low number of trials in certain ICD-10 categories. 
In short, there might be high levels of uncertanty in some parameter values,
which manifest as fat tails in the distributions of the $\beta$ parameters. 
Because of the logistic format of the model, these fat tails lead to 
extreme values of $p$, and potentally large changes $\delta_p$. 
%       - Could be uncertanty. If the model is highly uncertain, e.g. there isn't enough data, we could have a small percentage of large increases. This could be in general or just for a few categories with low amounts of data.
% - 
% - 

I believe that this second explanation -- a model artifact due to uncertanty --
is likely to be the cause. 
Three points lead me to believe this:
\begin{itemize}
    \item The low fractions of E-BFMI suggest that the sampler is struggling 
        to explore some regions of the posterior. 
        According to \cite{standevelopmentteam_RuntimeWarnings_2022} this is 
        often due to thick tails of posterior distributions.
    \item When we examine the results across different ICD-10 groups, 
        \ref{fig:pred_dist_dif_delay2}
        we note this same issue.
    \item In Figure \ref{fig:parameters_ANR_by_group}, we see that some some ICD-10 categories
        \todo{add figure}
        have \todo{note fat tails}.
    \item There are few trials available, particularly among some specific 
        ICD-10 categories.
\end{itemize}
%           - take a look at beta values and then discuss if that lines up with results from dist-diff by group. 
%       - My initial thought is that there is not enough data/too uncertain. I think this because it happens for most/all of the categories.
% - 
% - 
% - 

We can examine the per-group distributions of differences in \ref{fig:pred_dist_dif_delay2} to 
acertain that the high impact group does exist in each of the groups.
This lends credence to the idea that this is a modelling issue, potentially
due to the low amounts of data overall.


Figure \ref{fig:pred_dist_dif_delay2} shows how this overall
result comes from different disease categories.
\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_by_group}
    \caption{Distribution of Predicted differences by Disease Group}
    \label{fig:pred_dist_dif_delay2}
\end{figure}



% Examine beta parameters 
% - Little movement except where data is strong, general negative movement. Still really wide 
% - Note how they all learned (partial pooling) reduction in \beta from ANR?
% - Need to discuss the 5 different states. Can't remember which one is dropped for the life of me. May need to fix parameterization.
% - 
Finally, in figure \ref{fig:parameters_ANR_by_group}, we can see the estimated distributions of the $\beta$ parameter for
the status: \textbf{Active, not recruiting}.
The prior distributions were centered on zero, but we can see that the pooled learning has moved the mean
values negative, representing reductions in the probability of termination across the board. 
This decrease in the probability of termination is strongest in the categories of Neoplasms ($n=$),
Musculoskeletal diseases ($n=$), and Infections and Parasites ($n=$), the three categories with the most data.
As this is a comparison against the trial status XXX, we note that
\todo{The natural comparison I want to make is against the Recruting status. Do I want to redo this so that I can read that directly?It shouldn't affect the $\delta_p$ analysis, but this could probably use it.}
Overall, this suggests that extending a clinical trial's enrollment period will reduce the probability of termination.

\begin{figure}[H]
    \includegraphics[width=\textwidth]{../assets/img/betas/parameter_across_groups/parameters_12_status_ANR}
    \caption{Distribution of parameters associated with ``Active, not recruiting'' status, by ICD-10 Category}
    \label{fig:parameters_ANR_by_group}
\end{figure}
% - 

Overally it is hard to escape the conclusion that more data is needed across
many -- if not all -- of the disease categories.

\end{document}