You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
247 lines
10 KiB
TeX
247 lines
10 KiB
TeX
\documentclass[../Main.tex]{subfiles}
|
|
|
|
\begin{document}
|
|
|
|
In this section
|
|
I describe the model fitting, the posteriors of the parameters of interest,
|
|
and intepret the results.
|
|
|
|
|
|
\subsection{Data Summaries and Estimation Procedure}
|
|
|
|
% Data Summaries
|
|
Overall, I successfully processed 162 trials, with 1,347 snapshots between them.
|
|
Figure \ref{fig:snapshot_counts} shows the histogram of snapshots per trial.
|
|
Most trials lasted less than 1,500 days, as can be seen in
|
|
\ref{fig:trial_durations}.
|
|
Although there are a large number of snapshots that will be used to fit the
|
|
model, the number of trials -- the unit of observation -- are quite low.
|
|
Add to the fact that these are spread over multiple IDC-10 categories
|
|
and the overall quantity of trials is quite low.
|
|
|
|
To continue, we can use a scatterplot to get a rough idea of the observed
|
|
relationship between the number of snapshots and the duration of trials.
|
|
We can see this in Figure \ref{fig:snapshot_duration_scatter}, where
|
|
the correlation (measured at $0.34$) is apparent.
|
|
|
|
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay}
|
|
\todo{Replace this graphic with the histogram of trial durations}
|
|
\caption{Histograms of Trial Durations}
|
|
\label{fig:trial_durations}
|
|
\end{figure}
|
|
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay}
|
|
\todo{Replace this graphic with the histogram of snapshots}
|
|
\caption{Histogram of the count of Snapshots}
|
|
\label{fig:snapshot_counts}
|
|
\end{figure}
|
|
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay}
|
|
\todo{Replace this graphic with the scatterplot comparing durations and snapshots}
|
|
\caption{Scatterplot comparing the Count of Snapshots and Trial Duration}
|
|
\label{fig:snapshot_counts}
|
|
\end{figure}
|
|
|
|
% Estimation Procedure
|
|
I fit the econometric model using mc-stan
|
|
\cite{standevelopmentteam_StanModelling_2022}
|
|
through the rstan
|
|
\cite{standevelopmentteam_RStanInterface_2023}
|
|
interface using 4 chains with
|
|
%describe
|
|
2,500
|
|
warmup iterations and
|
|
2,500
|
|
sampling iterations each.
|
|
|
|
Two of the chains experienced a low
|
|
Estimated Baysian Fraction of Missing Information (E-BFMI) ,
|
|
suggesting that there are some parts of the posterior distribution
|
|
that were not explored well during the model fitting.
|
|
I presume this is due to the low number of trials in some of the
|
|
IDC-10 categories.
|
|
We can see in Figure \ref{fig:barchart_idc_categories} that some of these
|
|
disease categories had a single trial represented while others were
|
|
not represented at all.
|
|
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay}
|
|
\todo{Replace this graphic with the barchart of trials by categories.}
|
|
\caption{Bar chart of trials by IDC-10 categories}
|
|
\label{fig:barchart_idc_categories}
|
|
\end{figure}
|
|
|
|
|
|
\subsection{Primary Results}
|
|
|
|
The primary, causally-identified value we can estimate is the change in
|
|
the probability of termination caused by (counterfactually) keeping enrollment
|
|
open instead of closing enrollment when observed.
|
|
In figure \ref{fig:pred_dist_diff_delay} below, we see this impact of
|
|
keeping enrollment open.
|
|
|
|
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay}
|
|
\todo{Replace this graphic with the histdiff with boxplot}
|
|
\small{
|
|
Values near 1 indicate a near perfect increase in the probability
|
|
of termination.
|
|
Values near 0 indicate little change in probability,
|
|
while values near -1, represent a decrease in the probability
|
|
of termination.
|
|
The scale is in probability points, thus a value near 1 is a change
|
|
from unlikely to terminate under control, to highly likely to
|
|
terminate.
|
|
}
|
|
\caption{Distribution of Predicted Differences}
|
|
\label{fig:pred_dist_diff_delay}
|
|
\end{figure}
|
|
|
|
There are a few interesting things to point out here.
|
|
Let's start by getting aquainted with the details of the distribution above.
|
|
% - spike at 0
|
|
% - the boxplot
|
|
% - 63% of mass below 0 : find better way to say that
|
|
% - For a random trial, there is a 63% chance that the impact is to reduce the probability of a termination.
|
|
% - 2 pctg-point wide band centered on 0 has ~13% of the masss
|
|
% - mean represents 9.x% increase in probability of termination. A quick simulation gives about the same pctg-point increase in terminated trials.
|
|
|
|
A few interesting interpretation bits come out of this.
|
|
% - there are 3 regimes: low impact (near zero), medium impact (concentrated in decreased probability of termination), and high impact (concentrated in increased probability of termination).
|
|
The first this that there appear to be three different regimes.
|
|
The first regime consists of the low impact results, i.e. those values of $\delta_p$
|
|
near zero.
|
|
About 13\% of trials lie within a single percentage point change of zero,
|
|
suggesting that there is a reasonable chance that delaying
|
|
a close of enrollment has no impact.
|
|
The second regime consists of the moderate impact on clinical trials'
|
|
probabilities of termination, say values in the interval $[-0.5, 0.5]$
|
|
on the graph.
|
|
Most of this probability mass is represents a decrease in the probability of
|
|
a termination, some of it rather large.
|
|
Finally, there exists the high impact region, almost exclusively concentrated
|
|
around increases in the probability of termination at $\delta_p > 0.75$.
|
|
These represent cases where delaying the close of enrollemnt changes a trial
|
|
from a case where they were highly likely to complete their primary objectives to
|
|
a case where they were likely or almost certain to terminate the trial early.
|
|
% - the high impact regime is strange because it consists of trials that moved from unlikely (<20% chance) of termination to a high chance (>80% chance) of termination. Something like 5% of all trials have a greater than 98 percentage point increase in termination. Not sure what this is doing.
|
|
|
|
% - Potential Explanations for high impact regime:
|
|
How could this intervention have such a wide range in the intensity
|
|
and direction of impacts?
|
|
A few explanations include that some trials are suceptable or that this is a
|
|
result of too little data.
|
|
% - Some trials are highly suceptable. This is the face value effect
|
|
One option is that some categories are more suceptable to
|
|
issues with participant enrollment.
|
|
If this is the case, we should be able to isolate categories that contribute
|
|
the most to this effect.
|
|
Another is that this might be a modelling artefact, due to the relatively
|
|
low number of trials in certain IDC-10 categories.
|
|
In short, there might be high levels of uncertanty in some parameter values,
|
|
which manifest as fat tails in the distributions of the $\beta$ parameters.
|
|
Because of the logistic format of the model, these fat tails lead to
|
|
extreme values of $p$, and potentally large changes $\delta_p$.
|
|
% - Could be uncertanty. If the model is highly uncertain, e.g. there isn't enough data, we could have a small percentage of large increases. This could be in general or just for a few categories with low amounts of data.
|
|
% -
|
|
% -
|
|
|
|
I believe that this second explanation -- a model artifact due to uncertanty --
|
|
is likely to be the cause.
|
|
Three points lead me to believe this:
|
|
\begin{itemize}
|
|
\item The low fractions of E-BFMI suggest that the sampler is struggling
|
|
to explore some regions of the posterior.
|
|
According to \cite{standevelopmentteam_RuntimeWarnings_2022} this is
|
|
often due to thick tails of posterior distributions.
|
|
\item When we examine the results across different ICD-10 groups,
|
|
\ref{fig:pred_dist_dif_delay2}
|
|
\todo{move figure from below}
|
|
we note this same issue.
|
|
\item In Figure \ref{fig:betas_delay}, we see that some some ICD-10 categories
|
|
\todo{add figure}
|
|
have \todo{note fat tails}.
|
|
\item There are few trials available, particularly among some specific
|
|
ICD-10 categories.
|
|
\end{itemize}
|
|
% NOTE: maybe change order to be ebfmi, group hist-diff or distdiff, tail width, then data size.
|
|
% - take a look at beta values and then discuss if that lines up with results from dist-diff by group.
|
|
% - My initial thought is that there is not enough data/too uncertain. I think this because it happens for most/all of the categories.
|
|
% -
|
|
% -
|
|
% -
|
|
% -
|
|
Overally it is hard to escape the result that more data is needed, across
|
|
many, if not all, of the disease categories.
|
|
|
|
|
|
|
|
% The probability mass associated with a each 10 percentage point change are in table \ref{tab:regimes}
|
|
% \begin{table}[H]
|
|
% \caption{Regimes and associated probability masses}\label{tab:regimes}
|
|
% \begin{center}
|
|
% \begin{tabular}[c]{l|l}
|
|
% \hline
|
|
% \multicolumn{1}{c|}{\textbf{Interval}} &
|
|
% \multicolumn{1}{c}{\textbf{Probability Mass}} \\
|
|
% \hline
|
|
% $[,]$ & b \\
|
|
% $[,]$ & b \\
|
|
% $[,]$ & b \\
|
|
% $[,]$ & b \\
|
|
% $[,]$ & b \\
|
|
% \hline
|
|
% \end{tabular}
|
|
% \end{center}
|
|
% \end{table}
|
|
|
|
Figure \ref{fig:pred_dist_dif_delay2} shows how this overall
|
|
result comes from different disease categories.
|
|
\begin{figure}[H]
|
|
\includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-delay-group}
|
|
\caption{Distribution of Predicted differences by Disease Group}
|
|
\label{fig:pred_dist_dif_delay2}
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
% Although it is not causally identified due to population interactions,
|
|
% we can examine the direct effect from adding a single generic competitior drug
|
|
% and how the similar result decomposes very differently.
|
|
% This is shown just as a contrast to the enrollment results.
|
|
% Figure
|
|
% \label{fig:pred_dist_diff_generic}
|
|
% shows a very similar result with roughly the same regimes,
|
|
% while
|
|
% \label{fig:pred_dist_dif_generic2}
|
|
% shows that this breakdown is different.
|
|
% \todo{
|
|
% Consider moving these to an appendix as they are
|
|
% just additions at this point.
|
|
% }
|
|
%
|
|
% \begin{figure}[H]
|
|
% \includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-generic}
|
|
% \caption{
|
|
% Distribution of Predicted Differences for one additional generic
|
|
% competitor
|
|
% }
|
|
% \label{fig:pred_dist_diff_generic}
|
|
% \end{figure}
|
|
%
|
|
% \begin{figure}[H]
|
|
% \includegraphics[width=\textwidth]{../assets/img/current/pred_dist_diff-generic-group}
|
|
% \caption{}
|
|
% \label{fig:pred_dist_dif_generic2}
|
|
% \end{figure}
|
|
%
|
|
|
|
|
|
\end{document}
|