From fff56b52ea20682823e0fac77b54c2e04d529961 Mon Sep 17 00:00:00 2001 From: will king Date: Wed, 29 Jan 2025 18:36:17 -0800 Subject: [PATCH] Partial update to results, fixing appencicies. --- Paper/Main.tex | 8 +- Paper/sections/06_Results.tex | 305 ++++++------------ Paper/sections/11_intro_and_lit.tex | 198 +----------- .../sections/12_clinical_trial_background.tex | 164 +++++++++- Paper/sections/22_appendix_full_results.tex | 39 +++ assets/preambles/References.bib | 2 +- logs.org | 8 + todo.org | 33 +- 8 files changed, 333 insertions(+), 424 deletions(-) create mode 100644 Paper/sections/22_appendix_full_results.tex diff --git a/Paper/Main.tex b/Paper/Main.tex index b8e498c..5bfcf16 100644 --- a/Paper/Main.tex +++ b/Paper/Main.tex @@ -94,15 +94,19 @@ completion of clinical trials\\ \small{Preliminary Draft}} \printbibliography \newpage +\appendix %--------------------------------------------------------------- -\section{Appendicies} +\section{Diagnostics}\label{Appendix:Diagnostics} %--------------------------------------------------------------- \subfile{sections/21_appendix_diagnostics} +%--------------------------------------------------------------- +\section{Other Statistical Results}\label{Appendix:Results} +%--------------------------------------------------------------- +\subfile{sections/22_appendix_full_results} \newpage \tableofcontents \end{document} - % NOTES: % % diff --git a/Paper/sections/06_Results.tex b/Paper/sections/06_Results.tex index f7400d4..107175e 100644 --- a/Paper/sections/06_Results.tex +++ b/Paper/sections/06_Results.tex @@ -71,33 +71,6 @@ not represented at all. \label{FIG:barchart_idc_categories} \end{figure} -% Estimation Procedure -I fit the econometric model using mc-stan -\cite{standevelopmentteam_StanModelling_2022} -through the rstan -\cite{standevelopmentteam_RStanInterface_2023} -interface using 4 chains with -%describe -2,500 -warmup iterations and -2,500 -sampling iterations each. - -Two of the chains experienced a low -Estimated Baysian Fraction of Missing Information (E-BFMI) , -suggesting that there are some parts of the posterior distribution -that were not explored well during the model fitting. -I presume this is due to the low number of trials in some of the -ICD-10 categories. -We can see in Figure \ref{fig:barchart_idc_categories} that some of these -disease categories had a single trial represented while others were -not represented at all. - -\begin{figure}[H] - \includegraphics[width=\textwidth]{../assets/img/trials_details/CategoryCounts} - \caption{Bar chart of trials by ICD-10 categories} - \label{fig:barchart_idc_categories} -\end{figure} \subsection{Primary Results} @@ -111,7 +84,6 @@ keeping enrollment open. \begin{figure}[H] \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_boxplot} - \todo{Replace this graphic with the histdiff with boxplot} \small{ Values near 1 indicate a near perfect increase in the probability of termination. @@ -128,16 +100,14 @@ keeping enrollment open. There are a few interesting things to point out here. Let's start by getting aquainted with the details of the distribution above. +It can be devided into a few different regimes. % - spike at 0 % - the boxplot % - 63% of mass below 0 : find better way to say that % - For a random trial, there is a 63% chance that the impact is to reduce the probability of a termination. % - 2 pctg-point wide band centered on 0 has ~13% of the masss % - mean represents 9.x% increase in probability of termination. A quick simulation gives about the same pctg-point increase in terminated trials. - -A few interesting interpretation bits come out of this. % - there are 3 regimes: low impact (near zero), medium impact (concentrated in decreased probability of termination), and high impact (concentrated in increased probability of termination). -The first this that there appear to be three different regimes. The first regime consists of the low impact results, i.e. those values of $\delta_p$ near zero. About 13\% of trials lie within a single percentage point change of zero, @@ -155,71 +125,57 @@ from a case where they were highly likely to complete their primary objectives t a case where they were likely or almost certain to terminate the trial early. % - the high impact regime is strange because it consists of trials that moved from unlikely (<20% chance) of termination to a high chance (>80% chance) of termination. Something like 5% of all trials have a greater than 98 percentage point increase in termination. Not sure what this is doing. -% - Potential Explanations for high impact regime: -How could this intervention have such a wide range in the intensity -and direction of impacts? -A few explanations include that some trials are suceptable or that this is a -result of too little data. -% - Some trials are highly suceptable. This is the face value effect -One option is that some categories are more suceptable to -issues with participant enrollment. -If this is the case, we should be able to isolate categories that contribute -the most to this effect. -Another is that this might be a modelling artefact, due to the relatively -low number of trials in certain ICD-10 categories. -In short, there might be high levels of uncertanty in some parameter values, -which manifest as fat tails in the distributions of the $\beta$ parameters. -Because of the logistic format of the model, these fat tails lead to -extreme values of $p$, and potentally large changes $\delta_p$. -% - Could be uncertanty. If the model is highly uncertain, e.g. there isn't enough data, we could have a small percentage of large increases. This could be in general or just for a few categories with low amounts of data. -% - -% - - -I believe that this second explanation -- a model artifact due to uncertanty -- -is likely to be the cause. -Three points lead me to believe this: -\begin{itemize} - \item The low fractions of E-BFMI suggest that the sampler is struggling - to explore some regions of the posterior. - According to \cite{standevelopmentteam_RuntimeWarnings_2022} this is - often due to thick tails of posterior distributions. - \item When we examine the results across different ICD-10 groups, - \ref{fig:pred_dist_dif_delay2} - \todo{move figure from below} - we note this same issue. - \item In Figure \ref{fig:betas_delay}, we see that some some ICD-10 categories - \todo{add figure} - have \todo{note fat tails}. - \item There are few trials available, particularly among some specific - ICD-10 categories. -\end{itemize} -% - take a look at beta values and then discuss if that lines up with results from dist-diff by group. -% - My initial thought is that there is not enough data/too uncertain. I think this because it happens for most/all of the categories. -% - -% - -% - -Overally it is hard to escape the conclusion that more data is needed across -many -- if not all -- of the disease categories. - - - -Figure \ref{fig:pred_dist_dif_delay2} shows how this overall -result comes from different disease categories. +Based on the boxplot below, there are a couple of things to note. +First, the median effect is a 2.3 percentage point decrease +in the probability of termination. +Second, for a random selction from our trials, +there is a 63\% chance that the impact is to +reduce the probability of a termination. +Third, about 13\% of the probability mass is contained within the interval +[-0.1,0.1]. +Finally, the mean effect is measured as a 9.6 percentage point increase in +the probability of termination. +The full percentile table can be found in +\ref{TABLE:PercentilesOfDistributionOfDifferences} +in appendix +\ref{Appendix:Results} + +% Looking at the spike around zero, we find that 13.09% of the probability mass +% is contained within the band from [-1,1]. +% Additionally, there was 33.4282738% of the probability above that +% – representing those with a general increase in the +% probability of termination – and 53.4817262% of the probability mass +% below the band – representing a decrease in the probability of termination. +% On average, if you keep the trial open instead of closing it, 0.6337363% of +% trials will see a decrease in the probability of termination, but, due to +% the high increase in probability of termination given termination was +% increased, the mean probability of termination increases by 0.0964726. + + +% Pulled the data from the report +% ```{r} +% summary(pddf_ib$value) +% Min. 1st Qu. Median Mean 3rd Qu. Max. +% -0.99850 -0.12919 -0.02259 0.09647 0.14531 1.00000 +% quants <- quantile(pddf_ib$value, probs = seq(0,1,0.05), type=4) +% # Convert to a data frame +% quant_df <- data.frame( Percentile = names(quants), Value = quants ) +% kable(quant_df) +% Percentile Value +% SEE TABLE IN APPENDIX +%``` + +Figure \ref{fig:pred_dist_dif_delay2} shows how the different disease categories +tend to have a similar results: \begin{figure}[H] \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_by_group} \caption{Distribution of Predicted differences by Disease Group} \label{fig:pred_dist_dif_delay2} \end{figure} +Again, note the high mass near zero, the general decrease in the probability +of termination, and then the strong upper tails. - -\subsection{Secondary Results} - -% Examine beta parameters -% - Little movement except where data is strong, general negative movement. Still really wide -% - Note how they all learned (partial pooling) reduction in \beta from ANR? -% - Need to discuss the 5 different states. Can't remember which one is dropped for the life of me. May need to fix parameterization. -% - - +Continuing to the $\beta$ parameters, \begin{figure}[H] \includegraphics[width=\textwidth]{../assets/img/betas/parameter_across_groups/parameters_12_status_ANR} \caption{Distribution of parameters associated with ``Active, not recruiting'' status, by ICD-10 Category} @@ -227,147 +183,66 @@ result comes from different disease categories. \end{figure} % - -\subsection{Primary Results} - -The primary, causally-identified value we can estimate is the change in -the probability of termination caused by (counterfactually) keeping enrollment -open instead of closing enrollment when observed. -In figure \ref{fig:pred_dist_diff_delay} below, we see this impact of -keeping enrollment open. - - -\begin{figure}[H] - \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_boxplot} - \small{ - Values near 1 indicate a near perfect increase in the probability - of termination. - Values near 0 indicate little change in probability, - while values near -1, represent a decrease in the probability - of termination. - The scale is in probability points, thus a value near 1 is a change - from unlikely to terminate under control, to highly likely to - terminate. - } - \caption{Histogram of the Distribution of Predicted Differences} - \label{fig:pred_dist_diff_delay} -\end{figure} - -There are a few interesting things to point out here. -Let's start by getting aquainted with the details of the distribution above. -% - spike at 0 -% - the boxplot -% - 63% of mass below 0 : find better way to say that -% - For a random trial, there is a 63% chance that the impact is to reduce the probability of a termination. -% - 2 pctg-point wide band centered on 0 has ~13% of the masss -% - mean represents 9.x% increase in probability of termination. A quick simulation gives about the same pctg-point increase in terminated trials. - -A few interesting interpretation bits come out of this. -% - there are 3 regimes: low impact (near zero), medium impact (concentrated in decreased probability of termination), and high impact (concentrated in increased probability of termination). -The first this that there appear to be three different regimes. -The first regime consists of the low impact results, i.e. those values of $\delta_p$ -near zero. -About 13\% of trials lie within a single percentage point change of zero, -suggesting that there is a reasonable chance that delaying -a close of enrollment has no impact. -The second regime consists of the moderate impact on clinical trials' -probabilities of termination, say values in the interval $[-0.5, 0.5]$ -on the graph. -Most of this probability mass is represents a decrease in the probability of -a termination, some of it rather large. -Finally, there exists the high impact region, almost exclusively concentrated -around increases in the probability of termination at $\delta_p > 0.75$. -These represent cases where delaying the close of enrollemnt changes a trial -from a case where they were highly likely to complete their primary objectives to -a case where they were likely or almost certain to terminate the trial early. -% - the high impact regime is strange because it consists of trials that moved from unlikely (<20% chance) of termination to a high chance (>80% chance) of termination. Something like 5% of all trials have a greater than 98 percentage point increase in termination. Not sure what this is doing. +Finally, in figure \ref{fig:parameters_ANR_by_group}, we can see the estimated distributions of the $\beta$ parameter for +the status: \textbf{Active, not recruiting}. +The prior distributions were centered on zero, but we can see that the pooled learning has moved the mean +values negative, representing reductions in the probability of termination across the board. +This decrease in the probability of termination is strongest in the categories of Neoplasms ($n=49$), +Musculoskeletal diseases ($n=17$), and Infections and Parasites ($n=20$), the three categories with the most data. +As this is a comparison against the trial status XXX, we note that +\todo{The natural comparison I want to make is against the Recruting status. Do I want to redo this so that I can read that directly?It shouldn't affect the $\delta_p$ analysis, but this could probably use it. YES, THIS UPDATE NEEDS TO HAPPEN. The base needs to be ``active not recruiting.''} +Overall, this suggests that extending a clinical trial's enrollment period will reduce the probability of termination. % - Potential Explanations for high impact regime: -How could this intervention have such a wide range in the intensity -and direction of impacts? -A few explanations include that some trials are suceptable or that this is a -result of too little data. +This leads to the question: +``How could this intervention have such a wide range in the intensity +and direction of impacts?'' +The most likely explanations in my mind are that either +some trials are highly suceptable to enrollment struggles or that this is a +modelling artifact. % - Some trials are highly suceptable. This is the face value effect -One option is that some categories are more suceptable to -issues with participant enrollment. -If this is the case, we should be able to isolate categories that contribute -the most to this effect. -Another is that this might be a modelling artefact, due to the relatively -low number of trials in certain ICD-10 categories. +The first option -- that some categories are more suceptable to +issues with participant enrollment -- should allow us to +isolate categories or trials that contribute the most to this effect. +In figure +\ref{fig:pred_dist_dif_delay2}, it appears that most of the trials have +this high impact regime at $\delta_p > 0.75$. + +Another explanation is that this is a modelling artefact due to priors +with strong tails and the relatively low number of trials in +each ICD-10 categories. In short, there might be high levels of uncertanty in some parameter values, which manifest as fat tails in the distributions of the $\beta$ parameters. Because of the logistic format of the model, these fat tails lead to extreme values of $p$, and potentally large changes $\delta_p$. -% - Could be uncertanty. If the model is highly uncertain, e.g. there isn't enough data, we could have a small percentage of large increases. This could be in general or just for a few categories with low amounts of data. -% - -% - - I believe that this second explanation -- a model artifact due to uncertanty -- is likely to be the cause. -Three points lead me to believe this: +A few things lead me to believe this: \begin{itemize} \item The low fractions of E-BFMI suggest that the sampler is struggling to explore some regions of the posterior. - According to - \authorcite{standevelopmentteam_runtimewarningsconvergence_2022} - this is - often due to thick tails of posterior distributions. - \item When we examine the results across different ICD-10 groups, + According to \cite{standevelopmentteam_RuntimeWarnings_2022} this is + often due to thick tails of posterior distributions. + During earlier analysis, when I had about 100 trials, the number of + warnings was significantly higher. + \item When we examine the results across different ICD-10 category, \ref{fig:pred_dist_dif_delay2} - we note this same issue. - \item In Figure \ref{fig:parameters_ANR_by_group}, we see that some - ICD-10 categories have - \todo{note fat tails}. - \item There are few trials available, particularly among some specific - ICD-10 categories. - \todo{refer to figure ??} -\end{itemize} -\todo{Reformat so this refers to the original discussion of issues better.} -% - take a look at beta values and then discuss if that lines up with results from dist-diff by group. -% - My initial thought is that there is not enough data/too uncertain. I think this because it happens for most/all of the categories. -% - -% - -% - - -We can examine the per-group distributions of differences in \ref{fig:pred_dist_dif_delay2} to -acertain that the high impact group does exist in each of the groups. -This lends credence to the idea that this is a modelling issue, potentially -due to the low amounts of data overall. - - -Figure \ref{fig:pred_dist_dif_delay2} shows how this overall -result comes from different disease categories. -\begin{figure} - \includegraphics[width=\textwidth]{../assets/img/dist_diff_analysis/p_delay_intervention_distdiff_by_group} - \caption{Distribution of Predicted differences by Disease Group} - \label{fig:pred_dist_dif_delay2} -\end{figure} - - + we note that most categories have the same upper tail spike. + \item In Figure + % \ref{fig:betas_delay}, + \ref{fig:parameters_ANR_by_group}, + we see that most ICD-10 categories + have fat tails in the $\beta$s, even among the categories + relatively larger sample sizes. + -% Examine beta parameters -% - Little movement except where data is strong, general negative movement. Still really wide -% - Note how they all learned (partial pooling) reduction in \beta from ANR? -% - Need to discuss the 5 different states. Can't remember which one is dropped for the life of me. May need to fix parameterization. -% - -Finally, in figure \ref{fig:parameters_ANR_by_group}, we can see the estimated distributions of the $\beta$ parameter for -the status: \textbf{Active, not recruiting}. -The prior distributions were centered on zero, but we can see that the pooled learning has moved the mean -values negative, representing reductions in the probability of termination across the board. -This decrease in the probability of termination is strongest in the categories of Neoplasms ($n=$), -Musculoskeletal diseases ($n=$), and Infections and Parasites ($n=$), the three categories with the most data. -As this is a comparison against the trial status XXX, we note that -\todo{The natural comparison I want to make is against the Recruting status. Do I want to redo this so that I can read that directly?It shouldn't affect the $\delta_p$ analysis, but this could probably use it.} -Overall, this suggests that extending a clinical trial's enrollment period will reduce the probability of termination. - -\begin{figure}[H] - \includegraphics[width=\textwidth]{../assets/img/betas/parameter_across_groups/parameters_12_status_ANR} - \caption{Distribution of parameters associated with ``Active, not recruiting'' status, by ICD-10 Category} - \label{fig:parameters_ANR_by_group} -\end{figure} -% - +\end{itemize} -Overall it is hard to escape the conclusion that more data is needed across +Overally it is hard to escape the conclusion that more data is needed across many -- if not all -- of the disease categories. +At the same time, the median result is a decrease in the probability +of termination when the enrollment period is held open. + \end{document} diff --git a/Paper/sections/11_intro_and_lit.tex b/Paper/sections/11_intro_and_lit.tex index 1bb009f..ac78aae 100644 --- a/Paper/sections/11_intro_and_lit.tex +++ b/Paper/sections/11_intro_and_lit.tex @@ -31,7 +31,8 @@ one form of operational failure in Phase III clinical trials. Using a novel dataset constructed from administrative data registered on ClinicalTrials.gov, I exploit variation in enrollment timing and market -conditions to identify how extending the enrollment period affects trial completion. +conditions to identify how extending the enrollment period +affects trial completion. Specifically, I answer the question: \textit{ ``How does the probability of trial termination change @@ -43,199 +44,18 @@ pipeline and progression between clinical trial phases. -% In 1938 President Franklin D Rosevelt signed the Food, Drug, and Cosmetic Act, -% granting the Food and Drug Administration (FDA) authority to require -% pre-market approval of pharmaceuticals. -% \cite{commissioner_milestonesusfood_2023} -% As of Sept 2022 \todo{Check Date} they have approved 6,602 currently-marketed -% compounds with Structured Product Labels (SPLs) -% and 10,983 previously-marketed SPLs -% \cite{commissioner_nsde_2024}, -% %from nsde table. Get number of unique application_nubmers_or_citations with most recent end date as null. -% In 1999, they began requiring that drug developers register and -% publish clinical trials on \url{https://clinicaltrials.gov}. -% This provides a public mechanism where clinical trial sponsors are -% responsible to explain what they are trying to acheive and how it will be -% measured, as well as provide the public the ability to search and find trials -% that they might enroll in. -% Multiple derived datasets such as the Cortellis Investigational Drugs dataset -% or the AACT dataset from the Clinical Trials Transformation Intiative -% integrate these data. -% This brings up a question: -% Can we use this public data on clinical trials to identify what effects the -% success or failure of trials? -% In this work, I use updates to records on -% \url{https://ClinicalTrials.gov} -% to do exactly that, disentangle the effect of participant enrollment -% and competing drugs on the market affect the success or failure of -% clinical trials. - -\subsection{Background} - -%Describe how clinical trials fit into the drug development landscape and how they proceed -Clinical trials are a required part of drug development. -Not only does the FDA require that a series of clinical trials demonstrate sufficient safety and efficacy of -a novel pharmaceutical compound or device, producers of derivative medicines may be required to ensure that -their generic small molecule compound -- such as ibuprofen or levothyroxine -- matches the -performance of the originator drug if delivery or dosage is changed. -For large molecule generics (termed biosimilars) such as Adalimumab -(Brand name Humira, with biosimilars Abrilada, Amjevita, Cyltezo, Hadlima, Hulio, -Hyrimoz, Idacio, Simlandi, Yuflyma, and Yusimry), -the biosimilars are required to prove they have similar efficacy and safety to the -reference drug. - -%TODO? Decide whether to include this or not -%When registering these clinical trials -% discuss how these are registered and what data is published. -% Include image and discuss stages -% Discuss challenges faced - -% Introduce my work - -In the world of drug development, these trials are classified into different -phases of development\footnote{ -\cite{anderson_fdadrugapproval_2022} -provide an overview of this process -while -\cite{commissioner_drugdevelopmentprocess_2020} -describes the process in detail.}. -Pre-clinical studies primarily establish toxicity and potential dosing levels. -% \cite{commissioner_drugdevelopmentprocess_2020}. -Phase I trials are the first attempt to evaluate safety and efficacy in humans. -Participants typically are healthy individuals, and they measure how the drug -affects healthy bodies, potential side effects, and adjust dosing levels. -Sample sizes are often less than 100 participants. -% \cite{commissioner_drugdevelopmentprocess_2020}. -Phase II trials typically involve a few hundred participants and is where -investigators will dial in dosing, research methods, and safety. -% \cite{commissioner_drugdevelopmentprocess_2020}. -A Phase III trial is the final trial before approval by the FDA, and is where -the investigator must demonstrate safety and efficacy with a large number of -participants, usually on the order of hundreds or thousands. -% \cite{commissioner_drugdevelopmentprocess_2020}. -Occasionally, a trial will be a multi-phase trial, covering aspects of either -Phases I and II or Phases II and III. -After a successful Phase III trial, the sponsor will decide whether or not -to submit an application for approval from the FDA. -Before filing this application, the developer must have completed -``two large, controlled clinical trials.'' -% \cite{commissioner_drugdevelopmentprocess_2020}. -Phase IV trials are used after the drug has received marketing approval to -validate safety and efficacy in the general populace. -Throughout this whole process, the FDA is available to assist in decision-making -regarding topics such as study design, document review, and whether -they should terminate the trial. -The FDA also reserves the right to place a hold on the clinical trial for -safety or other operational concerns, although this is rare. -\cite{commissioner_drugdevelopmentprocess_2020}. - - -In the economics literature, most of the focus has been on describing how -drug candidates transition between different phases and their probability -of final approval. -% Lead into lit review -% Abrantes-Metz, Adams, Metz (2004) -\authorcite{abrantes-metz_pharmaceuticaldevelopmentphases_2004} -described the relationship between -various drug characteristics and how the drug progressed through clinical trials. -% This descriptive estimate was notable for using a -% mixed state proportional hazard model and estimating the impact of -% observed characteristics in each of the three phases. -They found that as Phase I and II trials last longer, -the rate of failure increases. -In contrast, Phase 3 trials generally have a higher rate of -success than failure after 91 months. -This may be due to the fact that the purpose of Phases I and II are different -from the purpose of Phase III. - -Continuing on this theme, -%DiMasi FeldmanSeckler Wilson 2009 -\authorcite{dimasi_trendsrisksassociated_2010} -examine the completion rate of clinical drug -development and find that for the 50 largest drug producers, -approximately 19\% of their drugs under development between 1993 and 2004 -successfully moved from Phase I to receiving an New Drug Application (NDA) -or Biologics License Application (BLA). -They note a couple of changes in how drugs are developed over the years they -study, most notably that -drugs began to fail earlier in their development cycle in the -latter half of the time they studied. -They note that this may reduce the cost of new drugs by eliminating late -and costly failures in the development pipeline. - -Earlier work by -\authorcite{dimasi_valueimprovingproductivity_2002} -used data on 68 investigational drugs from 10 firms to simulate how reducing -time in development reduces the costs of developing drugs. -He estimates that reducing Phase III of clinical trials by one year would -reduce total costs by about 8.9\% and that moving 5\% of clinical trial failures -from phase III to Phase II would reduce out of pocket costs by 5.6\%. - -A key contribution to this drug development literature is the work by -\authorcite{khmelnitskaya_competitionattritiondrug_2021} -who created a causal identification strategy -to disentangle strategic exits from exits due to clinical failures -in the drug development pipeline. -She found that overall 8.4\% of all pipeline exits are due to strategic -terminations and that the rate of new drug production would be about 23\% -higher if those strategic terminatations were eliminated. - -The work that is closest to mine is the work by -\authorcite{hwang_failureinvestigationaldrugs_2016} -who investigated causes for which late stage (Phase III) -clinical trials fail -- with a focus on trials in the USA, -Europe, Japan, Canada, and Australia. -They identified 640 novel therapies and then studied each therapy's -development history, as outlined in commercial datasets. -They found that for late stage trials that did not go on to receive approval, -57\% failed on efficacy grounds, 17\% failed on safety grounds, and 22\% failed -on commercial or other grounds. - -Unfortunately the work of both -\authorcite{hwang_failureinvestigationaldrugs_2016} -and -\authorcite{khmelnitskaya_competitionattritiondrug_2021} -ignore a potentially large cause of failures: operational challenges, i.e. when -issues running or funding the trial cause it to fail before achieving its -primary objective. -In a personal review of 199 randomly selected clinical trials which terminated -before achieving their primary objective, -I found that -14.5\% cited safety or efficacy concerns, -9.1\% cited funding problems (an operational concern), -and -31\% cited enrollment issues (a separate operational concern)\footnote{ -Note that these figures differ from -\authorcite{hwang_failureinvestigationaldrugs_2016} -because I sampled from all stages of trials, not just Phase III trials -focused on drug development. -}. - - - - -The main contribution of this work is the model I develop to separate -the causal effects of -market conditions (a strategic concern) from the effects of -participant enrollment (an operational concern) on Phase III Clinical trials. -This allows me to answer the question posed earlier: -\textit{ - ``How does the probability of trial termination change - when the enrollment period is extended?'' -} -using administrative data. - To understand how I do this, we'll cover some background information on -clinical trials and the administrative data I collected in section -\ref{SEC:ClinicalTrials}, -explain the approach to causal identification, the required data, -and describe how the data used matches these requirements in section +clinical trials, the current literature, +and the administrative data I collected in section +\ref{SEC:ClinicalTrials}. +Then I'll +explain the approach to causal identification and how the data collected +matches those results, \ref{SEC:CausalAndData}. Then we'll cover the econometric model (section \ref{SEC:EconometricModel}) -and results (section -\ref{SEC:Results}). +and results (section \ref{SEC:Results}). Finally, we acknowledge deficiencies in the analysis and potential improvements in section \ref{SEC:Improvements}, diff --git a/Paper/sections/12_clinical_trial_background.tex b/Paper/sections/12_clinical_trial_background.tex index 7adffde..c3b12d5 100644 --- a/Paper/sections/12_clinical_trial_background.tex +++ b/Paper/sections/12_clinical_trial_background.tex @@ -92,6 +92,7 @@ or termination. Termination occurs after enrollment has begun but before achieving the primary objective. + Understanding why trials terminate early is the key goal of this work, but is not straightforward. Terminated trials typically record a @@ -109,7 +110,8 @@ led to the termination, leaving us to use another way to infer the relative impact of operational difficulties. -To better descrobe termination causes, I suggest classifying them into +\todo{move the following} +To better describe termination causes, I suggest classifying them into three broad categories. The first category, Safety or Efficacy concerns, occurs when data suggests the treatment is unsafe or unlikely to achieve its therapeutic goals. @@ -127,7 +129,152 @@ These latter two categories represent true failures of the trial process, as they prevent us from learning whether the treatment would have been safe and effective. -\subsection{Data Summary} + +\subsection{Literature on Clinical Trials}\label{SEC:LitReview} + +%Describe how clinical trials fit into the drug development landscape and how they proceed +Clinical trials are a required part of drug development. +Not only does the FDA require that a series of clinical trials demonstrate sufficient safety and efficacy of +a novel pharmaceutical compound or device, producers of derivative medicines may be required to ensure that +their generic small molecule compound -- such as ibuprofen or levothyroxine -- matches the +performance of the originator drug if delivery or dosage is changed. +For large molecule generics (termed biosimilars) such as Adalimumab +(Brand name Humira, with biosimilars Abrilada, Amjevita, Cyltezo, Hadlima, Hulio, +Hyrimoz, Idacio, Simlandi, Yuflyma, and Yusimry), +the biosimilars are required to prove they have similar efficacy and safety to the +reference drug. + + +In the world of drug development, these trials are classified into different +phases of development\footnote{ +\cite{anderson_fdadrugapproval_2022} +provide an overview of this process +while +\cite{commissioner_drugdevelopmentprocess_2020} +describes the process in detail.}. +Pre-clinical studies primarily establish toxicity and potential dosing levels. +% \cite{commissioner_drugdevelopmentprocess_2020}. +Phase I trials are the first attempt to evaluate safety and efficacy in humans. +Participants typically are healthy individuals, and they measure how the drug +affects healthy bodies, potential side effects, and adjust dosing levels. +Sample sizes are often less than 100 participants. +% \cite{commissioner_drugdevelopmentprocess_2020}. +Phase II trials typically involve a few hundred participants and is where +investigators will dial in dosing, research methods, and safety. +% \cite{commissioner_drugdevelopmentprocess_2020}. +A Phase III trial is the final trial before approval by the FDA, and is where +the investigator must demonstrate safety and efficacy with a large number of +participants, usually on the order of hundreds or thousands. +% \cite{commissioner_drugdevelopmentprocess_2020}. +Occasionally, a trial will be a multi-phase trial, covering aspects of either +Phases I and II or Phases II and III. +After a successful Phase III trial, the sponsor will decide whether or not +to submit an application for approval from the FDA. +Before filing this application, the developer must have completed +``two large, controlled clinical trials.'' +% \cite{commissioner_drugdevelopmentprocess_2020}. +Phase IV trials are used after the drug has received marketing approval to +validate safety and efficacy in the general populace. +Throughout this whole process, the FDA is available to assist in decision-making +regarding topics such as study design, document review, and whether +they should terminate the trial. +The FDA also reserves the right to place a hold on the clinical trial for +safety or other operational concerns, although this is rare. +\cite{commissioner_drugdevelopmentprocess_2020}. + + +In the economics literature, most of the focus has been on describing how +drug candidates transition between different phases and their probability +of final approval. +% Lead into lit review +% Abrantes-Metz, Adams, Metz (2004) +\authorcite{abrantes-metz_pharmaceuticaldevelopmentphases_2004} +described the relationship between +various drug characteristics and how the drug progressed through clinical trials. +% This descriptive estimate was notable for using a +% mixed state proportional hazard model and estimating the impact of +% observed characteristics in each of the three phases. +They found that as Phase I and II trials last longer, +the rate of failure increases. +In contrast, Phase 3 trials generally have a higher rate of +success than failure after 91 months. +This may be due to the fact that the purpose of Phases I and II are different +from the purpose of Phase III. + +Continuing on this theme, +%DiMasi FeldmanSeckler Wilson 2009 +\authorcite{dimasi_trendsrisksassociated_2010} +examine the completion rate of clinical drug +development and find that for the 50 largest drug producers, +approximately 19\% of their drugs under development between 1993 and 2004 +successfully moved from Phase I to receiving an New Drug Application (NDA) +or Biologics License Application (BLA). +They note a couple of changes in how drugs are developed over the years they +study, most notably that +drugs began to fail earlier in their development cycle in the +latter half of the time they studied. +They note that this may reduce the cost of new drugs by eliminating late +and costly failures in the development pipeline. + +Earlier work by +\authorcite{dimasi_valueimprovingproductivity_2002} +used data on 68 investigational drugs from 10 firms to simulate how reducing +time in development reduces the costs of developing drugs. +He estimates that reducing Phase III of clinical trials by one year would +reduce total costs by about 8.9\% and that moving 5\% of clinical trial failures +from phase III to Phase II would reduce out of pocket costs by 5.6\%. + +A key contribution to this drug development literature is the work by +\authorcite{khmelnitskaya_competitionattritiondrug_2021} +who created a causal identification strategy +to disentangle strategic exits from exits due to clinical failures +in the drug development pipeline. +She found that overall 8.4\% of all pipeline exits are due to strategic +terminations and that the rate of new drug production would be about 23\% +higher if those strategic terminatations were eliminated. + +The work that is closest to mine is the work by +\authorcite{hwang_failureinvestigationaldrugs_2016} +who investigated causes for which late stage (Phase III) +clinical trials fail -- with a focus on trials in the USA, +Europe, Japan, Canada, and Australia. +They identified 640 novel therapies and then studied each therapy's +development history, as outlined in commercial datasets. +They found that for late stage trials that did not go on to receive approval, +57\% failed on efficacy grounds, 17\% failed on safety grounds, and 22\% failed +on commercial or other grounds. + +Unfortunately the work of both +\authorcite{hwang_failureinvestigationaldrugs_2016} +and +\authorcite{khmelnitskaya_competitionattritiondrug_2021} +ignore a potentially large cause of failures: operational challenges, i.e. when +issues running or funding the trial cause it to fail before achieving its +primary objective. +In a personal review of 199 randomly selected clinical trials which terminated +before achieving their primary objective, +I found that +14.5\% cited safety or efficacy concerns, +9.1\% cited funding problems (an operational concern), +and +31\% cited enrollment issues (a separate operational concern)\footnote{ +Note that these figures differ from +\authorcite{hwang_failureinvestigationaldrugs_2016} +because I sampled from all stages of trials, not just Phase III trials +focused on drug development. +}. + + + + + + + + +\subsection{Introduction to \href{https://ClinicalTrials.gov}{ClinicalTrials.Gov}} + + + %% Describe data here Since Sep 27th, 2007 those who conduct clinical trials of FDA controlled drugs or devices on human subjects must register @@ -176,13 +323,18 @@ information about the past state of trials. I combined these two sources, using the AACT dataset to select trials of interest and then scraping \url{ClinicalTrials.gov} to get a timeline of each trial. +The result is a series of snapshots, each documenting a specific set of +recorded changes in a trial. +It is these snapshots that provide the opportunity to estimate the +data generating process corresponding to the clinical trials for +which I have data. %%%%%%%%%%%%%%%%%%%%%%%% Model Outline -The way I use this data is to predict the final status of the trial -from the snapshots that were taken, in effect asking: -``how does the probability of a termination change from the current state -of the trial if X changes?'' +% The way I use this data is to predict the final status of the trial +% from the snapshots that were taken, in effect asking: +% ``how does the probability of a termination change from the current state +% of the trial if X changes?'' % - % - % - diff --git a/Paper/sections/22_appendix_full_results.tex b/Paper/sections/22_appendix_full_results.tex new file mode 100644 index 0000000..647bda3 --- /dev/null +++ b/Paper/sections/22_appendix_full_results.tex @@ -0,0 +1,39 @@ +\documentclass[../Main.tex]{subfiles} +\graphicspath{{\subfix{Assets/img/}}} + +\begin{document} + + +\begin{center} + \label{TABLE:PercentilesOfDistributionOfDifferences} + % \caption{Table of Percentiles of Distribution of Differences} + \begin{tabular}{cc} + \hline + Percentile & Value \\ + \hline + 0\% & -0.9985020 \\ + 5\% & -0.3763454 \\ + 10\% & -0.2639654 \\ + 15\% & -0.2053399 \\ + 20\% & -0.1628793 \\ + 25\% & -0.1291890 \\ + 30\% & -0.0980523 \\ + 35\% & -0.0734082 \\ + 40\% & -0.0547123 \\ + 45\% & -0.0385514 \\ + 50\% & -0.0225949 \\ + 55\% & -0.0045955 \\ + 60\% & -0.0000394 \\ + 65\% & 0.0010549 \\ + 70\% & 0.0509626 \\ + 75\% & 0.1453046 \\ + 80\% & 0.3425234 \\ + 85\% & 0.7084837 \\ + 90\% & 0.9250351 \\ + 95\% & 0.9820456 \\ + 100\% & 1.0000000 \\ + \hline + \end{tabular} +\end{center} + +\end{document} diff --git a/assets/preambles/References.bib b/assets/preambles/References.bib index 001b2f7..eb48171 100644 --- a/assets/preambles/References.bib +++ b/assets/preambles/References.bib @@ -5355,7 +5355,7 @@ California 90401-3208}, file = {/home/will/Zotero/storage/KAHW2ABD/Indexing-SPL-Fact-Sheet.pdf} } -@online{usnlm_fdaaa800finalrule, +@online{usnlm_fdaaa801finalrule, type = {Government}, title = {{{FDAAA}} 801 and the {{Final Rule}} - {{ClinicalTrials}}.Gov}, author = {{U.S. National Library of Medicine}}, diff --git a/logs.org b/logs.org index c0490ce..aab020c 100644 --- a/logs.org +++ b/logs.org @@ -5,3 +5,11 @@ Need to decide whether or not to include this set of sentences. **** [2025-01-18 Sat 11:58] [[[[file:/home/will/research/phd_deliverables/JobMarketPaper/Paper/sections/11_intro_and_lit.tex::45]]]] decide whether to include these details here +** 2025-W05 +*** 2025-01-29 Wednesday +**** [2025-01-29 Wed 10:12] Summary of yesterday, thoughts for today + Yesterday I got my draft mostly done. I rearranged the causal inference section + fixed some references, etc. + Today I want to remove a bunch of todos, read it backwards to fix things, + and get it sent to Tom. + I'll also run it by claude.ai. diff --git a/todo.org b/todo.org index c76e492..b2d3c38 100644 --- a/todo.org +++ b/todo.org @@ -4,19 +4,19 @@ **** DONE Push work to overleaf DEADLINE: <2025-01-15 Wed> CLOSED: [2025-01-20 Mon 11:46] *** 2025-01-17 Friday -**** TODO Redo analysis using "Recruitng" as the base status - - The goal is to get the $\beta$'s for active, not recruitng. -**** TODO Fix JMP based on Tom's Suggestions and send to committee -***** TODO Get references working properly +**** DONE Fix JMP based on Tom's Suggestions and send to committee + CLOSED: [2025-01-29 Wed 10:11] +***** DONE Get references working properly + CLOSED: [2025-01-29 Wed 09:58] - setup author date format - fix references, add to Overleaf version -***** TODO Read Backward - Identify poorly written portions (incomplete sentences and paragraphs) and what I was trying to communicate. -***** TODO fix issues +***** DONE fix issues + CLOSED: [2025-01-29 Wed 09:58] *** 2025-01-18 Saturday -**** TODO Decide if this section needs added +**** DONE Decide if this section needs added + CLOSED: [2025-01-29 Wed 09:58] [[[[file:/home/will/research/phd_deliverables/JobMarketPaper/Paper/sections/11_intro_and_lit.tex::45]]]] + nope **** RECINDED Update citations in lit review section. CLOSED: [2025-01-20 Mon 11:47] [[[[file:/home/will/research/phd_deliverables/JobMarketPaper/Paper/sections/05_LitReview.tex::25]]]] @@ -31,8 +31,19 @@ Realized that this was readded by mistake. I integrated lit review into intro in 11 ** 2025-W04 *** 2025-01-20 Monday -**** TODO get a citation for the AACT project +**** DONE get a citation for the AACT project + CLOSED: [2025-01-29 Wed 09:55] [[[[file:/home/will/research/phd_deliverables/JobMarketPaper/Paper/sections/10_CausalStory.tex::114]]]] *** 2025-01-23 Thursday -**** TODO Pickup citation fixes here +**** DONE Pickup citation fixes here + CLOSED: [2025-01-29 Wed 09:55] [[[[file:/home/will/research/phd_deliverables/JobMarketPaper/Paper/sections/06_Results.tex::174]]]] +** 2025-W05 +*** 2025-01-29 Wednesday +**** TODO Review JMP, list areas that need rewritten. +***** TODO Read Backward + Identify poorly written portions (incomplete sentences and paragraphs) and what I was trying to communicate. + +**** TODO Redo analysis using "Recruitng" as the base status + + The goal is to get the $\beta$'s for active, not recruitng.