Organizations such as the Food and Drug Agency can make both type I (approving bad drugs) and type II errors (rejecting good ones). Optimal reliability entails balancing these two kinds of mistakes just so. This paper addresses the question of whether and when an imperfectly rational agency - in particular, one using an adaptive decision-making policy - would become optimally reliable. We establish a necessary condition that an adaptive scheme must satisfy if it is to guide the agency to optimal reliability. The necessary condition takes the form of a constraint on the agency’s propensity to explore the space of alternatives: it must become ‘stoic’ in the face of errors. The paper supplements this normative analysis by providing a descriptive examination of a common class of adaptive rules that could be used by an agency coping with type I and type II errors. Here we analyze how technological factors, external political pressure, and changes in the quality of information affect the agency’s reliability, both dynamically and in the steady-state.
