Murphy Seminar in Political Science: Alexis Palmer

Each semester The Murphy Institute sponsors a series of seminars organized by the Tulane Department of Political Science that provides an opportunity for faculty, researchers, and practitioners to present their latest research and pressing issues related to topics in political economy. Research presented covers all aspects of contemporary politics science, including comparative politics, public policy, international relations, American politics, and normative theory. Papers are distributed beforehand to the participants who read the paper and prepare discussion questions for the presenter.

Alexis Palmer is an expert in computational discourse and semantics; computational linguistics for low-resource languages and language documentation; discourse structure and coherence, and modes of discourse and social analytics, including automated detection of offensive language in social media. She received her PhD from UT Austin in 2009, has held a number of prestigious post docs and research positions in Germany (including positions at the Institute for Computational Linguistics in Heidelberg and the Institut für Deutsche Sprache in Mannheim). Until her move to CU in 2021, she was an assistant professor at the University of North Texas, Denton. Dr. Palmer brings a prestigious National Science Foundation CAREER grant with her to CU. In this project, she is working on cross-linguistic methods for better development of language processing tools for low-resource languages. The project is called FOLTA (From One Language to Another). She has also recently become interested in the question of how we can make the outcomes of linguistic documentation more useable and accessible, particularly to support development of pedagogical materials for a language. 

"Replication for Language Models: Problems, Principles, and Best Practices for Political Science"  (ABSTRACT)

Large Language Models (LMs) are exciting tools: they require minimal researcher input and but make it possible to annotate and generate large quantities of data. Yet there has been almost no systematic research into the reproducibility of research using LMs. This is a potential problem for scientific integrity. We give a theoretical framework for replication in the discipline and show that LM work is perhaps uniquely problematic. We demonstrate the problem empirically using a rolling iterated replication design in which we compare crowdsourcing and LMs on multiple repeated tasks, over many months. We find that LMs can be accurate, but the observed variance in performance is often unacceptably high. Strict “temperature” control does not resolve these issues. This affects downstream results. In many cases the LM findings cannot be re-run, let alone replicated. We conclude with recommendations for best practice, including the use of locally versioned ‘open’ LMs.