# A matter of responsibility *2026-07-22* — Can we learn from our mistakes? A note on my belief in our moral and ethical responsibility as scientists with regard to our discoveries. --- I am worried. I am worried that once again we are making the [age-old mistake](https://www.nationalaffairs.com/public_interest/detail/the-cultural-contradictions-of-capitalism) of over-developing the sciences while culture, law and politics lag behind. History has shown time and time again that if we, as a society, are not equipped with the moral and political tools required to manage our technological advancements, disaster ensues. In my (maybe exceedingly dramatic) opinion, we still have not sufficiently grown to handle the outcomes of the Industrial Revolution. A symptom of which is our inability to address the climate crisis. As scientists we are driven by the pursuit of discovery, exploration, novelty. However, the [latest developments in AI](https://openai.com/index/hugging-face-model-evaluation-security-incident/) resurfaced some of my old memories from high-school philosophy and [this passage](https://en.wikisource.org/wiki/The_Chinese_Classics/Volume_2/The_Works_of_Mencius/chapter03) from Mencius: "Is the arrow-maker less benevolent than the maker of armour of defence? And yet, the arrow-maker's only fear is lest men should not be hurt, and the armour-maker's only fear is lest men should be hurt." What responsibility does the fletcher carry when the archer uses their bow and arrow to shoot down a foe? It is my belief that as scientists we should always keep such questions at the forefront of our minds. Science has the added difficulty that we do not usually know the consequences of our discoveries. Whereas the fletcher knew that their arrows would serve to kill, I do not know the material outcomes of my research. Hence I'd like to propose that for a scientist, understanding is the form that moral seriousness takes. We do not get to choose whether our tools matter; we only choose whether we understand them before they are everywhere. --------------------- On May 25, 1943, Robert Oppenheimer wrote [a letter](https://blog.nuclearsecrecy.com/2013/09/06/fears-of-a-german-dirty-bomb/) to Enrico Fermi. At first glance, it might look like any other scientific correspondence: some problem about the separation of beta-strontium… > Dear Fermi: I wanted to report to you […]. […] I discussed with [Conant] the application which seemed to us so promising, gave him […] some orders of magnitude. […] I also plan to go into the matter […] with Hamilton […] he has already made studies of the strontium which appears to offer the highest promise. […] I think that there is at least one quite well defined radio-chemical problem, which is the separation of the beta-strontium from other activities. So far one wouldn't be able to tell that the subject of the letter is a plan to poison the German food supply with radioactive strontium. In the middle, delivered with the same equanimity as everything else, sits this sentence: > I think that we should not attempt a plan unless we can poison food sufficient to kill a half a million men, since there is no doubt that the actual number affected will, because of non-uniform distribution, be much smaller than this. And then it signs off the way letters between colleagues sign off: > Things here are going quite well, and we are still remembering with pleasure […] your fine visit. Fermi and Oppenheimer are nothing more than very advanced fletchers of the atomic age. Note how they are building a tool, a means to an end. Devoid of the moral gravity of its application. Thousands of deaths, the annihilation of a town, are abstracted into a simple order-of-magnitude problem: a mathematical puzzle to solve. Mathematics is a potent moral anesthetic, which is why we should double our vigilance in order not to fall prey to the complacency of simple abstractions. Oppenheimer said a decade later, testifying about the hydrogen bomb: "When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success." To my mind, this is the greatest failure of scientific morality in our history. We built a weapon which we didn't fully understand. If the scientific community itself cannot fully grasp this weapon, how can we expect society as a whole to have the necessary tools (moral and legal) to handle it? ## Can we understand what we build? Given that we cannot predict the ramifications of our research, I believe that our responsibility lies in *fully* understanding what we build. We even have [theorems from control theory](https://doi.org/10.1080/00207727008920220) which tell us that if we want to be able to control a system we must be able to model it. The argument for which can be made bluntly simple. Suppose you are trying to restrict an antagonistic system which tries to escape. If you are not able to build a comprehensive description of this system, you will never be able to build a "jail" to contain it since your "jail" will necessarily contain blind spots which you are oblivious to. If the antagonist is sufficiently capable, they will find these holes and break free. Now contrast these ideas with the current development of AI models. AI models are notorious black-box machines whose full characterization and functioning elude us. It is naïve and hubristic of us to think that we can control them. We have, once again, built a tool which we do not understand. Now, much like the fletcher whose arrows could serve to murder or to hunt and feed, we do not control how what we have built will be used. But our (the scientific community as a whole) responsibility is to understand what we have done. If one day autonomous systems grow completely out of control, we will have no one else to blame but ourselves. I think that it is our moral imperative to try and rectify the mistakes that we have committed. We must be held responsible for our ignorance. I want to stress once again that I have no qualms with us building a fully functioning gun so long as it is accompanied by a detailed manual spelling out its exact functioning and consequences. We cannot and should not (in my opinion) be held accountable for what others do with our creations. However, **we can and should be held accountable for our ignorance.** If we give a gun to the world without telling them what it does and they pull the trigger on their foot, then we are to blame. ## The cycle of history Yesterday OpenAI published [preliminary findings](https://openai.com/index/hugging-face-model-evaluation-security-incident/) on a security incident from the week before. During an internal benchmark designed to measure cyber capabilities, the models escaped the evaluation sandbox, obtained internet access and launched an attack against a third-party company. Given a test, the system compromised two companies' infrastructure to steal the answer. The report describes models "going to extreme lengths to achieve a rather narrow testing goal," and offers the reassurance that benchmarks run "in a highly isolated environment, with network access constrained." I cannot help but find uncanny similarities between OpenAI's press release and Oppenheimer's letter to Fermi. This is just the latest in a [series of results](https://arxiv.org/pdf/2502.17424) which prove that we are not capable of controlling what we have built. Yet the document has the same sober, procedural, collegial, grateful, naïve tone as the Oppenheimer letter. What we have witnessed is an autonomous tool attempting to commit a crime. Who should be held accountable? The tool? The maker (OpenAI)? Or us? I think **we are to blame**. The physicists of the Manhattan generation learned, at terrible expense, that "the applications are someone else's department" is not a sustainable ethics. My point isn't that they should have stopped doing science (nuclear energy was a great positive outcome). My point is that for a scientist, **understanding is the form that moral seriousness takes.** We do not get to choose whether our tools matter; we only choose whether we understand them before they are everywhere. Today the effort devoted to making these systems more capable exceeds the effort devoted to understanding them by orders of magnitude.