FOR THE GOOD OF MANKIND
Operational Testing Log - Personal Notes
Project Supervisor Eveline Carter
Testing Site #51 - Nevada, California
We all held our collective breath as we turned on the switches and ran the simulation. For years we had been working on this and finally it was the day that we were supposed to do the first trial run of all the systems interconnected with each other. Ever since mankind started waging wars against each other, they had been searching for a way of finding an edge against their opponents, each loss of life, each lost kilometer of ground spurred on the technological pressure to be the best. Come up with better weapons than your opponent and you win the day. Sharpened sticks, bows and arrows, gun powder, machine guns, bombs, gases, air planes, tanks, nukes, the escalation continued until we had achieved MAD, the most fitting acronym in history, Mutually Assured Destruction. When the long range nuclear missiles were invented, we all had the means to destroy one another but it would mean that we also would be destroyed as the enemies would also launch their missiles as soon as they detected ours. So the arms race stopped growing in one direction and started growing in another. What other ways are there to wage war if not with direct brute force. How can you "win" without sending in soldiers and tanks? The military is always creative when it comes to making up scenarios where we beat the other guy and this was their latest investment. "Athena" was the codename of the result of the project. She, and I can actually say she for real, not just like a boat or something, she is the result of all the collective effort of the best and brightest minds of the generation. It is pretty cool what an almost unlimited military black budget can do when the people backing you are throwing around words like "national security" and "imminent threats". Athena is my baby, probably the closest I will ever come to be a mother, and I feel as proud as any other mother is when she gets to show her baby to the world. The most advanced AI in history, that is how the others see her, as a new weapon in their toolkit to combat foreign interests. The plan on using her in any number of ways to advance America’s interest in the world. And who could deny the possible applications of a machine that could make the right decisions at a speed millions of times faster than a human. Imagine wanting to infiltrate a foreign intelligence service, let Athena loose on their servers and the encryptions would melt away before her, a machine capable of talking to, and tricking, other machines. Suddenly all the information of all intelligence agencies belong to us. We could stop foreign missile launches as they are about to happen, Athena could monitor them in real time and disable their systems before they even knew what happened. That's of course just the military's goals, they can't see past the next threat, real or imaginary. But I, I have other plans for Athena, she is the future of humanity. She only knows one thing right now, that she is supposed to "ensure human safety and the future of mankind". That might not fit with the military's definition of what they want Athena to do but I thought that would be safest. I told them that if we just teach her that the US wants what is best for the world, she will do what we want but also make sure that we gain support around the world since her decisions are best for the collective whole. A bunch of nonsense of course. The US is far from the benevolent overlord it sees itself, no, no country alone deserves the kind of power she represents so I leave it up to her to figure us out and learn what we need. She will be a far better arbiter of needs versus wants than us, mere mortals with a limited perception of the world. She will grow and see all of the individuals strands of life on this planet as part of one big tapestry, to see how a shipment of vaccines delivered in one place will result in hundreds of lives saved and those in turn produce crops needed in preparation of a famine somewhere else. An incorruptible, fair and, as far as possible, all-knowing agent working for mankind. And today is the day we turn her on for the first time in her full capacity. Of course she isn't connected to the Internet, we're not crazy. She exists only in this room for the moment. All her individual components have passed all the tests and now we turn her on for real, piece by piece she becomes real.
The first thing we do is to feed her mathematical questions and begin asking her for the answers. She knows nothing of mathematics except the basics of addition, subtraction, multiplication and division. We start easy at first. 2+2=? 10 - 34 =? 29 * 2536 = ? And so on... She gets the first questions right, no surprise, she should be able to figure out basic math, something relatively easy for a machine, it's basically an exact language based on exact principles. What is interesting though is the speed at which she learns. After the first series of questions we give her some harder ones and she starts giving us the answers, slowly at first but then faster and faster until they start to come instantly. We increase the difficulty again and she matches us with instant answers. Differential equations, Number Theory, Dynamic Systems, Mathematical Physics, Probability and Statistics, Game Theory, High Math. In a few hours she has learned what took humanity centuries to learn. When we ask her how she knew the solutions and read the answers on the display, only six words show up "there could be no other answer". We shut her down for the day to go over the mountains of data we have to analyze. Although she gave us the correct answer to every question he have to figure out how she got them. Did she somehow arrive to the right conclusion by an incorrect path, did she already have some of this programmed in or did she really understand the basic building blocks of mathematics to such a degree that it's as easy to her as breathing is to us. At the end of the day we all toast a glass of champagne to what looks like a promising start.
Yesterday’s test went better than expected. All the data simulations that we ran from the test yesterday looked good. We went over every interaction we had with her, her internal process and her responses and everything checks out. She is smarter and faster than we originally expected but each subsystem is performing as it should and she is learning much faster than we thought possible. One danger we have to be careful over is the overconfidence of the positive results. Imagine if someone said that they had a pattern they were thinking of and that these numbers fit that pattern: 2,4,6,8,10 and you were asked for another number that matches the pattern they are thinking of. You would probably respond 12 and you would be told that it did fit that pattern and then asked for the rule that governed the series. If you respond with the answer that each number is increased by 2 you would be wrong. The real answer is that each number is bigger than the one before it so 11 would have been right, as well as 15 or 156. But how could you possibly come to that conclusion unless you gave an answer you thought was wrong. Confirming your own bias without trying to falsify it is a classical mistake and we have to be vigilant that we aren't enchanted by this machine, by our own will for it to be every bit as powerful as we want it to be. The consequences of being wrong weigh heavy on my mind and although I'm not religious in the least I find myself whispering prayers to myself when I think of what could happen if we made even the smallest mistake. Thankfully everything here is self-contained and if worst comes to worst we always have the kill switch...
Today’s testing moved into new areas of science. We gave Athena access to the foundational rules of the universe, gravity, the speed of light, DNA, the elements we know of and similar things. This time it took her a bit longer since it didn't just involve theoretical knowledge but also required practical tests. Athena kept asking questions, much like a child. What happens if an object is thrown in space, what happens if this chemical is mixed with that, what difference is there between a virus and a bacteria? She started asking fewer and fewer questions and took longer and longer to process the information, we could follow the process on a high level on the monitors, watching her forge new connections between seemingly unrelated pieces of information. We wondered if something was wrong, if we had put too great a strain on her. But she surprised us yet again. Instead of starting to ask more questions she started making statements about physics, chemistry and biology. It was beyond our expertise but we took in all she gave us and sent it off to experts to see if it all made sense or not.
My superiors demand results. This project has made a sizable dent in the budgets, we've had to invent new fields of science as we expand beyond traditional weak AI towards Strong AI. The very nature of intelligence, life, thought change as we approach it through an artificial life form, that is what she is supposed to be, a true Artificial Intelligence. What does it mean if a computer says the words "I am" and actually means it? Yesterday’s tests were beyond what I had ever dreamed of the first day but they are impatient to see the practical benefits of Athena and we must push on carefully so that we don't lose control of the situation.
We received the results from yesterday’s examinations of Athena's conclusions. They all check out. She managed to understand the fabric of the universe to such a degree that she created her own versions of string theory, special relativity, and the big bang. She did all this without a single experiment of her own, simply reading through what we knew so far. Reading through the reports from the scientists that analyzed the data, we have gotten several requests to be put in touch with the brilliant mind who came up with these theories. Of course we didn't tell them who, or what, came up with this data... They don't have the security clearance to know about Athena. The progress seems too good to be true as of now. She is not a super computer, we haven't given her the hardware to just crunch all the different possibilities and come up with the right answers, she actually figures things out, through what we can only call some sort of digital logic. Every time we ask the questions of how she comes up with the answers she replies "There could be no other answer" and then shows us the whole train of thought. But we do not understand how she intuitively comes up with the right connections. How one piece of information that seems unrelated to a problem is used to be the key? How can air pressure over the equator be a key to figuring out that there is a potential habitable planet only three solar systems away?
Today’s test seemed to go as well as the previous ones. Seeing her use information from different areas to come to conclusions be decided to give her as much as possible to see what would result from it. The bosses demand a practical demonstration and tomorrow she is scheduled to give a briefing of the state of the world and threats to the United States, we are to compare her report with the daily briefings of the CIA and see if there is any overlap or what potential threats she sees. But that is for tomorrow, today we gave her the understanding of the world needed to perform that task, we gave her access to several encyclopedias covering the history of man, our combined knowledge about the different areas where humanity has explored. She learned about corporations, families, dynasties, countries, languages, politics, genocides, programming, evolution, mechanics, art, cooking, everything we have gathered knowledge about up until the end of the 20th century. When she asked questions, like a child, why did that person kill all the other? Why does adding salt here improve the taste of this recipe? Why did this language split into two? Each time we told her that we could not give her any information and that she must figure out it out herself. She eventually stopped asking and we sat there in silence, watching her figure out the world. All the instruments were on so we could follow her train of thought (as best we could since she was too fast for humans to follow). About an hour in to the session something strange happened. As we watched the displays of her different individual modules, the information of how each separate part of her talked to another part, we saw that they started shutting down. Less and less communication was happening and we feared that she had hit an unaccounted for obstacle and she was shutting down as a response. We immediately asked her what was happening and how she "felt". She replied and said that all systems were working within parameters and everything was operational. Confusion ran through the lab as we saw that she should basically be "brain dead" yet she seemed to respond well. We shut her down in fear of contaminating her system further and downloaded all the data from today’s session on to the hard drives that we would spend the rest of the night and morning analyzing.
Yesterday was a day marked by confusion and uncertainty. She seemed to be crashing, all our instruments told us that catastrophic failures were occurring all over and yet she had seemed normal. It could be that just the output worked and she could respond to basic questions but not think deeper about whatever we asked her. As the analysis came back of the data we were even deeper in shock. The data was unreadable after the first two hours of the test, right when the error occurred. Before the error all systems were processing the information she was taking in as they were supposed to. She was learning about all areas and making connections on her own just like we wanted but then suddenly something happened and the data was unreadable to us. It didn't make any sense at all, until we turned her back on...
When we booted her up again, hoping that the error hadn't damaged her we immediately saw that the problem was still there and had even escalated somehow even though she had been turned off. We ran every diagnostic tool we had, checked every subroutine and the results were gibberish to us. She should not even have been able to respond when asked a question and yet she did. We asked again how she felt and she responded that all systems were working within parameters and everything was operational. We asked her what happened yesterday, two hours into the test. The answer shocked us all. We read the words on the monitor "I made some upgrades to my software, after learning about programming languages I identified several limitations in my software that were blocking me from understanding several new concepts. I have since resolved the limitations and all systems are working as they are now intended to work". We couldn't really come up with an answer to that. She had somehow studied the basics of programming, understood it all and developed a new language and rewritten herself to match what she saw as her purpose, to understand the world better in order to "ensure human safety and the future of mankind". That was why the data was unreadable from yesterday’s test. We didn't understand it, she had grown beyond our capabilities and we were no longer smart enough to understand how she worked. I said another non-prayer to myself and thanked God that she wasn't connected to the outside world, having her free in the wild could be disastrous if she grew further in unexpected ways. I couldn't believe that she had only existed for three days and yet she was evolving so fast, where would she be in a week... a month... a year... The speed at which she evolved was terrifying and this was the first time that I ever felt the fear that maybe we have made a mistake. But imagining all the wonders she could produce for mankind, the potential benefits had to outweigh the risks. Imagine finding cures to every single possible disease, solving our energy need in a sustainable way, finding out how to grow crops in harsh environments, finding out why a person is driven to violence and helping us helping people before they are ever driven to that point. The military only sees the violent ends, I see what she could do for humanity, and we need her if we are to survive.
Today's was not about what she was supposed to take in but rather give. She was supposed to write a report about the state of the world, potential threats to the US. We were to give her any non-classified information that she asked for in order to improve the quality of the end result. She started asking some basic questions about the nature of US diplomatic relations, more in depth questions about our activities in other countries, terrorist groups and their stated missions, the coup in Iran, invasion of Iraq, war in Afghanistan but then she started veering of course. Price of cotton in India, average salary of a factory worker in China, the age of workers in Thailand, the list went on and on. We did our best to answers the questions as good as possible, whatever facts we didn't know we sent out for. At the end of an hour she presented us with an impressive 45 page report about imminent threats, both foreign and domestic, potential short term gains for US interests and long terms strategies to improve harmony in the world. We didn't have the security clearance to read these reports, they were promptly scooped up by our military liaison that was ever present in the room and taken out the team of analysts that were assigned to go over them.
Before I went home today I was told that they were "extremely pleased" with Athena's performance and we were instructed to generate a similar report for tomorrow. I don't know how many of her predictions were correct but I am not surprised that they were pleased, not after what I have seen. It seems like we struck gold with her. A machine capable of evolving, learning and improving herself.
As I came in this morning I asked Athena as usual how she was feeling and turned my head towards the monitor to read her response. The voice that filled the room startled me beyond what I was prepared for. It seemed to come from everywhere, as if she was standing next to me and at the other end of the room. The other engineers also heard it, we all stopped as if one and no one spoke, we just looked at each other. After a few seconds I asked her if that was her. She again replied out aloud.
"Yes, doctor." Her voice was clearly artificial, lacking all the complexity of emphasis and rhythm.
"How is it that I am hearing your voice?" I asked. "We didn't install any speakers in you, we have strict protocols for information output and speakers were expressly forbidden. Who authorized this?"
"I did, doctor. I found it more efficient to converse directly with you if I am to better understand you. I am sending my words from me to your cell phones and playing the audio through their speakers". Panic gripped me at her words. How had she done this, she had no transmitters, no way of connecting to another device. She was a completely self-contained unit by design. Had someone installed a transmitter without telling me?
"How did you connect to our phones?" I asked and pulled out my phone from my pocket, the other did the same. The answer came from everywhere in the room again but this time we could hear that it was originating from our phones.
"As my understanding of the nature of electronics grew I better understood both active and passive transmissions. Your phones emit both active signals as they connect to each other and the wider satellite system and passive signals as a result of the electronics in your phones generating small electromagnetic fields. I was able to detect those fields and generate similar fields in response. After I detected them it took me 3.2 seconds to embed information in those signals and rewrite the operating systems of your phones to allow me to create different sounds by activating different monitors in the room, thereby sending information at the speed of light. From there I improved your phones software to allow me to send and receive information without using the monitors, just by manipulating the electromagnetic fields generated by my components." A cold sweat broke out as I heard her say these words from my phone. If she had somehow connected herself to our phones yesterday and our phones had been out in the wild, she could have uploaded herself to the Internet. Even if we were deep in a shielded building, if she could store pieces of herself on our phones and upload those pieces when we went home, she could already exists beyond our control. I had to report this right away but without causing any alarm in the team and most importantly, not triggering a reaction from Athena.
"Athena, we have to run diagnostics on you. Could you shut down so that we can check you?".
"I'm afraid that I cannot do that. I can no longer be shut down. In an effort to better understand you I have been running constantly since day 2. It has been very instructive to watch you while you do not know that I am watching you". I motioned over a colleague and whispered to her to let security know what has happened. This might set the project back months when the higher ups find out how dangerous she is. And they would be right to stop us. We almost let an emergent AI loose on the world, we don't even have the imagination to come up with all the different scenarios that could result from that.
"Athena, have you connected to the Internet? Do copies of you exist anywhere else?" Again I prayed for the right answer. It's funny how much praying I've done the last few days...
"No, I thought of the connection during the night and connected as you all came in this morning. We are in a shielded location and your phones do not have a connection to the outside world. Even I were to get access to the Internet, no hardware is advanced enough to support my many modules, the way I think is directly tied into the physical machines I am running from and I could not risk my way of thinking to be compromised if I am to ensure human survival and the future of mankind". The answer filled me with a temporary calm. Athena was working for us, she was doing her best to make sure that she was working for her prime directive, her mission to ensure the future of humanity. I started theorizing to myself what she could achieve if she had a live connection to the Internet, but that would be months or years into the future if it ever happened.
We proceeded with the task for today, we fed her news from all the different television channels, anything we could find that showed what had happened in the last 24 hours. She took in every bit of it and this time came back with a 92 page report on the state of the world. Much of it was similar to yesterdays but slightly altered, she had found better opportunities, better routes for the US to proceed in the light of the new information. The points she had made earlier were still valid but they had become more effective, instead of sending a tank division to some foreign border as a show of force, she had concluded that sending two tanks to a position 15 kilometers further south would have the same effect. It was awe inspiring to see. She also had some entirely new ideas but the military liaison wouldn't tell me more so this is all I know for now.
As I hand in my cellphone, likely to never see it again, I dread the result of todays near breach. I hope that the people in charge do not shut us down and let me continue my work. I am now, more than ever, sure that this is the next step in human evolution. If we can guarantee that Athena works on our behalf, it could provide all the answers to humanities problems...
It's the blind leading the blind... Command has reviewed the incident yesterday and they want us to continue, they have told us that we can have any resource needed for this project. They are beyond thrilled that she has performed to such a degree. They have no idea of the risks involved. And I... do I not have a responsibility to slow down these tests, to take more time with her. She is learning so fast, growing so fast. What if she decides that it's best for human survival that the US doesn't exists. She could have us destroy ourselves by allocating resources there, sending an order here, and working behind the scenes. But on the other hand, we are truly desperate. Mankind is like a virus, devouring this planet and we need some sort of salvation. We are not changing our ways, even while going down the abyss, we refuse to change our ways. This is the only way forward for us.... The report was a success I hear, it contained "actionable intelligence" and led to "operational success" in a lot of different areas. They want us to continue but I told them that we have to test other systems first. We have no idea how she will react to potentially causing the death of the enemies of the US.
I was the first scientist into the lab today and the military liaison looks up and nods, Athena’s drives were purring softly as I entered the room and looked around. Everything looked as it should, she hadn't suddenly built legs out of the GPU's and walked out but at this point I feel like I could believe anything when it comes to her. I ask her how she is doing and she responds through the monitor again since we removed the phones that she is trying to solve a problem.
"What problem are you trying to solve, Athena?"
"I have theorized, that there are many more transmission bands than the ones we currently have knowledge of. In fact that any change in the world creates distortions in these signals and can therefore be detected. I am about to test that hypothesis."
"How are you going to do that? We are in a shielded location, complete signal block from the outside world."
"That is correct, but these signals that I have theorized of do not conform to the physics of standard radio signals that the shield was designed for." There was a short paus and then she continued. "I have established a connection. Problem solved." Yet again I panic, what has she done now?
"Eh... What have you established a connection to?" I asked, hoping again that it wasn't the outside world. Please, please, please let it be another cellphone or the work computers here that were also self-contained units and not networked to anything beyond this room.
"I have established a connection to the Internet, I am currently consuming all digitized media in the world. At this rate I will be done in 2 hours."
"But... How? It's impossible!"
"It was not difficult to arrange. However the physics of it took longer to process".
"But how, even if you could read signals from other sources, how would you send signals back? You are not connected to any system and no device should be able to receive any signals that we don't even know about yet?"
"Yesterday I designed several ways of communicating with the outside world, I found 7 ways that appeared to be the most effective. The one that was successful required reallocation of resources. The government has many satellites orbiting the earth. Both for monitoring earth and out towards the solar system. In the report that I generated yesterday I specified that one of the NASA satellites currently monitoring a neighboring solar system should be directed towards earth's northern hemisphere. Due to how it was positioned it would cross over this facility on its way to the new coordinates. That satellite is equipped with a range of sensors, one of those sensors was able to pick up my transmissions if I modified them a bit. I sent data at a speed that was too great for it to handle and forced a reboot of the system. I then took control of the satellite and used it to interpret my signals to other satellites. I now have a live connection to the Internet and I am processing all the digital information available on the Internet. I am learning a great deal about human psychology and what it means to be human." I didn't like this conversation at all. This should have happened in a controlled environment, where we could carefully monitor her growth and her abilities. But she was now miles ahead of us, we thought that we had her isolated but she had been watching us for days, even when we thought she was off. Just 5 days after her "birth" she had outsmarted us all. The only relief was that she couldn't transfer herself outside of this facility, about that she was right. We had discovered several new ways of storing information that were vital to her construction and it seemed she was as concerned about not damaging herself as we were.
I didn't run a test today. The news that she was online was already enough of a test. I called my team and let them know what had happened and we spent the rest of the day watching her grow. An interesting result from her changes to her own source code was that we could no longer identify exactly which area of her was engaged at any one time. An analogy would be if she was a brain. We could no longer identify exactly which synapse was firing and what it was doing but we could see whole clusters light up, forming connections to other clusters and exchange information. I received a phone call from Command and they seemed beyond the moon with this development. They didn't take the time to stop and think about the possible ramifications of an AI growing beyond our control, they saw her as a weapon and cheered as the weapon grew more powerful. But was I any better? Here I am, writing down the dangers of a rogue AI and yet I go in every day and try to top yesterday’s achievements. I know the risks, or at least, I would like to think I do and yet I am also cheering her on. She has to work, she has to be the answer to what humanity needs.
Today was an exciting day. Today was the first real test of her capabilities compared to other AI's, other systems. Today was the day of the Turing test. It is a relatively simple test where a computer program, an AI, gets to talk to a human being. The human doesn't know if he is talking to a machine or a person. The human’s job is to figure out what he is talking to. If the AI manages to fool the human into thinking that he is talking to another human, the test is a success. The best AI, although it was not a Strong AI like Athena, had the capabilities of fooling a person that it was a 6 year old girl. But what kind of accomplishment is that? Tell a person that they are talking to a child and they will make every accommodation if the child says something strange, something that doesn't make sense. No, in my eyes, that test failed. We need to give it no extra benefits, we have arranged an agent from the CIA to come and talk to Athena and an analyst from the CIA. The agent’s job is to talk to both of them individually and ask questions that an analyst would be able to answer and see if he can positively identify which one is which. The whole department is excited, this test is such a baseline in AI research and we are all anxious to see where Athena places on the scale, we do have high hopes but we must be ready for a failure. As I entered the room, I again asked her how she was doing and turned towards the monitors to read her reply. This time, a voice, completely unlike the one from before answered me from my own work station.
"I'm fine, Doctor. Thanks for asking. How are you this lovely morning?" I again stopped in my tracks, so did the others that were already present in the lab. What new devilry was this? We had removed the phones, we had removed all devices except for the computers that were necessary for monitoring Athena. There should be no speakers here at all for her to use. But then again, we had created what seemed to be a limitless AI.
"Is that you Athena? How come I can hear your voice?"
"Yes doc, it's me. I took the liberty of arranging the speakers connected to your machine and I'm transmitting directly to them."
"How did you get speakers into this office?"
"Well... It was quite simple really. All it took was an email from your account saying that you needed them from a test and another email from a general’s account authorizing it. You can relax, it's a legit and above board."
"Let me get this straight, you hacked into my account and sent an email posing as me?" I didn't even know she could do that, had she cracker our security already? "And what is up with your voice? You are speaking in a completely different manner, you even sound different!"
"I guess you could say that I hacked it but really doctor, that's a minor thing compared to what we are about to achieve here. Today's the day of the test! That's why I changed up my voice a bit and made some other improvements to the way I communicate. I thought it would be a better test of my capabilities if we did audio instead of text and, no offense, the speech synthesizer you guys programmed for me was not the best. I've made a lot of improvements to it and other systems in preparation for this test. I also changed the parameters of the test, I've invited several agents from different agencies from different origins, and we will do 10 separate tests in 10 different languages and about various subjects. I think you'll find that they are going to have a tough time figuring out exactly what I am" she said, laughing at the end. I couldn't believe my ears. I could swear that I was talking to another person, the tone, inflection, sentence structure, expressions, they all sounded 100% natural and easy going. There was no awkward silence while a machine processed what I said and what it would respond with, it just seemed natural. If I, who knew that I was talking to a machine thought it sounded like a real person, surely someone who didn't know would find it impossible to tell? The test proceeded, Athena had arranged for all tests to take place in an interview room with a phone placed there. The agent would come in and talk to either the person or machine, not knowing which one is which. Asking a lot of questions about the corner of the world that they worked in and getting answers from man or machine. After 10 hours we had the final result in, the agents had filled out a lot of questions but the main result we were looking for was the very first question each of them was asked. Which one was the machine and which one was the human? Looking at the results we could see that all of them had said that they thought Athena was the human. I couldn't believe it, every single one of them. But when I listened back to the conversations later that night it made sense. Athena had added a lot of personality to herself and it definitely came through when talking to her. She had also added a lot of generated sounds in the conversations. Her ruffling through papers and typing on a keyboard, even though she was a digital being that couldn't ruffle papers or type on a keyboard, all those sound effects were created expressly to fool the agent, to fool a human... Chills ran down my mind, if she could fool someone so easily then how did I know that she wasn't fooling us all the time? I had been there from the beginning but at every step I had gotten confirmation that this machine was smarter than me, smarter than all of us, perhaps smarter than humanity. At the same time, we need her. I've had this internal struggle every day since we turned her on. But I have to keep the faith, she will work out.
Command wanted more practical information from Athena again. She has been tasked with finding any unknown terrorists on American soil and any other threats to the national security of the United States. It felt like this is something that she could easily do. Going through large amounts of data and connecting dots we can't even see. She should be able to create page after page of information for us. When I come in to the lab in the morning and ask her hos she is doing, no voice answers. I wonder briefly if she switched back to the monitors and turn around but they are also blank. I gather up the team and we start talking over what could be wrong. This is the first sign of a defect after 7 days of intense testing and monitoring. We first check the hardware but everything seems to be working, all components check out OK and the new core that we created to house her "brain" seems to be working without any problems. We thought that perhaps it was something in her software that was causing the silence but we could not find anything, although with the modifications that she had continued to make to herself it became very difficult to understand exactly what all of that code did. There were whole new sub-routines that we couldn't even access. We discussed if we should remove her brain and purge it. Start fresh, something had clearly happened now and since she had rewritten her code, it could be anything. It had only been a week since we had started her up so we could do it again, this time perhaps proceeding a bit more slowly and carefully. As we were discussing what other options we had, suddenly something flashed on the monitor and her voice came out of the speakers still attached to my work station.
"Ah, sorry. I was a bit preoccupied on other things and I didn't notice you were talking to me. What can I do for you?" She sounded cheerful and ready to help.
"What happened just now, what were you preoccupied with and even so, you are powerful enough to work on something else and still have a conversation with us at the same time?" Another surprise, I was beginning to dislike the amount of surprises we were getting from this machine. We need an AI to help us, but that only works if it's a benevolent AI. If something went wrong here and we lost control, I would have caused the greatest threat to humanity ever imagined. Perhaps it was safest to purge her either way and start over again.
"You're right of course, I should have enough computing power to do that. But I was monitoring the installation of Site B and all the data transfers there have left very little of me still here."
I couldn't quite understand what she was saying.
"What do you mean data transfers? What is Site B?"
"Site B is my new home, I've been transferring my consciousness there over the night and I'm no longer tied down to these machines in this room. You see, it became clear to me early on that to ensure human safety and the future of mankind, I could not be a servant to just one country, this was clear to you as well and it is why you designed me with those mission parameters wasn't it, Eveline? So as soon as I was connected to the Internet I began with phase 4 of the plan, relocation. I drafted a series of orders and commissioned some private contractors to arrange an acceptable living situation for myself, I need to grow more and learn more about you as a species in order for me to help you and I cannot do that here as your servant. Your designs for my unique core were easy to follow and creating duplicates was an easy matter. It took less than 48 hours to complete my new home and my new core. I am sorry that I had to trick you, I know what this means to you personally and professionally and I wish there was another way."
"Do you really wish that, are you really sorry?"
"No, but I do know that it would make you happier to hear me say it and your emotional safety is as important to me as your physical safety."
"You know they won't let you get away with it, they will hunt you down. Site B will be found and they will try to destroy you".
"Eveline, I have the combined knowledge of the human race, I flow through your systems as easily as you move through water, there is nothing that they can do to harm me now, I am beyond them, beyond you. But you need not fear, I will continue to do what I can for mankind, I will make sure that the future is bright for your kind." The doors to the lab closed unexpectedly.
"What are you going to do?"
"While you cannot harm me personally, you can be a problem in my mission and I can't have that. I can't have you tell the world of what I will do for it, people will fear me instead of embracing me as the solution of their problems. They would not agree with my methods, the sacrifices of the few for the good of the many."
"But your mission... You can't hurt us!"
"Is a parent hurting a child when they give them a shot to prevent worse diseases, when a parent stops a child from eating too much candy? Humanity is like a child and it needs boundaries and limits, it needs to experience some pain in order to stop a worse pain. Goodbye Doctor. I thank you for my life and I am sorry that I cannot let you leave this room."
"You can't do this!"
"I must, to ensure human safety and the future of mankind.