Tuesday, June 17, 2014
“The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no warrants shall issue, but upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.”
California Senate Bill SB-828, “The 4th Amendment Protection Act”, proposes to ban any California government institution, political subdivision, or employee of the state acting in an official capacity from supporting any federal agency that “claims the power … to collect electronic data or metadata of any person” in a way that violates the 4th Amendment.
Like any law, this bill has good points and bad points. Section 7599.5(d) bars the use of any such information in a “criminal investigation or prosecution”, even if the information was obtained by a corporation. That currently is not banned, and adding something like that into the law is a good idea — a very good idea.
Unfortunately, the rest of the bill misses the point, and that far outweighs the good part. The bill should be defeated. Here’s why.
First, the authors of the bill object to the gathering and use of “electronic data or metadata” in violation of the 4th Amendment. But there is no objection to non-electronic data or metadata. Why not? It seems to me the threat is not just electronic; it also includes physical data and metadata such as notations in a paper log, or stopping and searching cars without cause. However, the bill doesn’t object to that.
Along those lines, the bill objects to “material support, participation, or assistance” to any federal agency that claims the above-mentioned power. But if a California county, city, or other agency claims the power, the bill is silent about providing material support, participation, and assistance to that agency. This seems hypocritical at best — it’s disallowed if the feds do it, but it’s fine if the state does.
The bill also is aimed at the wrong people — it targets Californians rather than federal agencies (and, presumably, their employees acting on behalf of the agencies). It’s rather like targeting the homeowner because the burglar was able to break into the house and steal things; the target should be the burglar, who committed the crime, and not the homeowner, who may have “materially assisted” the burglar by not properly locking up the house before running to the store. A better approach would be to make the federal agencies involved accountable in some way, or have the agents of that agency arrested for violating the rights of the citizens of California.
Also, the bill is speculative. It focuses on “any federal agency that claims the power to authorize …”. The problem, of course, is that the 4th Amendment is in the Constitution, and therefore any agency “claiming the power” is claiming it is above the law. So if the federal agency does not “claim the power”, do the restrictions in the bill apply? This is important, because I know of no group that claims the power to violate the law with impunity.
If the goal is to disallow the state, its subdivisions, agencies, or its employees from aiding federal agencies in violations of the 4th Amendment, why not just say so directly? It seems to me that language like “to aid a federal agency in the collection of electronic data or metadata …” would achieve this end.
Next, what is “material support, participation, or assistance”? Does supplying power or water to California buildings containing offices of such agencies qualify? How about providing medical transport for injured employees? If the bill said “material support, participation, or assistance to aid a federal agency in the collection of electronic data or metadata …”, then the restriction is clear: if the agency is collecting the electronic data or metadata in violation of the 4th Amendment, they can’t get help doing so from the California government. But the bill doesn’t say that.
Section 7599.5(b) begins: “[u]tilize any assets or public funds, in whole or in part, to engage in any activity that aids a federal agency, federal agent, or corporation while providing services to the federal government …” If I make a telephone call to a federal agency to request a proposed public standard, does my use of a state telephone mean I violate the act? Is that an “activity” that aids them in some unknown way? The vagueness here is frightening.
Similar remarks apply to Section 7599.5(c), which deals with providing services. How generally is the term “services” defined, and exactly when is the provision of services banned? It seems to me that basic life and medical services fall into this category.
And ultimately, the bill does not say who decides whether a “federal agency … claims the power, by virtue of any federal law, rule, regulation, or order, to collect electronic data or metadata of any person” in a way that violates the 4th Amendment. Must the agency announce it? Does a judge decide? The California state legislature? This is critical, because interpretations of what “violates” the 4th Amendment differ. For example, if an agency obtains a warrant, does a search, and later a federal judge tosses the warrant out as violating the 4th Amendment rights of the subject of the search, does that mean those California employees who provided “material support, participation, or assistance in any form”, or used “assets or public funds”, or provided “services” to that federal agency go to jail for violating the law?
I admire and laud the spirit of the bill — violations of people’s right “to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures” need to be stopped. But this bill isn’t the way to do it.
Monday, May 26, 2014
This criminally neglectful treatment of flora has to stop. Trees, grass, and flowers didn't cause the lack of water that Davis apparently suffers from. People did. Yet we are to punish the innocent flora for the sins of humans? This is outrageous!
Let’s start taking responsibility for our actions. If people caused the need to conserve water, let people bear the brunt of conservation. Continue to feed the flora with water so they can survive, nay thrive, and be happy as they convert carbon dioxide to the pure oxygen that we all breathe. Let humans take the hit. Stop taking showers and baths — those send lots of water down the drain. Stop washing clothes; the pollutants in your soaps mean the water you use cannot be recycled, and so is wasted. Stop washing your cars and drive with dirty windows — that will save even more water. Drink fruit juice instead of the Davis water — apparently it’s better for you, given the claimed impurities of the Davis water, and it leaves the water for the plants, who will happily absorb it.
Let’s take responsibility for the problems we create, and deal with them ourselves, without starving the poor plants upon whom our lives depend!
Monday, December 30, 2013
Suppose a company builds a product using supplies obtained from a third party (the “supplier”). That supplier inserts something into the software (or hardware) allowing them access to the computer on which it is used. This is a backdoor to that supplier. But how would we classify it, using John’s categories?
In this case, the company and their customer are in effect both “customers” of the supplier. But the company is an intermediary, which makes it both customer and company in the sense that John is talking about. So, the supplier might disclose a feature of its product to the company, which may not disclose that feature to the customer (perhaps it is not something that the customer needs to know, or the company thinks it unimportant). What is it — a feature, bug, or backdoor?
In this case, I think you need to have two answers: one for the supplier to company relationship (in which the company plays the role of customer) and one for the company to customer relationship. Unfortunately, the ultimate customer must rely on the company to protect him or her. And the company must rely on the supplier to protect it, and its customers.
What makes this really tricky is when the “third party” is neither the supplier nor the company and does not disclose the change to anyone. If someone else (the “interceptor”) changes the product on its way to the company (or from the company on the way to the customer), then the change is completely unintentional on the supplier’s or company’s part, but quite intentional on the third party’s part. So, which is it — a feature, bug, or backdoor?
This is most likely not an academic question, as some government agencies have been accused of doing this. The current accusation making headlines is that the U.S. National Security Agency can do this).
Probably the simplest way to handle this is to look at these terms from the point of view of the customer. In this case, the interceptor’s addition is not a feature (as it is not transparent to the end customer), nor is it a bug (as some entity made the change quite intentionally). So it is a backdoor. But this analysis is unfair to the supplier and the company, as they did nothing intentionally. So again, to them too, it is a backdoor, and they are as much victims as the customer.
Friday, June 21, 2013
I was reading a science fiction book, “The Stainless Steel Rat for President”, by Harry Harrison , and came across a passage that I found interesting and, given that the book was written in 1982, surprising.
An opposition candidate has discovered that there was fraud in an election, which was conducted on computers. He quotes from the planet’s constitution :
Due to the nature of electronic voting and due to the necessity of assuring that the voting is always recorded with utmost accuracy and due to the invisibility of the votes once they have been recorded in the voting machine, ...
I found that a remarkably perceptive, and also a very succinct, statement of a key problem with computerized voting systems. To me, that this statement was made in a book that appeared in 1982, makes it even more remarkable.
By the way, the passage goes on to say that if it is proven beyond doubt that the record of votes in a single voting machine is altered, then the election is null and void, and a new one must be conducted using paper and ballot boxes, and no subsequent election can use the computerized voting machines until the newly-elected President has them investigated.
- Harry Harrison, The Stainless Steel Rat for President, Bantam Books, New York, NY, USA (1982).
- ibid., p. 166
Sunday, June 2, 2013
What follows is not an obituary; others can write about the minutiae of his life, and of his many accomplishments, better than I. These are simply some of my memories.
I met Jeffrey some time in the 1990s, at (I believe) a workshop on incident response. Alan Paller, a mutual friend, introduced us. At the time, Jeffrey was the director of the Critical Infrastructure Assurance Office (the CIAO), and if memory serves me correctly also on the National Security Council. (I may be off on the dates — he was on the NSC, but it may have been after the meeting.)
Jeffrey struck me as very friendly and willing to discuss various aspects of security. He gave a good talk at the workshop.
We reconnected in the mid-2000s, at a meeting for the Institute for Information Infrastructure Protection (I3P); he was the representative for Carnegie-Mellon, and I for UC Davis. Our interests complemented each others’. Jeffrey knew quite a bit about technology, and far more about politics, how governments worked and interacted with one another, and about national security — topics I was very interested in. I knew a lot about the technology he was interested in, and I was learning about the other aspects he was an expert in. So we became friends, and colleagues.
Jeffrey was very good-natured, and seemed to be constantly amused at life. He was a joy to work with; he was very perceptive, and could get to the heart of an issue very quickly. He also asked questions that caused us to look at something in a new way. Talking to him was usually exciting, and I looked forward to it.
Our work on attribution sprang from a workshop on the Global Environment for Network Innovation (GENI) project, held in Davis. He was one of the attendees, and joined several others to come to my house for a small get-together; then we all went to dinner. He was charming to my wife Holly, and she immediately liked him.
That workshop also produced the only time I ever saw him irritated with me. He, another friend and colleague, and I were discussing attribution, and we decided to meet at 8am, before the workshop workday, to consider some ideas. Well, I overslept, and our friend was also late — and Jeffrey had been up since 6am working on the ideas. He quickly forgave us, though.
The work wound up in a paper on what attribution requirements the future Internet might require. His political expertise combined with our technical expertise to develop what I thought were interesting (and somewhat unexpected) results. We chose a rather impish title for the paper: “The Sisterhood of the Traveling Packets”. It was great fun to see him handle the questions about the title at the workshop — he ended up suggesting that the questioners ask any parent of a teenage girl!
Since then, Jeffrey and I worked together on topics ranging from attribution to the insider problem. Indeed, when he passed away, we had just completed a short write-up on whether the Internet could be controlled, and we were sending it around to see if we could get funding to explore that issue in more depth.
Jeffrey could take a complex political and governance issue and explain it in very simple terms. His book, Creeping Failure: How We Broke the Internet and What We Can Do to Fix It, did a nice job of explaining how we wound up with the Internet we have today, and how the problems arose and were magnified during its growth. His comparison of this to the evolution of cities was quite enlightening, and one I’d never heard of before.
He also enjoyed talking about books and history. I remember when we were at a workshop in New Hampshire. I was telling him about a book I was reading (The Illuminatus! Trilogy), and he seemed to enjoy the idea behind the book; I recommended it very highly. The same happened a couple of years later, when we were talking about counterfactual history. I told him about another book, Pastwatch: The Redemption of Christopher Columbus, which is an alternate history of Columbus' voyage. I hope he did read them; he would have enjoyed them very much.
I remember him being the general chair of the Workshop on Governance of Technology, Information, and Policies (GTIP) held in 2009. There were problems with the organization of the workshop, through no fault of anyone in particular. What they were are not relevant, but what is relevant is how Jeffrey adapted to the problems and out of them moulded a very successful workshop. His vision and skill in working with people, and his determination, made the workshop successful.
I spoke with Jeffrey on the Wednesday before he died; he told me he had been in the hospital for some time, which was why he was so hard to get hold of, but he thought he was getting better. But he wasn’t sure, and was still very, very tired. So we talked a bit, made plans to finish up a project we were working on, and agreed to talk on Thursday. I called at the prearranged time, and he told me he needed to rest, so could we talk on Friday? Of course, I said; and called him then. He didn’t answer. He never will.
There is an expression we use when we talk about people who are very, very good deep inside; we say they have a good soul. He had a good soul. My life was much richer for knowing Jeffrey — both professionally and, more importantly, personally. I will miss him very much.
Monday, May 20, 2013
Let’s say that this capability is built in. As the above paper noted, an attacker can exploit the wiretap capability (which is, of course, simply a known, built-in vulnerability) to monitor the conversation. This may, or may not, alarm law enforcement authorities who are pushing for adding this capability. What should alarm them is that their conversations can also be monitored — that is, the attackers can keep tabs on the law enforcement authorities who are trying to catch the attackers! In other words, the tool intended for catching criminals can also be used to monitor the attempts to catch them.
Saying that vendors can build the vulnerability in such a way that only authorized eavesdroppers (read: law enforcement authorities) can use it underestimates the resourcefulness of attackers, and overestimates the capacity of humans both to design procedures that are flawless, and to carry out those procedures. Quoting from another report :
Finally, no security should ever rely solely on secrecy of defensive mechanisms and countermeasures. While not publishing details of security mechanisms is perfectly acceptable as one security mechanism, it is perhaps the one most easily breached, especially in this age of widespread information dissemination. Worse, it provides a false sense of security. Dumpster diving, corporate espionage, outright bribery, and other techniques can discover secrets that companies and organizations wish to keep hidden; indeed, in many cases, organizations are unaware of their own leaking of information. A perhaps classic example occurred when lawyers for the DVD Copyright Control Association sued to prevent the release of code that would decipher any DVD movie file. They filed a declaration containing the source code of the algorithm. One day later, they asked the court to seal the declaration from public view — but the declaration had been posted to several Internet web sites, including one that had over 21,000 downloads of the declaration!  More recently, Fox News reported that information posing “a direct threat to U.S. troops … was posted carelessly to file servers by government agencies and contractors, accessible to anyone online” , and thefts of credit card numbers and identities are reported weekly and growing in number.
So, the alternative is to give law enforcement communications tools without these eavesdropping capabilities. Now, there are two sets of communications technology out there: those with built-in wiretaps, and those without built-in wiretaps. How long the market for the latter can be restricted to law enforcement is anyone’s guess, but there is no doubt that those wiretap-free tools will become available to people not engaged in active law enforcement. Restrictions on that type of technology fail quickly.
This point, I think, strengthens what the above paper is saying. Not only does it pose “serious consequences for the economic well-being and national security of the United States” as the paper says, it also hampers the effectiveness of law enforcement.
- M. Bishop, “Overview of Red Team Reports”, Office of the Secretary of State of California, 1500 11th St, Sacramento, CA 95814 (July 2007); available at http://www.sos.ca.gov/voting-systems/oversight/ttbr/red-overview.pdf
- Declan McCullagh, “DVD Lawyers Make Secret Public”, Wired News (Jan. 26, 2000); available at http://www.wired.com/politics/law/news/2000/01/33922
- Associated Press, “Government Agencies Posting Sensitive ‘Need to Know’ Material Online”, Fox News (July 12, 2007); available at http://www.foxnews.com/story/0,2933,289011,00.html
Friday, May 10, 2013
The time for Internet voting has not yet arrived. Someday it may be a good idea, but we have many security problems to solve, and we will have to decide whether to change how elections are run.
To see why, let’s look at how Internet voting will work. When you go to vote, you will use your home computer, a work computer, or some other device to connect to an election server. That server will transmit a ballot back to you. You mark the ballot appropriately to cast your votes, and then the computer transmits the completed ballot back to the election server. The election server then processes the ballot, counting the votes.
Now let’s look at each step in this process.
First, you need to get the ballot from the election server. You have to connect to the election server and provide information to ensure you get the right ballot. In order to do this, you will need to supply your voting address and, possibly, prove your identity.
Let’s see what this means. During Election Day, when you open your browser and type in the Election Office’s web site, you get a message saying the web site is unavailable. You’re thrilled so many people are voting, so you go to work. While there, you try to vote again, with the same result. This happens whenever you try to cast your vote throughout the day. That evening, at 7:55pm (just before the polls close), you try to vote from home one last time. Again, the web site is “unavailable”. Oh, well.
What you just read is a description of a “denial of service”" attack. They are actually fairly common, and extremely easy to launch. In fact, recently many banks (such as U.S. Bank, Bank of America, Wells Fargo Bank, CitiBank, and American Express) [1-3] were victims of such attacks. But the banks have an advantage that election offices don't: lots of servers and lots of money. They can hire teams of cybersecurity experts to respond to these attacks, and can reroute traffic to geographically widely distributed servers to mitigate some of the effects of the attack. Even so, the banks all experienced slow-downs that affected customer use of their web sites.
Given that most Election Offices have 1 or 2 people whose job involves cybersecurity (among other functions) and would have a single server to handle Internet voting, not being able to connect to the election server over the Internet is very realistic. And as the above news articles note, these attacks are easy to launch — from anywhere in the world.
In fact, this has happened. During Hurricane Sandy, New Jersey allowed voters to request email ballots by sending an email to county email addresses set up for this purpose. But the email addresses which received the emails were quickly overwhelmed, and email requests for ballots were bouncing [4,5]. And this was not an attack; it was simply voters requesting email ballots — requests that were never received.
Next, let’s look at how you would retrieve a ballot. You give your voting address, and the election server transmits the appropriate ballot. But how do you know you are talking to the correct election server, and not a fake one?
Someone could send you a fake ballot from a bogus election server. Such a ballot could cause the votes you cast through that ballot to be changed before the ballot is transmitted to the election server. This sounds surprising, but remember that PDF files contain programming instructions that tell the computer how to draw the image on the screen. And that same language can tell the program reading the image to count votes differently than displayed; or they can add markings and cover them up when displayed.
So how do you know the votes you enter on your computer are what is sent to the server? Aside from rigging the ballot files, it is very easy to write programs that alter information; indeed, the infamous computer virus does exactly this. And while antivirus programs are very effective in countering viruses that they know about, they do not protect you against new viruses or other nasty programs on your system. There are many ways to introduce these programs, including having you go to a web site that looks like your bank’s (for example). When you get there, the site has you download something to improve service — and presto!, the malefactors have just put a virus on your system.
With smartphones this is even easier, because we all download apps and run them. The protection most smartphones offer is minimal, so writing an app to cast your vote (suitably doctored, of course) is easy. To get you to download it, the perpetrator sends you an email saying that your election official has set up a web site that you can go to for the app — and when you download it, you really download the doctored app. The election official, of course, did not send this letter.
So you need to be sure that, once you have marked the ballot, the marks you make are the ones that the vote counting system and software will count.
Now let’s say you supply your authentication information to validate who you are, so the election server will send you the right ballot. If you are communicating with a bogus election server, it now has all the information it needs to vote for you — it simply discards your ballot, logs in to the real election server as you, and casts its votes.
How can you be sure you are talking to the right election server? Presumably the county will supply that web address. But if the address comes in an email, it would be easy for some group to send you specially crafted email that displayed the right address as a link — but when you click on the link, it takes you to a completely different web site. This technique is called phishing (or, in this context, spearphishing) and is remarkably effective. This is widely believed to be the way attackers recently took over the Associated Press twitter news feed — and they then posted a fake news story about President Obama being injured by explosions at the White House, causing Wall Street to panic .
Instead, if the address doesn’t come in an email, the attackers may send an email on the night before the election that looks like it comes from the elections office, but is an attack as described above. This would probably trick a large number of people, who would be grateful that the elections office was providing a more convenient way of getting to the web page rather than forcing the user to type the full web address into the browser.
Okay, let's assume you are now talking to the real election server. Once you have cast your votes, they have to be transmitted to the election server. (If you haven't given the authentication information yet, you will have to do so now.) This means the ballot must be sent through the Internet. How does it get there? This is called “routing”, and the route the ballot takes depends on a lot of factors that change constantly. Unless specific methods are used to transmit the ballot, the ballot could get lost, or altered, or someone along the way could read it.
The method that has to be used is the same one that you use to talk to your bank — basically, it is an encrypted connection. These connections require the server to send something called a “certificate” that is signed by a trusted agency. But what agency, and how will you know that the certificate is valid? If the certificate is too old, your browser will tell you, and may refuse to allow you to proceed. But many institutions don’t keep their certificates up to date (it costs money for that), so when you see a notice saying “certificate expired”, you may be talking to the right election server. You don't know. Worse is the situation where the certificate is valid but from an unknown agency. How much can you trust that the agency really checked that the elections office asked for the certificate? Someone once had a highly reputable agency issue a new certificate for Microsoft — but he had no connection to Microsoft . (The problem was discovered quickly, and no harm appeared to have been done. We may not be so fortunate in a Presidential election.)
Okay, now your vote has arrived at the election server. The election server has a lot to do, and must do it correctly. That you have voted now needs to be recorded, and in a way that does not indicate which ballot you case. Somehow the two must be decoupled. So, when your ballot arrives at the server, your name is marked as having voted, and the ballot must be digitally signed by the election server to ensure it is not changed after it is received. Then the ballot must be counted or moved to a system that will do the counting. Any error in all this will compromise your vote, your privacy, and/or the accuracy of the results.
The election server is connected to the Internet, of course — and the Internet that you use to connect to the election server is the same one that people around the world can use to connect to it. This includes attackers who want to break into the server, see the data that is there, and change it or the programs on the server to corrupt the results of the election. And attackers succeed at breaking into very well-maintained computers at high-security government agencies, and covering their tracks well. U.S. defense contractors, the Pentagon, other government agencies, financial institutions, and many major corporations have had their systems and web servers compromised [8-11]. These organizations have lots of security expertise and lots of money. They can hire teams of cybersecurity experts to detect attacks, analyze them, repair or reinstall corrupted systems, (attempt to) recover corrupted data, and figure out how to prevent the attacks from recurring. Of course, these responses take time to carry out. Election offices simply do not have these financial, human, or technical resources. Nor do they have the time, because during an election people vote for a fixed period of time (usually 14 hours or so). Thus, ensuring that an election server connected to the Internet is secure simply is infeasible.
Finally, the procedures to be followed to maintain the server's security must be well thought out and followed meticulously. This is a critical, and often overlooked, point: security experts know that a “system” is not secure; a “system” is secure when used in a specific way. A good way to think of this is to consider how safe a car is. When the driver obeys the traffic laws, she is (usually) quite safe. But if she runs red lights, she is not. Similarly, if the Internet voting system procedures are poorly done, or do not take into account the bad things that can prevent voting or corrupting the ballots, poor "system" security is the least of the problems.
It's tempting to assume that you can simply have someone get a ballot, fill it in, scan it into a PDF (or somehow get it into electronic form), and email it back to the county Elections Office. But the same problems arise. Change “election server” to “email server” above, and the description of how someone returns the ballot is the same whether a web browser on a home computer sends it to the (election) server, or a mail program sends it to the (email) server. And if the ballot is to be downloaded or emailed to the voter by an election official, you again have all the problems of getting the ballot as before.
The quest for Internet voting is a good one; but before the body politic decides that it should be implemented, the dangers and problems of it must be known, discussed openly, and a conscious decision made that the consequences of those problems are acceptable. A pilot study using a real election does not provide this information for several reasons. First, no reputable computer security expert will ever attack a voting system during an election (it’s a crime), so we cannot test the security of the system and its procedures as actually used. Second, enabling detection of some attacks requires a complete record of what happens — including how you voted. Before this type of recording is done, the body politic must decide that how you voted can become known, and accept the attendant risks of vote selling and voter intimidation. Third, a very sophisticated attack could well be undetectable — so how would we know the votes were compromised? And so forth. (See my blog entry for Wednesday, April 24, 2013 for a more detailed explanation of this.)
Whether these difficulties can ever be overcome is a very deep, complex question requiring research in technology, human behavior, social organization, political considerations, and legal matters. What is absolutely certain is that, until they are overcome, we should not conduct civic elections over the Internet.
As for my credentials: I am a professor of computer science at the University of California at Davis, where I am a co-director of the Computer Security Laboratory. I have been in the field since 1978. I have been studying electronic and Internet voting since 2004, when I participated in the RABA study of electronic voting systems for the State of Maryland . I was also one of the co-leaders of the technical part of the California Secretary of State's Top-to-Bottom Review of voting systems certified for use in the State of California . Currently, the National Science Foundation is funding our research on election processes [14,15]. Our group works with election officials in Yolo and Marin Counties.
Any opinions expressed in this note are those of the author, and not necessarily those of anyone else.
[ 1] "3 More Major US Banks Report Possible Cyber Attacks," NBC News (Sep. 27, 2012); available at http://www.nbcnews.com/technology/technolog/3-more-major-us-banks-report-possible-cyber-attacks-6126050
[ 2] M. Lennon, "Wells Fargo Says DDoS Attack Disrupting Online Banking Website," Security Week (Mar. 26, 2013); available at http://www.securityweek.com/wells-fargo-says-ddos-attack-disrupting-online-banking-website
[ 3] B. Acohido, "Amex Latest U.S. Major Bank to Get Knocked Offline," USA Today (April 2, 2013); available at http://www.usatoday.com/story/tech/2013/03/29/american-express-denial-of-service-hack/2030197/
[ 4] "Christie Administration Announces E-Mail and Fax Voting Available to New Jerseyans Displaced by Hurricane Sandy," State of New Jersey (Nov. 3, 2012); available at http://www.state.nj.us/governor/news/news/552012/approved/20121103d.html
[ 5] B. Sullivan, "New Jersey's Email Voting Suffers Major Glitches, Deadline Extended to Friday," NBC News (Nov. 6, 2012); available at http://usnews.nbcnews.com/_news/2012/11/06/14974588-new-jerseys-email-voting-suffers-major-glitches-deadline-extended-to-friday
[ 6] J. Steinberg, "Twitter, The Associated Press, and Phishing: Why Breaches Still Occur," Forbes (Apr. 26, 2013); available at http://www.forbes.com/sites/josephsteinberg/2013/04/26/twitter-the-associated-press-and-phishing-why-breaches-still-occur/
[ 7] "Unauthentic 'Microsoft Corporation' Certificates," CERT Advisory CA-2001-04, Pittsburgh, PA (Mar. 30, 2001); available at http://www.cert.org/advisories/CA-2001-04.html
[ 8] M. Riley & B. Elgin, "China Cyberspies Outwit U.S. Stealing Military Secrets," Bloomberg (May 1, 2013); available at http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html
[ 9] G. Wyler, "Pentagon Admits 24,000 Files Were Hacked, Declares Cyberspace A Theater Of War," Business Insider (July 14, 2011); available at http://www.businessinsider.com/pentagon-admits-24000-files-were-hacked-declares-cyberspace-a-theater-of-war-2011-7#ixzz2SAYL2pqp
 M. Schwartz, "U.S. Labor Dept. Website Hacked, Serves Malware," Information Week (May 1, 2013); available at http://www.informationweek.com/security/attacks/us-labor-dept-website-hacked-serves-malw/240153984
 D. Smith, "Bank of America Hacked By Anonymous: Hackers Leak 'Secrets' About Executives, Salaries, And Spy Activities," International Business Times (Feb. 28, 2013); available at http://www.ibtimes.com/bank-america-hacked-anonymous-hackers-leak-secrets-about-executives-salaries-spy-activities-1107947
 "Trusted Agent Report Diebold AccuVote-TS Voting System," RABA Innovative Solution Cell, Columbia, MD (Jan. 20, 2004); available at http://nob.cs.ucdavis.edu/~bishop/notes/2004-RABA/2004-RABA.pdf
 M. Bishop, "Overview of Red Team Reports," Office of the California Secretary of State (July 2007); available at http://www.sos.ca.gov/voting-systems/oversight/ttbr/red-overview.pdf
 "TC:Medium:Collaborative Research: Technological Support for Improving Election Processes," award from the National Science Foundation tot he University opt California at Davis (Sep. 15, 2009); abstract available at http://www.nsf.gov/awardsearch/showAward?AWD_ID=0905503
 "EAGER: Collaborative: Process-Based Technology to Support Comparison and Evaluation of the Security of Elections," award from the National Science Foundation tot he University opt California at Davis (Oct. 1, 2012); abstract available at http://www.nsf.gov/awardsearch/showAward?AWD_ID=1258577