Big Uncle: looking for the needle in the security data haystack

Big Uncle is the concept of benevolent security. We have been looking in previous posts at the loss of privacy and the positive side of what it means. One application is finding enemies of society.

As the amount of data grows and as it becomes more integrated, a natural trend is to use data mining for more advanced security. The world has barely started down this path. There is much more that can be done to apply existing data technologies and techniques for
• Statistical analysis: deviance reporting (“take the ratio of funds transferred to declared income and investigate the top 2%”)
• Pattern analysis: exception reporting (“no-one has signed on from there before” or “this individual has never come in at night before” or “there are an unusual number of deductions happening from this country”)
• Profiling: (“this individual will probably react this way”)
There are some fundamental technical criticisms of this kind of data mining (leaving aside the social issues for now). As Scientific American put it in an editorial “Every data set has patterns. At issue is whether they mean anything … Terrorism is very rare – which is good for us but bad for data miners. Even with a low error rate, the vast majority of red flags will be red herrings … In short, the data miners commit the fallacy of determinism: they falsely assume that if you just amass enough data, you will know what is going to happen”.

In one year post-9/11, more than 30,000 travellers were mistakenly linked to names on terror watch lists when they crossed the border, boarded commercial airliners or were stopped for traffic violations, according to a report by the Government Accountability Office.

Nevertheless, the application of advanced technologies like neural networks gives us hope of finding the evil needle in the haystack. And the stakes are too high not to try.

As our security capabilities become more advanced, the world is changing to make it all the more urgent that we deploy these capabilities. As we face escalating terrorism, hacking and identity theft, it is clear society must respond to protect itself. But to do so raises challenges. How do we protect without destroying what we seek to protect: freedom and democracy? How can we use the collected data for the benefit of its subjects? How will we respect the right to privacy?