Verizon Network Security Services collects netflow from internal devices, edge routers and the Internet backbone. The group is also the central repository for logs in hundreds of formats and thousands of machines, including firewalls, IDS engines, web proxies, SNMP managers, BGP aggregators, DNS servers and desktops. Deriving useful information from all this data is a task shared by the data owners, repository operators and security analysts.
In this presentation we will go over the growth of the Verizon Network Security data repository; the infrastructure in place that receives and processes 100GB of data an hour, including two billion flows. We will also cover some of the open source, commercial and homegrown software that helps the security, network planning, and network performance teams gain insight into the current state of networks from local offices to the Internet.
We will also discuss some of the challenges encountered along the way, various attempts to make searching flow faster, and some recent developments using machine learning to identify attacks on the network.
