hin255 - Fotolia

Track users with correlated data from multiple log files

Managing rising server, app and user counts requires using dozens of incompatible log files. Arizona State turned to Splunk to create a unified, correlated data view from the jumble.

Which American university is ranked tops for innovation? Stanford? MIT? You'd be wrong.

When it comes to innovation in academia in the United States, Arizona State University sits atop the prestigious U.S. News & World Report rankings for both 2016 and 2017. Stanford University and the Massachusetts Institute of Technology could do no better than second and third, respectively.

As ASU innovates by extending its technological reach into artificial intelligence, augmented reality, machine learning and cognitive computing, the school's computing infrastructure, its mix of cloud-based and on-premises resources, including mainframes, continues to grow. Keeping track of thousands of servers and applications, along with activities of tens of thousands of student and faculty users -- and then collating and correlating all of that information for viewing and tracking in a single, unified auditable view -- became a top priority, according to Chris Kurtz, a system architect in ASU's department of university analytics and data services.

A matter of correlated data

Diagnosing potential security issues or locating a broken multisystem integration necessitates looking at log files from each system involved in context, necessitating a consistent correlated data view.

Chris Kurtz, system architect, Arizona State UniversityChris Kurtz

"The problem we needed to solve is getting disparate logs from Windows, Linux, firewalls, switches, and more all in one place that's easily searchable and can be audited and distributed in a protected environment," Kurtz said. It's all about obtaining logs from operational servers and network devices, and putting the information into the correct order chronologically or by user, to create correlated data for personnel charged with overseeing IT infrastructure operations and security. Think of it as an aggregation engine. "You want to see individual user logs and how that user transits across systems," Kurtz said.

Those individual users add up, according to Kurtz. With more than 80,000 enrolled students and 20,000 faculty and others, ASU has a lot to keep track of. To aid with the machine data collection and collation of logs, ASU turned to Splunk Inc., a San Francisco provider of software that aims to transform machine-generated data into what the company calls "operational intelligence."

You want to see individual user logs and how that user transits across systems.
Chris Kurtzsystem architect, Arizona State University

The collated and correlated data is necessary to give IT personnel clues where to look when something goes wrong, according to Kevin Davis, Splunk's vice president of public sector.

"Moving at the speed of any IT system and the internet, systems become massive and complex and we tend to create silos," he said. Silos hinder visibility across the totality of IT systems, making it difficult to find problems or to track a particular user's travels. "It's not something you think about it until something goes wrong."

Kevin Davis, vice president of public sector, SplunkKevin Davis

When something does go wrong, it's somebody's job -- or the job of a lot of people -- to figure out what went wrong and get the system back up and running as fast as possible. That's job No. 1, Davis said. "After that, you can finally do a bit of triage."

Larry Ponemon, chairman, Ponemon InstituteLarry Ponemon

That job is not getting any easier, said Larry Ponemon, chairman of the Ponemon Institute, a Traverse City, Mich., research firm specializing in security. "There are so many devices, trying to stop the craziness and get[ting] a listing is more than a herculean task," he said, adding that the proliferation of IoT devices is resulting in many more device types to track, raising the difficulty level.

Kurtz said ASU first looked at several products, including the ArcSight enterprise security manager from Hewlett Packard Enterprise and Elasticsearch, offered as a service by Amazon Web Services, before settling on the Splunk software in 2012. The university now uses it to correlate infrastructure issues with user activity, something that seems obvious, but which is difficult to do when each subsystem's log exists in vacuum.

Next Steps

Yelp deploys Splunk to manage data growth

Splunk combines multiple data sources through data synthesis

Secure stacks require a collaborative architecture

Dig Deeper on Cloud governance