Motivation

• Growing importance of security issues in computing
• Completely secure system may not be feasible
• New security model:
  • prevention
  • detection
  • investigation
  • analyzation
• Intrusion detection systems (IDSs) are becoming important applications
• The more important an application, the more important are testing methods

Methods and tools are needed to

• Test IDS functionality (development)
• Find weaknesses in an IDS (customization, complementary mechanisms)
• Compare different IDSs (requirements)

 

A Concrete Approach

Two papers, published in IEEE software magazines, written by Nicholas J. Puketza, Mandy Chung, Biswanath Mukherjee, Ronald A. Olsson and others:

• "A Methodology for Testing Intrusion Detection Systems"
  (IEEE Trans. Software Eng., Oct. 1996, pp. 719-729)
• "A Software Platform for Testing Intrusion Detection Systems",

 

Testing Approach

• simulation of computer users (both normal and intruders)
• Test case = simulated user session

 

Tested Aspects

Three main aspects are tested:

• Detection range
• Resource usage
• Stress resilence

 

Selection of Test Cases

• Collect as much intrusion data as possible
• Partition the set of intrusions into classes
• Create representative subset: "equivalence partitioning"
• Ideally: equivalence classes in respect to the IDS's detection ability (generally not possible)
• Classification strategies:
  • intrusion technique
  • taxonomy of exploited system vulnerabilities
  • signatures
• Select test cases, especially site-specific

 

Concurrent Attacks

• Intrusive behaviour can be distributed among several separate channels
• Activities on each channel alone seem less suspicious
• Types of concurrent or distributed attack:
  • One source, many targets
  • Many sources, one target
  • Many sources, many targets

 

Limitations of the Methodology

• The IDS is tested only against known attacks (new attacks often similar to known ones)
• May not be possible to identify the set of all possible intrusions

 

Software platform

• Tcl (Tool command language)
• expect tool (control of interactive programs):
  "spawn": create new process
  "expect" waits for specific string patterns from the process
  "send" sends string input to the process

 

Extensions of the expect tool:

• synchronization and communication, needed for distributed scripts; guarantee reproducable results (overall event order otherwise non-deterministic)
• record-and-replay function

 

Tool Limitations

• Not all user activity can be simulated, especially not GUI usage
• The actions that are directly related to the attack should be simulated
• Anomaly-detection based IDSs may not be properly tested.

 

Test Scenarios

• Basic Detection Test: environment free of unrelated activity
• Normal User Test: run non-intrusive scripts to check for false positives
• Resource Usage Test: run various (intrusive or non-intrusive) scripts, observe how many resources the IDS uses (e.g. disk space for logs).
• Stress Tests:
  • Smokescreen Noise: intersperse intrusive commands with non-intrusive ones
  • Background Noise: run intrusion scripts parallel to several non-intrusive "noise" scripts, or one that produces a lot of activity.
  • High Volume Sessions, Intensity: (not really different from the Background Noise Test)
  • Load: run additional programs that take away CPU cycles from the IDS and/or give IDS a lower scheduling priority
  In all cases: compare IDS output with that from the Basic Detection Test.

 

Actual Experiments

• Tested System: UC Davis' Network Security Monitor (NSM) monitors network traffic, recognizes individual computer-to computer connections and assigns numerical warning values ranging from 0 to 10
• Platform: Sun SPARCstation 2, Connected to the CS LAN segment at UCD.

 

Results

• Some types of intrusive behaviour were not detected at first
• NSM could be configured to detect them
• Load Stress Test: loads above 4 caused the IDS to miss network packets
• NSM itself reported this, and showed the percentage of missed data: about 10% at load 10, 40% at load 14
• Concurrent Intrusion Test Cases:

  Intrusion type	Max. warning value
  	Sequential	Concurrent
  Transmission of password file	7.472	7.472
  Password cracking	3.160	3.160
  Password guessing	8.722	7.785
  Exploiting buffer overrrun	7.472	4.972

 

Future Plans

• Develop suite of intrusion test cases
• Eventually: a comprehensive "benchmark suite"

Evaluation of the Paper

 

Strengths

• Lots of background explanations (motivation for ID, approaches), easy to understand
• Basic introduction to the field
• Practically no prior knowledge necesary

 

Shortcomings

• Practical side of research focused on NSM, limited view
• Some important issues were left out
  • Network-based vs. host-based
  • Different protocols
  • Low-level DoS or surveillance attacks, installing backdoors, removing traces
  • Encrypted sessions

 

Real-World Aspects