Customer Success: BLENDING ART & SCIENCE

Since 2002, Security Innovation

has been testing software applications that the world runs on — from breathalyzers and mobile applications to enterprise ERP systems and cloud infrastructure. Our experts augment the use of commercial and open-source automated security tools with specifically crafted methodologies and tools to deliver the most efficient assessments and maximize risk reduction in our customer’s technologies. Tools, like humans, excel at certain tasks. Over the years, we’ve formulated assumptions around when to use tools versus manual techniques, and often conduct experiments comparing the two so that we can optimize our own resources and advise our clients to do the same.

While it is possible to dig a ditch with a teaspoon, a backhoe is far more efficient. However, a backhoe is overkill when pouring sugar into a cup of coffee. The same is true with application security testing. Both automated tools and manual effort play a critical role in an organization’s application security program because they find different types of vulnerabilities. Tools find common problems quickly and humans hunt down vulnerabilities that elude tools. The challenge is finding the optimal balance of both to achieve breadth and depth of coverage in the most efficient manner – and ensure test efforts are commensurate with each application’s level of risk and complexity.

For this study, our team used a variety of scanners, utilities, and manual techniques to test twenty large-scale enterprise applications. Elements, such as time, were recorded to find vulnerabilities using manual and automated techniques; vulnerability type found by each; and the specific tool type used for each vulnerability found. The goal was to understand which techniques were most effective:

• in which Testing Conditions
• by Type of Application
• by Vulnerability Class

AUTOMATION

The goal of automated tools, especially scanners, is to attain a breadth of coverage quickly. Automation can streamline pattern matching and identify recognizable vulnerabilities with speed and accuracy humans can’t match. Tools fall short with false positives, false negatives, and missed coverage due to constraints such as authentication for role-sensitive applications or the use of technology that eludes “spidering” or “crawling” in web applications. The result is often an incomplete and enumerated attack surface. Tools also can’t provide business or risk context and only offer high-level best practices versus prescriptive and contextual remediation guidance that an expert can provide. Additionally, knowing how to use a tool effectively and its strengths and limitations will maximize its utility.

MANUAL TESTING

The goal of manual security testing is to achieve the depth of testing with a focus on finding serious and complex security defects that automation cannot. The tester interacts with an application in all of the different user roles (end-user, administrator, etc.) to force the application to behave in ways it was never intended. Manual security testing is an important part of a security testing regimen to exercise applications in ways automation cannot. For example, interacting with Flash elements that scanners are blind to. Manual testing is slower than automated testing but necessary to fill code coverage gaps left by tools.

STUDY SET-UP AND OBJECTIVES

For this internal study, the Security Innovation team selected twenty large-scale, enterprise applications with varying complexities that represent a spectrum of operational applications commonly deployed:
eCommerce Web sites, knowledge and support portals, social media platforms, CRM, and other business
applications. The methodologies, techniques, and specific tools used during testing were documented
along with every defect found. The goal was to understand which technique is most effective at uncovering
vulnerabilities based on application and vulnerability type.

Several variables were identified that could affect outcomes and require future data collection
and correlation:

• Tool/support for the application under test (AUT)
  A tool that is unaware of a rich user interface, for example, will probably miss entire classes of security defects. This is why it’s important to select the best tool for the job at hand
• Consistency of tool operator for each test
  Changing operators can impact even simple tasks when using different automation tools for the same AUT. Whenever possible, this control was considered a variable.
• Time
  Automated tools are commonly used early on in testing to “crawl” an application, whereas
  manual testing in those areas is limited to save time. In order to measure the independent
  capabilities of both approaches, each application was put through a consistent battery
  of tests, acknowledging the superior coverage capabilities of automation in certain areas.
  Mindful of these variables, tests were conducted in a way that would not provide an unfair
  advantage to one tool or approach over another.

RESULTS DETAIL

Unique Defects

Unique defects are those that are uncovered by either automation or manual testing, but not by both.

Across the 20 applications, a total of 117 defects were uncovered. Out of these, manual testing uncovered 59 unique defects whereas automation uncovered one.

Total Defects

The total number of defects uncovered by each approach is related to the number of unique defects found. Manual testing uncovered 116 of 117 defects, whereas automation uncovered 58. The effectiveness of automated testing depends on the type of applications being tested. Given the wide range of applications tested, the team expected automation to uncover approximately half of the total defects. Results were in line with expectations: automation detected 49.57% of the total number of vulnerabilities. Given the absence of time constraints, the team expected manual testing to uncover almost all of the defects. As expected, this technique uncovered 99.15% of them.

Types of Defects

30 types of defects were uncovered between the sample applications. Manual testing found 100% of the
vulnerabilities, whereas automated testing found about half. As anticipated, manual testing uncovered
uncommon vulnerabilities that tools cannot like client-side validation.

Only one Business Logic defect was found during manual testing and as expected, automation was unable
to detect this. Automation fell short of our expectations by finding fewer information disclosure defects – manual testing uncovered 13 instances as compared to 8 found by automation.

The following chart shows the number of defect types detected by each approach in descending order
and highlights three key findings:

1. All of the 30 defect types were uncovered during Manual testing.
2. Automated testing never yielded a higher number of defects for any defect type.
3. There are a large number of defect types that were not found using automation.

All of the 30 defect types were uncovered during Manual testing. Automated testing never yielded
a higher number of defects for any defect type.

Common Defects

The table below lists 14 defects types uncovered by both automation and manual testing. It also shows
the number of defects each approach found and the average time multiplier manual testing took.

Figure 5: efficacy of manual testing vs. automated by vulnerability type

STUDY USE CASES

Since our team has often found automation to provide credible leads towards uncovering session mismanagement, we were surprised to find that it did not uncover any during our study. Below is a detailed use case demonstrating the inability of automation to uncover session mismanagement vulnerabilities in this specific instance.

Security Innovation discovered several key areas where manual testing showed advantages
over automation:

• Weak Session Management
• Vulnerable File Upload
• XSS (Cross-site scripting)

Weak Session Management

During automated testing of a CRM application, the results did not find Session Management Weaknesses. Manual testing was then performed, discovering a Direct Object Access (DOA) vulnerability in a POST parameter not discovered in the prior automated test. When this previously missing parameter was added to a request, the user was given access to previously unavailable resources.
This indicated that the application:

1. Was not performing proper authorization checks.
2. May was not securely handing its other POST parameters either.

Determined to find other vulnerable parameters, further manual testing was performed on the sensitive
parameter types, such as those maintaining user identity and environment. Any tampering of those
parameters resulted in the server session being terminated as well as the browser being escorted back
to the login page. After tampering, deleting, and injecting forbidden payloads into seven other parameters, the team found a post-authentication parameter named “department” to be the most resistant to
change – there was something special about it and the server killed the session if you tried to change/
edit it. That post-authentication parameter was saved and added to an authentication request. Under
a special test case where the department parameter was the lone parameter in the authentication
request (no username, no password) it was discovered that the application server lets the user in.

Automation can be programmed to perform very complicated checks, but not intuition, like the example
described above. Such testing requires analysis and the use of variables across multiple requests and
responses, potentially using them in a context where they are not supposed to exist, and therefore often
with unanticipated consequences. Such defects are almost always critical in severity and usually require
manual testing to discover.

Vulnerable File Upload & XSS

Many applications allow users to upload files to the application server. In our study, 3 of the 20 applications were found to have vulnerabilities in their upload functionality. Simple file upload features can usually be well evaluated via automation. Automation is also very good at identifying XSS defects based on a dataset of exploit strings targeting known weaknesses in popular defense mechanisms as well as finding the same XSS defect in multiple locations of the application. This is an area where a manual endeavor would be too time-consuming to be effective. Below is a use-case on file upload and XSS:

This example demonstrates a combo-bug (one that exposes two types of defects and where one would
be difficult to find without the other), that is much more common than one may think. The team tested
a file upload feature that implements special functionality to prevent malicious file uploads. However,
this special security functionality has a weakness of its own, which not only allows malicious files to be
uploaded but also can exploit XSS defects that may have otherwise gone unnoticed.

The application also allows the user to upload PNG or JPEG files and instead of relying on filename
extensions, the application checks the uploaded le for valid PNG or JPEG headers. We observed the
application accepting a PNG image even when the file was named test.html.

Upon further investigation, the team discovered that the uploaded file was not being assigned system-generated names and was actually saved as test.html on the server. After trying to upload an HTML file,
the application rejected the upload. This told us a few things:

1. The application did not care what the extension of the uploaded file was
2. The application checked the header of the uploaded file.

The objective was to upload HTML content on the server, which could then be reviewed by other users
of the system. Based on observations, the team initiated the upload of a valid PNG file, intercepted the
request in a proxy, and modified the image content by adding HTML content with a Javascript payload in
the body of the PNG file. We left the PNG headers untouched and changed the file name from the pic.png topic. HTML (having the correct file extension was necessary for the client browser to execute it.) The
application accepted the modified image content because the image header was valid and the server did not check the file name. Navigating to the image location caused the application to execute the Javascript payload embedded within the image — game over!

The endless possibilities of what an application can do with uploaded files, coupled with the negative
actions that a malicious file may cause on the server or to other users of the application, likely ensures
that it will be some time before automation can be programmed to detect such complex file upload vulnerability defects.

CONCLUSION

This study examines real-world test scenarios to reaffirm the strengths and limitations of automated and manual testing, which includes the following observations:

• When used and interpreted properly, automation provides a fast and efficient method for flagging common vulnerabilities. However, it yields only partial code coverage, cannot detect certain vulnerability classes, and can lose efficiency gains with an abundance of false positives.
• Humans can leverage expertise and creativity to hunt down complex, business logic, and other stealth-like vulnerabilities; however, this can be time-consuming so they need to focus on critical areas.
• Different tools will find different types of vulnerabilities in different types of applications – ensure your teams know what their tools are capable of.
• Leveraging automated tools early on to identify low-hanging fruit and hot spots provides a useful jumpstart for testers who can then determine where they need to dig deeper.

In the absence of unlimited time and resources, organizations must find the right mix of breadth (automation) and depth (manual attacks). This is driven by application complexity and risk. Harnessing the power of both automated and manual testing will optimize results, reduce false positives, and shorten the time necessary for remediation.