I have listed some notes, based on my personal experience so far, about working with some limited direct and indirect bug bounty programs and performing vulnerability assessment. I hope this is useful for the readers; please feel free to contact me on my Twitter (@irsdl) to discuss any of these if you wish.

In my humble opinion, direct bug bounty programs and vulnerability assessments are both necessary although may not match some business models where one can be ineffective in practice. For example, a bug bounty program may not work for sensitive platforms or targets that are only accessible from certain locations or IP addresses. On the other hand, a broad time limited vulnerability assessment may not work for large open source applications that have been around and tested for quite some time. As far as I am aware, bug bounty programs cannot offer any assurance to the customers based on the regulations at the moment, but this may change in future.

By the way, I am not discussing indirect bug bounty programs in which a third party vendor pays for vulnerabilities of other vendors’ products.

My notes about each of these solutions are as follows:

Direct Bug Bounty Programs

Pros:

• More testers to cover more targets and to find more vulnerabilities. However, normally in practice only attractive programs receive high-quality vulnerabilities.
• More time or unlimited time to cover more targets and to find more vulnerabilities. Again, this happens when a program is attractive to highly skilled testers and treats them well.
• They can change the scope dynamically at any time. That said, this can cause problems between the program and reporters who have already started looking at targets that are in the scope.
• Only pays for valid/in-scope findings (not saying it cannot be expensive though).
• Can prevent breaches by buying bugs for a good price.
• It can help companies with a huge attack surface to identify hidden issues that were not identified throughout previous assessments.
• Works well for open source or non-profitable applications when sponsors such as Internet Bug Bounty pays for it.
• Customers may feel safer when a company security model is mature and is handling reported security issues gracefully.
• Encourage responsible disclosure culture.
• Testers might care about the patching process when their bounties depend on it so they might be eager to help with the retest. However, this might make program less attractive to those who want to be paid quickly.
• It helps security researchers to legally test the targets and makes the responsible disclosure process simpler for anyone who might want to report a security issue.

Cons:

• It is a competitive market. If a few companies pay more for vulnerabilities, it will make other programs less attractive and will reduce their quality. Therefore, quality in the bug bounty world is relative and those programs that cannot afford paying as much as giant companies for their security issues will be on the losing end.
• Unless a program is attractive enough and is restricted to top bug bounty reporters in the world, the majority of testers in bug bounty are not experienced penetration testers and have less skills and are only after the low hanging fruits. As a result, there are high amount of false positives, fake, or duplicates from less experienced testers or those who have not read the bug bounty policy properly. In practice, less than %5 of reported issues can be valid for a popular program that increases human mistakes in handling real issues and put a high pressure on the triage team (https://www.synack.com/2017/05/05/managed-bug-bounties-quality-is-in-the-secret-sauce/).
• Highly skilled penetration testers who are new to the bug bounty platforms cannot join private programs that are normally more attractive as they do not have enough points on the system. As a result, when most attractive programs are private, they can never climb up the point-based ladder without spending their time on less attractive programs which is not ideal.
• The main problem with the penetration testing assessments is their time constraints because of limited budget. A Bug bounty program with limited time and limited scope is not even close to a penetration testing assessment and might not be useful at all unless it can attract good and strong testers.
• Free or low paid programs can give customers false sense of security as they are not attractive to strong testers.
• It is impossible to stop all the testers at once when needed as the communication method is normally via email and can be slow.
• Newly developed functionality or less visible ones normally remain hidden to the testers. Therefore, a new section may remain untested while other more visible sections are being tested over and over again by different testers.
• Testers are not normally known and therefore it is more difficult to trace their activities if needed in compare with a penetration testing. Attackers may also claim that they were after responsible disclosure.
• Hard to design the right policy with a correct scope (requires knowledge of the whole system). Badly designed scope can reduce the effectiveness of bug bounty as well as damaging the vendor’s reputation regarding its security.
• It may give attackers the opportunity to legally assess an organisation while having no intention to report the most serious issues. However, it is possible in some platforms to limit a penetration testing to certain testers who are better known because of their longer activities (this may make a bug bounty program private).
• It may cause denial of service if a system is not ready for a heavy load as testers may use automated scanners even if using it is forbidden. Although systems are constantly being scanned automatically on the Internet, defining a bug bounty program on them will certainly increase the amount of scanners being used on the targets.
• A lot of dispute over prioritising and paying for a vulnerability. Reporters constantly compare their issues with each other.
• Mistrust over duplicate issues due to the lack of transparency in most of programs.
• Reputation damage over solving issues badly (e.g. by ignoring the reports, not paying the right amount, or keeping the reporter in the dark while patching an issue)
• Sorting the tax money is harder outside of the US as they use different currencies. Options in the UK are currently limited to use accountants, commercial software, or paper-based forms by using the correct currency conversion chart (if you receive them in different currencies than British Pound).

Vulnerability Assessment (commonly known as penetration testing*)

Pros:

• An official report or certificate of security assessment can be produced by the penetration testing company. This can be presented to partners or other organisations when a proof of security assessment is needed.
• A higher quality report in compare with bug bounty reports in general.
• Full attention on the specific target as days and scope are pre-defined and will not be deviated from.
• A responsible company that can be contacted and is accountable for their assessment and their testers.
• Level of support after the assessment in order to rectify the issues.
• No false-positives due to the consultants’ experience and the QA process. However, human errors can happen but at least it will be investigated and the report will be updated when it is incorrect.
• Better communication between the vendor and the testers in order to resolve any possible issues.
• Regardless of the number of issues identified by the penetration testers, all will be presented to the client for the same fixed cost.
• Testing and progress visibility
• Good assurance for the customers regarding the assessment coverage, quality of testing, and an accountable testing company.
• Accepted by different regulations such as by PCI.

Cons:

• Limited time of testing may reduce the likelihood of discovering well-hidden vulnerabilities that require more research or different skillset.
• Vendors should normally pay additional fees for a retest when required as the retest might not be part of the original assessment.
• Assigning more time to an assessment can be very expensive
• Human factor plays an important role as number of testers on each assessment are limited. For example, experience and knowledge of assigned testers on an assessment and how they feel during the test plays an important role in identifying security issues! However, in a bug bounty program anyone can look at the system at the same time or later in time.
• Difficult to change the scope during an assessment or assign more testers upon a short notice as resources are normally limited and testers’ specialities and skills are different.

I do like Bug Bounties! For me, it is like catching a fish! 🎣

With all that, bug bounty is still young and has a great future ahead!

A number of the following suggestions have already been implemented in some of the well-known bug bounty platforms but not in all.

Recommendations to Improve Bug Bounty Platforms **:

• Giving people access to previously reported issues when they have reported a duplicate issue to make it clearer (similar to what Mozilla does in their issue tracker). If the duplicate issues have been received from another source, perhaps an additional tracking number and name of the finder can help people understand that the vendor is not cheating on them by announcing an issue as a duplicate.
• Registering testers’ mobile number to keep them up to date with the latest changes when they sign up for a certain bug bounty program. The scope can only be visible to those who have signed up and confirmed a text message. Testers can then opt out if they are not interested anymore in receiving text messages about scope changes or any issue with the testing.
• A verification process (perhaps in different levels) in which the consultants can provide their identification documents, certificates, address, and CV to the bug bounty platform in order to make their account verified. This can be used if vendors only want to pass their bug bounty to trusted/known people based on their verification level or their certificates (not just their bug bounty reputation).
• A VPN service for the verified testers so they can always use certain IP addresses while testing. This gives vendors the opportunity to design appropriate firewall rules and to make their testing system only accessible to certain IPs.
• A survey for those who have reported vulnerabilities measuring their investment in time or other resources, their view on the specific bug bounty program’s performance alongside possible vendor’s view about the tester. This data can be publicly available upon approval from the tester and the vendor.
• A transparent platform to show statistics of every bug bounty programs. This should include but not be limited to the following information selectable/searchable by date: number of reported/accepted/duplicate/invalid issues with their severity, number of top 100 testers (or if there is another measure to show strong testers) reported issues, average amount of days to fix the issues/pay the bounty, average amount of bounty based on issue severity, number of accepted but unfixed issues.
• Defining a consultancy system for the vendors so they can refine their scope based on feedbacks from the testers.
• Creating a centralised platform to include the bug bounty targets for research purposes. It is often the case the people go to a great length to obtain a list like this that can be used for research purposes. For instance, to check a specific vulnerability on targets that use a certain technology.
• Working with the governments to make it easier for testers to pay their tax or to give the money to charities outside the US.

Improvements for Bug Bounty Program Owners **:

A bug bounty program will be successful when it can attract strong testers to spend some time assessing the targets. It is important to design and perform it in a way that it encourages testers to come back to spend more time and report more vulnerabilities.

The following list shows some of the features that a bug bounty program should have to be popular among testers:

• Clear scope and exclusion list
• Quick acknowledgement of receiving issues
• Quick decision on issues validity for the bounty
• Reasonably short patching time
• Quick pay-out (e.g. on validity rather than resolution when patching will take time)
• Generous rewards for the findings, good quality reports, and help in retesting
• Surprise rewards on nice findings and reports even when they are out of the scope
• Supporting multiple payment options
• Polite and professional triage team that can handle difficult cases gracefully

 

 

* IMHO, they are different though but that discussion is not the topic!
** whoami to comment on this? Just an experienced security researcher like many others.