Refer to Setting up Clusterfuzz if you are interested in setting up a local instance of cluster fuzz to run fuzzing on your own OEMCrypto implementations on linux.

Objective

Run fuzzing on OEMCrypto public APIs on linux using google supported clusterfuzz infrastructure to find security vulnerabilities.

Design Document - https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit?usp=sharing

Fuzzing at google - go/fuzzing

Monitoring

Cluster fuzz statistics

Performance of OEMCrypto fuzz binaries running continuously using cluster fuzz infrastructure can be monitored here.

The options to select are Job type: libfuzzer_asan_oemcrypto and Fuzzer: fuzzer name you are looking for

Example: load_license_fuzz

Issues filed by clusterfuzz - Fixing those issues

Any issues found with the fuzz target under test are reported by clusterfuzz here.
The bug will have a link to the test case that generated the bug. Download the test case and follow the steps from testing fuzzer locally section to run the fuzzer locally using the test case that caused the crash.
Once the issue is fixed, consider adding the test case that caused the crash to the seed corpus zip file. Details about seed corpus and their location are mentioned in this section.

Corpus

Once the fuzzer scripts are ready and running continuously using clusterfuzz or android infrastructure, we can measure the efficiency of fuzzers by looking at code coverage and number of new features that have been discovered by fuzzer scripts here Fuzz script statistics.

A fuzzer which tries to start from random inputs and figure out intelligent inputs to crash the libraries can be time consuming and not effective. A way to make fuzzers more effective is by providing a set of valid and invalid inputs of the library so that fuzzer can use those as a starting point. These sets of valid and invalid inputs are called corpus.

The idea is to run OEMCrypto unit tests and read required data into binary corpus files before calling into respective OEMCrypto APIs under test. Writing corpus data to binary files is controlled by --generate_corpus flag.

Build OEMCrypto unit tests to generate corpus

Install Pre-requisites
```
$ sudo apt-get install gyp ninja-build
```
download cdm source code (including ODK & OEMCrypto unit tests):
```
$ git clone sso://widevine-internal/cdm
```

Build OEMCrypto unit tests and run with --generate_corpus flag to generate corpus files:

$ cd /path/to/cdm/repo
$ export CDM_DIR=/path/to/cdm/repo
$ export PATH_TO_CDM_DIR=..
$ gyp --format=ninja --depth=$(pwd) oemcrypto/oemcrypto_unittests.gyp
$ ninja -C out/Default/
$ mkdir oemcrypto/test/fuzz_tests/corpus/<fuzzername>_seed_corpus
# Generate corpus by excluding buffer overflow tests.
$ ./out/Default/oemcrypto_unittests --generate_corpus \
  --gtest_filter=-"*Huge*"

There can be lot of duplicate corpus files that are generated from unit tests. We can minimize the corpus files to only a subset of files that cover unique paths within the API when run using fuzzer. Run following command to minimize corpus.

$ cd /path/to/cdm/repo
# build fuzzer binaries
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
$ mkdir /tmp/minimized_corpus
# minimize corpus
$ ./out/Default/<fuzz_target_binary> -merge=1 /tmp/minimized_corpus \
  <FULL_CORPUS_DIR>

To avoid uploading huge binary files to git repository, the minimized corpus files will be saved in fuzzername_seed_corpus.zip format in blockbuster project's oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need permissions for blockbuster project, contact widevine-engprod@google.com.
```
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
    <destination_path>
```

Testing fuzzer locally

Corpus needed to run fuzz tests locally are available in blockbuster project's oemcrypto_fuzzing_corpus GCS bucket. If you need permissions for this project, contact widevine-engprod@google.com. Download corpus.
```
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
  <destination_path>
```
Add flags to generate additional debugging information. Add '-g3' flag to oemcrypto_fuzztests.gypi cflags_cc in order to generate additional debug information locally.

Build and test fuzz scripts locally using following commands. The build script builds fuzz binaries for opk implementation.

$ cd PATH_TO_CDM_DIR
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
$ mkdir /tmp/new_interesting_corpus
$ ./out/Default/fuzzer_binary /tmp/new_interesting_corpus \
    /path/to/fuzz/seed/corpus/folder

In order to run fuzz script against a crash input, follow the above steps and run the fuzz binary against crash input rather than seed corpus.
```
$ ./out/Default/fuzzer_binary crash_input_file
```

Adding a new OEMCrypto fuzz script

In order to fuzz a new OEMCrypto API in future, a fuzz script can be added to oemcrypto/test/fuzz_tests folder which starts with oemcrypto and ends with fuzz.cc(GCB build script for oemcrypto fuzzers expects the format).
In the program, define the function LLVMFuzzerTestOneInput with the following signature:
```
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
    <your test code goes here>
    return 0;
}
```
Note: Make sure LLVMFuzzerTestOneInput calls the function you want to fuzz.
Add a new target to oemcrypto_fuzztests.gyp file and follow instructions in testing fuzzer locally to build and test locally.

Building OEMCrypto fuzz scripts and uploading them to Google Cloud Storage:

We are using Google Cloud Buid (GCB) in order to setup continuous integration which uploads OEMCrypto fuzz binaries to Google Cloud Storage. GCB expects build script in form of a docker image that is uploaded to Google Container Registry(GCR).

The cloud build scripts (docker images) for widevine projects are here

Refer to README of the project to setup a new docker image and uploading the image to GCR.
Git on borg repository needs to be integrated with GCB and a git trigger needs to be set up in order to achieve continuous integration. Git trigger will mention which docker image the GCB needs to use in order to build fuzz binaries. GCB searches for docker images from GCR.

Design document lists the steps to create a git trigger.

Adding a new fuzz script to the build script:

As long as a new fuzz script is added which starts with oemcrypto and ends with fuzz, the build command can be added to build_oemcrypto_fuzztests. GCB script uses build_oemcrypto_fuzztests script to build fuzz binaries and make them available for clusterfuzz to run continuously.
If the new fuzzer cannot follow the naming convention OR GCB script needs to be updated for any other reason, refer to this section section.

Generate code coverage reports locally

Code coverage is a means of measuring fuzzer performance. We want to make sure that our fuzzer covers all the paths in our code and make any tweeks to fuzzer logic so we can maximize coverage to get better results.

Coverage reports for git on borg project is not automated and needs to be generated manually. Future plan is to build a dashboard for git on borg coverage reports.

Generate code coverage reports using script from Google cloud build

A docker image with script to generate code coverage reports for oemcrypto fuzz scripts is linked with a GCB trigger oemcrypto-fuzzing-code-coverage-git-trigger. More information about clang source based coverage can be found here.
This trigger when invoked compiles oemcrypto fuzz scripts with clang source based code coverage enabled, downloads latest corpus from cluster fuzz for the respective fuzzer, generates and uploads code coverage html reports to GCS.
The trigger can be invoked manually using cloud scheduler oemcrypto_fuzzing_code_coverage_reports.
In order to generate latest code coverage reports from master branch, go to pantheon->cloud scheduler->oemcrypto_fuzzing_code_coverage_reports and click on RUN NOW button.
The above step should invoke a google cloud build. Go to cloud build console and find latest build job with Trigger Name oemcrypto-fuzzing-code-coverage-git-trigger.
Once the build job is successful, latest code coverage reports can be downloaded from GCS. The coverage report folder uploaded to GCS is appended with timestamp.