Run fuzzing on OEMCrypto public APIs on linux using google supported clusterfuzz infrastructure to find security vulnerabilities.

Design Document - https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit?usp=sharing

Fuzzing at google - go/fuzzing

Monitoring

Cluster fuzz statistics

Performance of OEMCrypto fuzz binaries running continuously using cluster fuzz infrastructure can be monitored here.

The options to select are Job type: libfuzzer_asan_oemcrypto and Fuzzer: fuzzer name you are looking for

Example: load_license_fuzz

Issues filed by clusterfuzz - Fixing those issues

Any issues found with the fuzz target under test are reported by clusterfuzz here.
The bug will have a link to the test case that generated the bug. Download the test case and follow the steps from testing fuzzer locally section to run the fuzzer locally using the test case that caused the crash.
Once the issue is fixed, consider adding the test case that caused the crash to the seed corpus zip file. Details about seed corpus and their location are mentioned in this section.

Corpus

Once the fuzzer scripts are ready and running continuously using clusterfuzz or android infrastructure, we can measure the efficiency of fuzzers by looking at code coverage and number of new features that have been discovered by fuzzer scripts here Fuzz script statistics.

A fuzzer which tries to start from random inputs and figure out intelligent inputs to crash the libraries can be time consuming and not effective. A way to make fuzzers more effective is by providing a set of valid and invalid inputs of the library so that fuzzer can use those as a starting point. These sets of valid and invalid inputs are called corpus.

The idea is to run OEMCrypto unit tests and read required data into binary corpus files before calling into respective OEMCrypto APIs under test. Writing corpus data to binary files is controlled by --generate_corpus flag.

Build OEMCrypto unit tests to generate corpus

Install Pre-requisites
```
$ sudo apt-get install gyp ninja-build
```
download cdm source code (including ODK & OEMCrypto unit tests):
```
$ git clone sso://widevine-internal/cdm
```

Build OEMCrypto unit tests and run with --generate_corpus flag to generate corpus files:

$ cd /path/to/cdm/repo
$ export CDM_DIR=/path/to/cdm/repo
$ export PATH_TO_CDM_DIR=..
$ gyp --format=ninja --depth=$(pwd) oemcrypto/oemcrypto_unittests.gyp
$ ninja -C out/Default/
$ ./out/Default/oemcrypto_unittests --generate_corpus

To avoid uploading huge binary files to git repository, the corpus files will be saved in fuzzername_seed_corpus.zip format in blockbuster project's oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need permissions for blockbuster project, contact widevine-engprod@google.com.
```
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
    <destination_path>
```

Testing fuzzer locally

Corpus needed to run fuzz tests locally are available in blockbuster project's oemcrypto_fuzzing_corpus GCS bucket. If you need permissions for this project, contact widevine-engprod@google.com. Download corpus.
```
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
  <destination_path>
```
Add flags to generate additional debugging information. Add '-g3' flag to oemcrypto_fuzztests.gypi cflags_cc in order to generate additional debug information locally.

Build and test fuzz scripts locally using:

$ export CXX=clang++
$ export CC=clang
$ export GYP_DEFINES="clang=1"
$ cd /path/to/cdm/repo
$ export PATH_TO_CDM_DIR=.
$ gyp --format=ninja --depth=$(pwd) \
    oemcrypto/test/fuzz_tests/oemcrypto_fuzztests.gyp
$ ninja -C out/Default/
$ mkdir /tmp/new_interesting_corpus
$ ./out/Default/fuzzer_binary /tmp/new_interesting_corpus \
    /path/to/fuzz/seed/corpus/folder

In order to run fuzz script against a crash input, follow the above steps and run the fuzz binary against crash input rather than seed corpus.
```
$ ./out/Default/fuzzer_binary crash_input_file
```

Adding a new OEMCrypto fuzz script

In order to fuzz a new OEMCrypto API in future, a fuzz script can be added to oemcrypto/test/fuzz_tests folder which ends with _fuzz.cc.
In the program, define the function LLVMFuzzerTestOneInput with the following signature:
```
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
    <your test code goes here>
    return 0;
}
```
Note: Make sure LLVMFuzzerTestOneInput calls the function you want to fuzz.
Add a new target to oemcrypto_fuzztests.gyp file and follow instructions in testing fuzzer locally to build and test locally.

Building OEMCrypto fuzz scripts and uploading them to Google Cloud Storage:

We are using Google Cloud Buid (GCB) in order to setup continuous integration which uploads OEMCrypto fuzz binaries to Google Cloud Storage. GCB expects build script in form of a docker image that is uploaded to Google Container Registry(GCR).

The cloud build scripts (docker images) for widevine projects are here

Refer to README of the project to setup a new docker image and uploading the image to GCR.
Git on borg repository needs to be integrated with GCB and a git trigger needs to be set up in order to achieve continuous integration. Git trigger will mention which docker image the GCB needs to use in order to build fuzz binaries. GCB searches for docker images from GCR.

Design document lists the steps to create a git trigger.

Adding a new fuzz script to the build script:

In order to update build script such as adding a new fuzzer to build script, we need to update the build script in docker image from cloud repository. [Build script.](https://widevine-internal.googlesource.com/cloud/+/refs/heads/master/docker /cloud_build/oemcrypto/release/ubuntu/fuzz/build.sh)

Add the new fuzz script name to fuzzers variable and follow steps in README to upload new docker image. Make sure you update the tag to be higher than latest version in GCR.

Run the following command from your machine to update the docker image tag in the git trigger.

  stubby call --rpc_creds_file=/tmp/mint.txt \
  blade:alphasource-ci-proctor-metadata-service-prod \
  ProctorMetadataService.UpdateTrigger --proto2 <<EOF
  trigger {
    cloud_project_number: 257246079067
    name: "cdm-git-trigger"
    id: "e8939c9a-d971-4c05-91b5-e0544abf872b"
    state: LIVE
    git_trigger {
      url: "https://widevine-internal.googlesource.com/cdm"
      branch_name: "master"
    }
    build_configs {
      build {
        steps {
          name: "gcr.io/google.com/blockbuster-1154/
          cloud-build-oemcrypto-release-ubuntu-fuzz:LATEST_TAG_VERSION"
        }
      }
    }
    result_config {
      email_config {
        notify_condition {
          condition: ON_FAILURE
        }
        to_address: "wideving-engprod@google.com"
      }
    }
  }
  EOF

Generate code coverage reports locally

Code coverage is a means of measuring fuzzer performance. We want to make sure that our fuzzer covers all the paths in our code and make any tweeks to fuzzer logic so we can maximize coverage to get better results.

Coverage reports for git on borg project is not automated and needs to be generated manually. Future plan is to build a dashboard for git on borg coverage reports.

Generate code coverage reports using script from Google cloud build

A docker image with script to generate code coverage reports for oemcrypto fuzz scripts is linked with a GCB trigger oemcrypto-fuzzing-code-coverage-git-trigger. More information about clang source based coverage can be found here.
This trigger when invoked compiles oemcrypto fuzz scripts with clang source based code coverage enabled, downloads latest corpus from cluster fuzz for the respective fuzzer, generates and uploads code coverage html reports to GCS.
The trigger can be invoked manually using cloud scheduler oemcrypto_fuzzing_code_coverage_reports.
In order to generate latest code coverage reports from master branch, go to pantheon->cloud scheduler->oemcrypto_fuzzing_code_coverage_reports and click on RUN NOW button.
The above step should invoke a google cloud build. Go to cloud build console and find latest build job with Trigger Name oemcrypto-fuzzing-code-coverage-git-trigger.
Once the build job is successful, latest code coverage reports can be downloaded from GCS. The coverage report folder uploaded to GCS is appended with timestamp.