# OEMCRYPTO Fuzzing

## Objective

*   Run fuzzing on OEMCrypto public APIs on linux using google supported
    clusterfuzz infrastructure to find security vulnerabilities.
    
    Design Document - https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit?usp=sharing
    
    Fuzzing at google - 
    [go/fuzzing](https://g3doc.corp.google.com/security/fuzzing/g3doc/fuzzing_resources.md?cl=head)
## Monitoring
### Cluster fuzz statistics

*   Performance of OEMCrypto fuzz binaries running continuously using cluster
    fuzz infrastructure can be monitored
    [here](https://clusterfuzz.corp.google.com/fuzzer-stats).

    The options to select are `Job type: libfuzzer_asan_oemcrypto` and `Fuzzer:
    fuzzer name you are looking for`
    
    Example: [load_license_fuzz](https://clusterfuzz.corp.google.com/fuzzer-stats?group_by=by-day&date_start=2020-07-11&date_end=2020-07-17&fuzzer=libFuzzer_oemcrypto_load_license_fuzz&job=libfuzzer_asan_oemcrypto)

### Issues filed by clusterfuzz - Fixing those issues

*   Any issues found with the fuzz target under test are reported by clusterfuzz
    [here](https://b.corp.google.com/hotlists/2442954).

*   The bug will have a link to the test case that generated the bug. Download
    the test case and follow the steps from
    [testing fuzzer locally](#testing-fuzzer-locally) section to run the fuzzer
    locally using the test case that caused the crash.

*   Once the issue is fixed, consider adding the test case that caused the crash
    to the seed corpus zip file. Details about seed corpus and their location
    are mentioned in
    [this section](#build-oemcrypto-unit-tests-to-generate-corpus).

## Corpus

*   Once the fuzzer scripts are ready and running continuously using clusterfuzz
    or android infrastructure, we can measure the efficiency of fuzzers by
    looking at code coverage and number of new features that have been
    discovered by fuzzer scripts here Fuzz script statistics.

    A fuzzer which tries to start from random inputs and figure out intelligent
    inputs to crash the libraries can be time consuming and not effective. A way
    to make fuzzers more effective is by providing a set of valid and invalid
    inputs of the library so that fuzzer can use those as a starting point.
    These sets of valid and invalid inputs are called corpus.

    The idea is to run OEMCrypto unit tests and read required data into binary
    corpus files before calling into respective OEMCrypto APIs under test.
    Writing corpus data to binary files is controlled by --generate_corpus flag.

### Build OEMCrypto unit tests to generate corpus

*   Install Pre-requisites

    ```shell
    $ sudo apt-get install gyp ninja-build
    ```

*   download cdm source code (including ODK & OEMCrypto unit tests):

    ```shell
    $ git clone sso://widevine-internal/cdm
    ```

*   Build OEMCrypto unit tests and run with --generate_corpus flag to generate
    corpus files:

    ```shell
    $ cd /path/to/cdm/repo
    $ export CDM_DIR=/path/to/cdm/repo
    $ export PATH_TO_CDM_DIR=..
    $ gyp --format=ninja --depth=$(pwd) oemcrypto/oemcrypto_unittests.gyp
    $ ninja -C out/Default/
    $ ./out/Default/oemcrypto_unittests --generate_corpus
    ```

*   To avoid uploading huge binary files to git repository, the corpus files
    will be saved in fuzzername_seed_corpus.zip format in blockbuster project's
    oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need permissions
    for blockbuster project, contact widevine-engprod@google.com.

    ```shell
    $ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
        <destination_path>
    ```

## Testing fuzzer locally

*   Corpus needed to run fuzz tests locally are available in blockbuster
    project's oemcrypto_fuzzing_corpus GCS bucket. If you need permissions for
    this project, contact widevine-engprod@google.com. Download corpus.

    ```shell
    $ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
      <destination_path>
    ```

*   Add flags to generate additional debugging information. Add '-g3' flag to
    oemcrypto_fuzztests.gypi cflags_cc in order to generate additional debug
    information locally.

*   Build and test fuzz scripts locally using:

    ```shell
    $ export CXX=clang++
    $ export CC=clang
    $ export GYP_DEFINES="clang=1"
    $ cd /path/to/cdm/repo
    $ export PATH_TO_CDM_DIR=.
    $ gyp --format=ninja --depth=$(pwd) \
        oemcrypto/test/fuzz_tests/oemcrypto_fuzztests.gyp
    $ ninja -C out/Default/
    $ mkdir /tmp/new_interesting_corpus
    $ ./out/Default/fuzzer_binary /tmp/new_interesting_corpus \
        /path/to/fuzz/seed/corpus/folder
    ```

*   In order to run fuzz script against a crash input, follow the above steps
    and run the fuzz binary against crash input rather than seed corpus.

    ```shell
    $ ./out/Default/fuzzer_binary crash_input_file
    ```
## Adding a new OEMCrypto fuzz script
*   In order to fuzz a new OEMCrypto API in future, a fuzz script can be added to
    oemcrypto/test/fuzz_tests folder which ends with _fuzz.cc.

*   In the program, define the function LLVMFuzzerTestOneInput with the following signature:
    ```
    extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
        <your test code goes here>
        return 0;
    }
    ```
    *Note*: Make sure LLVMFuzzerTestOneInput calls the function you want to fuzz.

*   Add a new target to oemcrypto_fuzztests.gyp file and follow instructions in
    [testing fuzzer locally](#testing-fuzzer-locally) to build and test locally.

## Building OEMCrypto fuzz scripts and uploading them to Google Cloud Storage:

*   We are using Google Cloud Buid (GCB) in order to setup continuous
    integration which uploads OEMCrypto fuzz binaries to Google Cloud Storage.
    GCB expects build script in form of a docker image that is uploaded to
    Google Container Registry(GCR).

    The cloud build scripts (docker images) for widevine projects are
    [here](https://widevine-internal.googlesource.com/cloud/+/refs/heads/master/docker/README.md)

    Refer to README of the project to setup a new docker image and uploading
    the image to GCR.

*   Git on borg repository needs to be integrated with GCB and a git trigger
    needs to be set up in order to achieve continuous integration. Git trigger
    will mention which docker image the GCB needs to use in order to build fuzz
    binaries. GCB searches for docker images from GCR.

    Design document lists the steps to create a git trigger.

### Adding a new fuzz script to the build script:

*   In order to update build script such as adding a new fuzzer to build script,
    we need to update the build script in docker image from cloud repository.
    [Build script.](https://widevine-internal.googlesource.com/cloud/+/refs/heads/master/docker
    /cloud_build/oemcrypto/release/ubuntu/fuzz/build.sh)

    Add the new fuzz script name to fuzzers variable and follow steps in README
    to upload new docker image. Make sure you update the tag to be higher than
    latest version in GCR.

    Run the following command from your machine to update the docker image tag
    in the git trigger.

    ```shell
      stubby call --rpc_creds_file=/tmp/mint.txt \
      blade:alphasource-ci-proctor-metadata-service-prod \
      ProctorMetadataService.UpdateTrigger --proto2 <<EOF
      trigger {
        cloud_project_number: 257246079067
        name: "cdm-git-trigger"
        id: "e8939c9a-d971-4c05-91b5-e0544abf872b"
        state: LIVE
        git_trigger {
          url: "https://widevine-internal.googlesource.com/cdm"
          branch_name: "master"
        }
        build_configs {
          build {
            steps {
              name: "gcr.io/google.com/blockbuster-1154/
              cloud-build-oemcrypto-release-ubuntu-fuzz:LATEST_TAG_VERSION"
            }
          }
        }
        result_config {
          email_config {
            notify_condition {
              condition: ON_FAILURE
            }
            to_address: "wideving-engprod@google.com"
          }
        }
      }
      EOF
    ```

## Generate code coverage reports locally

*   Code coverage is a means of measuring fuzzer performance. We want to make
    sure that our fuzzer covers all the paths in our code and make any tweeks to
    fuzzer logic so we can maximize coverage to get better results.

    Coverage reports for git on borg project is not automated and needs to be
    generated manually. Future plan is to build a dashboard for git on borg
    coverage reports.

### Generate code coverage reports using script from Google cloud build
*   A docker image with script to generate code coverage reports for oemcrypto
    fuzz scripts is linked with a GCB trigger
    `oemcrypto-fuzzing-code-coverage-git-trigger`. More information about clang
    source based coverage can be found
    [here](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html).

*   This trigger when invoked compiles oemcrypto fuzz scripts with clang source
    based code coverage enabled, downloads latest corpus from cluster fuzz
    for the respective fuzzer, generates and uploads code coverage html reports
    to [GCS](https://pantheon.corp.google.com/storage/browser/oemcrypto_fuzzing_code_coverage_reports;tab=objects?forceOnBucketsSortingFiltering=false&project=google.com:blockbuster-1154&prefix=).

*   The trigger can be invoked manually using cloud scheduler
    `oemcrypto_fuzzing_code_coverage_reports`.

*   In order to generate latest code coverage reports from master branch,
    go to pantheon->cloud scheduler->oemcrypto_fuzzing_code_coverage_reports and
    click on `RUN NOW` button.

*   The above step should invoke a google cloud build. Go to cloud build console
    and find latest build job with Trigger Name
    `oemcrypto-fuzzing-code-coverage-git-trigger`.

*   Once the build job is successful, latest code coverage reports can be
    downloaded from [GCS](https://pantheon.corp.google.com/storage/browser/oemcrypto_fuzzing_code_coverage_reports;tab=objects?forceOnBucketsSortingFiltering=false&project=google.com:blockbuster-1154&prefix=).
    The coverage report folder uploaded to GCS is appended with timestamp.