231 lines
10 KiB
Markdown
231 lines
10 KiB
Markdown
# OEMCRYPTO Fuzzing
|
|
|
|
Refer to [Setting up Clusterfuzz](build_clusterfuzz.md) if you are interested
|
|
in setting up a local instance of cluster fuzz to run fuzzing on your own
|
|
OEMCrypto implementations on linux.
|
|
|
|
## Objective
|
|
|
|
* Run fuzzing on OEMCrypto public APIs on linux using google supported
|
|
clusterfuzz infrastructure to find security vulnerabilities.
|
|
|
|
Design Document - https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit?usp=sharing
|
|
|
|
Fuzzing at google -
|
|
[go/fuzzing](https://g3doc.corp.google.com/security/fuzzing/g3doc/fuzzing_resources.md?cl=head)
|
|
## Monitoring
|
|
### Cluster fuzz statistics
|
|
|
|
* Performance of OEMCrypto fuzz binaries running continuously using cluster
|
|
fuzz infrastructure can be monitored
|
|
[here](https://clusterfuzz.corp.google.com/fuzzer-stats).
|
|
|
|
The options to select are `Job type: libfuzzer_asan_oemcrypto` and `Fuzzer:
|
|
fuzzer name you are looking for`
|
|
|
|
Example: [load_license_fuzz](https://clusterfuzz.corp.google.com/fuzzer-stats?group_by=by-day&date_start=2020-07-11&date_end=2020-07-17&fuzzer=libFuzzer_oemcrypto_load_license_fuzz&job=libfuzzer_asan_oemcrypto)
|
|
|
|
### Issues filed by clusterfuzz - Fixing those issues
|
|
|
|
* Any issues found with the fuzz target under test are reported by clusterfuzz
|
|
[here](https://b.corp.google.com/hotlists/2442954).
|
|
|
|
* The bug will have a link to the test case that generated the bug. Download
|
|
the test case and follow the steps from
|
|
[testing fuzzer locally](#testing-fuzzer-locally) section to run the fuzzer
|
|
locally using the test case that caused the crash.
|
|
|
|
* Once the issue is fixed, consider adding the test case that caused the crash
|
|
to the seed corpus zip file. Details about seed corpus and their location
|
|
are mentioned in
|
|
[this section](#build-oemcrypto-unit-tests-to-generate-corpus).
|
|
|
|
## Corpus
|
|
|
|
* Once the fuzzer scripts are ready and running continuously using clusterfuzz
|
|
or android infrastructure, we can measure the efficiency of fuzzers by
|
|
looking at code coverage and number of new features that have been
|
|
discovered by fuzzer scripts here Fuzz script statistics.
|
|
|
|
A fuzzer which tries to start from random inputs and figure out intelligent
|
|
inputs to crash the libraries can be time consuming and not effective. A way
|
|
to make fuzzers more effective is by providing a set of valid and invalid
|
|
inputs of the library so that fuzzer can use those as a starting point.
|
|
These sets of valid and invalid inputs are called corpus.
|
|
|
|
The idea is to run OEMCrypto unit tests and read required data into binary
|
|
corpus files before calling into respective OEMCrypto APIs under test.
|
|
Writing corpus data to binary files is controlled by --generate_corpus flag.
|
|
|
|
### Build OEMCrypto unit tests to generate corpus
|
|
|
|
* Install Pre-requisites
|
|
|
|
```shell
|
|
$ sudo apt-get install gyp ninja-build
|
|
```
|
|
|
|
* download cdm source code (including ODK & OEMCrypto unit tests):
|
|
|
|
```shell
|
|
$ git clone sso://widevine-internal/cdm
|
|
```
|
|
|
|
* Build OEMCrypto unit tests and run with --generate_corpus flag to generate
|
|
corpus files:
|
|
|
|
```shell
|
|
$ cd /path/to/cdm/repo
|
|
$ export CDM_DIR=/path/to/cdm/repo
|
|
$ export PATH_TO_CDM_DIR=..
|
|
$ gyp --format=ninja --depth=$(pwd) oemcrypto/oemcrypto_unittests.gyp
|
|
$ ninja -C out/Default/
|
|
$ mkdir oemcrypto/test/fuzz_tests/corpus/<fuzzername>_seed_corpus
|
|
# Generate corpus by excluding buffer overflow tests.
|
|
$ ./out/Default/oemcrypto_unittests --generate_corpus \
|
|
--gtest_filter=-"*Huge*"
|
|
```
|
|
|
|
* There can be lot of duplicate corpus files that are generated from unit
|
|
tests. We can minimize the corpus files to only a subset of files that
|
|
cover unique paths within the API when run using fuzzer. Run following
|
|
command to minimize corpus.
|
|
|
|
```shell
|
|
$ cd /path/to/cdm/repo
|
|
# build fuzzer binaries
|
|
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
|
|
$ mkdir /tmp/minimized_corpus
|
|
# minimize corpus
|
|
$ ./out/Default/<fuzz_target_binary> -merge=1 /tmp/minimized_corpus \
|
|
<FULL_CORPUS_DIR>
|
|
```
|
|
|
|
* To avoid uploading huge binary files to git repository, the minimized corpus
|
|
files will be saved in fuzzername_seed_corpus.zip format in blockbuster
|
|
project's oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need
|
|
permissions for blockbuster project, contact widevine-engprod@google.com.
|
|
|
|
```shell
|
|
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
|
|
<destination_path>
|
|
```
|
|
|
|
## Testing fuzzer locally
|
|
|
|
* Corpus needed to run fuzz tests locally are available in blockbuster
|
|
project's oemcrypto_fuzzing_corpus GCS bucket. If you need permissions for
|
|
this project, contact widevine-engprod@google.com. Download corpus.
|
|
|
|
```shell
|
|
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
|
|
<destination_path>
|
|
```
|
|
|
|
* Add flags to generate additional debugging information. Add '-g3' flag to
|
|
oemcrypto_fuzztests.gypi cflags_cc in order to generate additional debug
|
|
information locally.
|
|
|
|
* Build and test fuzz scripts locally using following commands. The build
|
|
script builds fuzz binaries for both oemcrypto reference implementation
|
|
as well as opk implementation.
|
|
|
|
```shell
|
|
$ cd PATH_TO_CDM_DIR
|
|
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
|
|
$ mkdir /tmp/new_interesting_corpus
|
|
$ ./out/Default/fuzzer_binary /tmp/new_interesting_corpus \
|
|
/path/to/fuzz/seed/corpus/folder
|
|
```
|
|
|
|
* In order to run fuzz script against a crash input, follow the above steps
|
|
and run the fuzz binary against crash input rather than seed corpus.
|
|
|
|
```shell
|
|
$ ./out/Default/fuzzer_binary crash_input_file
|
|
```
|
|
## Adding a new OEMCrypto fuzz script
|
|
* In order to fuzz a new OEMCrypto API in future, a fuzz script can be added
|
|
to oemcrypto/test/fuzz_tests folder which starts with oemcrypto and ends
|
|
with fuzz.cc(GCB build script for oemcrypto fuzzers expects the format).
|
|
|
|
* In the program, define the function LLVMFuzzerTestOneInput with the following signature:
|
|
```
|
|
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
|
|
<your test code goes here>
|
|
return 0;
|
|
}
|
|
```
|
|
*Note*: Make sure LLVMFuzzerTestOneInput calls the function you want to fuzz.
|
|
|
|
* Add a new target to oemcrypto_fuzztests.gyp file and follow instructions in
|
|
[testing fuzzer locally](#testing-fuzzer-locally) to build and test locally.
|
|
|
|
## Building OEMCrypto fuzz scripts and uploading them to Google Cloud Storage:
|
|
|
|
* We are using Google Cloud Buid (GCB) in order to setup continuous
|
|
integration which uploads OEMCrypto fuzz binaries to Google Cloud Storage.
|
|
GCB expects build script in form of a docker image that is uploaded to
|
|
Google Container Registry(GCR).
|
|
|
|
The cloud build scripts (docker images) for widevine projects are
|
|
[here](https://widevine-internal.googlesource.com/cloud/+/refs/heads/master/docker/README.md)
|
|
|
|
Refer to README of the project to setup a new docker image and uploading
|
|
the image to GCR.
|
|
|
|
* Git on borg repository needs to be integrated with GCB and a git trigger
|
|
needs to be set up in order to achieve continuous integration. Git trigger
|
|
will mention which docker image the GCB needs to use in order to build fuzz
|
|
binaries. GCB searches for docker images from GCR.
|
|
|
|
Design document lists the steps to create a git trigger.
|
|
|
|
### Adding a new fuzz script to the build script:
|
|
|
|
* As long as a new fuzz script is added which starts with oemcrypto and ends
|
|
with fuzz, the build command can be added to build_oemcrypto_fuzztests.
|
|
GCB script uses build_oemcrypto_fuzztests script to build fuzz binaries
|
|
and make them available for clusterfuzz to run continuously.
|
|
|
|
* If the new fuzzer cannot follow the naming convention OR GCB script needs
|
|
to be updated for any other reason, refer to [this section](https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit#heading=h.bu9yfftdonkg)
|
|
section.
|
|
|
|
## Generate code coverage reports locally
|
|
|
|
* Code coverage is a means of measuring fuzzer performance. We want to make
|
|
sure that our fuzzer covers all the paths in our code and make any tweeks to
|
|
fuzzer logic so we can maximize coverage to get better results.
|
|
|
|
Coverage reports for git on borg project is not automated and needs to be
|
|
generated manually. Future plan is to build a dashboard for git on borg
|
|
coverage reports.
|
|
|
|
### Generate code coverage reports using script from Google cloud build
|
|
* A docker image with script to generate code coverage reports for oemcrypto
|
|
fuzz scripts is linked with a GCB trigger
|
|
`oemcrypto-fuzzing-code-coverage-git-trigger`. More information about clang
|
|
source based coverage can be found
|
|
[here](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html).
|
|
|
|
* This trigger when invoked compiles oemcrypto fuzz scripts with clang source
|
|
based code coverage enabled, downloads latest corpus from cluster fuzz
|
|
for the respective fuzzer, generates and uploads code coverage html reports
|
|
to [GCS](https://pantheon.corp.google.com/storage/browser/oemcrypto_fuzzing_code_coverage_reports;tab=objects?forceOnBucketsSortingFiltering=false&project=google.com:blockbuster-1154&prefix=).
|
|
|
|
* The trigger can be invoked manually using cloud scheduler
|
|
`oemcrypto_fuzzing_code_coverage_reports`.
|
|
|
|
* In order to generate latest code coverage reports from master branch,
|
|
go to pantheon->cloud scheduler->oemcrypto_fuzzing_code_coverage_reports and
|
|
click on `RUN NOW` button.
|
|
|
|
* The above step should invoke a google cloud build. Go to cloud build console
|
|
and find latest build job with Trigger Name
|
|
`oemcrypto-fuzzing-code-coverage-git-trigger`.
|
|
|
|
* Once the build job is successful, latest code coverage reports can be
|
|
downloaded from [GCS](https://pantheon.corp.google.com/storage/browser/oemcrypto_fuzzing_code_coverage_reports;tab=objects?forceOnBucketsSortingFiltering=false&project=google.com:blockbuster-1154&prefix=).
|
|
The coverage report folder uploaded to GCS is appended with timestamp.
|