Source release 17.1.0

This commit is contained in:
John "Juce" Bruce
2022-07-07 17:14:31 -07:00
parent 8c17574083
commit 694cf6fb25
2233 changed files with 272026 additions and 223371 deletions

View File

@@ -1 +0,0 @@
0a4c08001248000000020000101907d9ffde13aa95c122678053362136bdf8408f8276e4c2d87ec52b61aa1b9f646e58734930acebe899b3e464189a14a87202fb02570640bd22ef44b2d7e3912250a230a14080112100915007caa9b5931b76a3a85f046523e10011a09393837363534333231180120002a0c313838363738373430350000

View File

@@ -1 +0,0 @@
0a4c020:0d1190d79fef02570640bd22ef44b2d7e3912250a200

View File

@@ -1 +0,0 @@
0a4c000000200:0101907d9ffde02570640bd22ef44b2d7e3912250a230a1407363534333231180120002a0c313838363738373430350000

View File

@@ -1 +0,0 @@
0a4c000000220:01019dd79fef02570640bd22ef44b2d7e3912250a200

View File

@@ -1 +0,0 @@
0a4c000000200:010197d9ffde02570640bd22ef44b2d7e3912250a230a1407363534333231180120002a0c313838363738373430350000

View File

@@ -1 +0,0 @@
0a4c00000020000101907d9ffde02570640bd22ef44b2d7e3912250a230a14080112100915007caa9b5931b76a3a85f046523e10011a09393837363534333231180120002a0c313838363738373430350000

View File

@@ -1 +0,0 @@
0a4c08001248000000020000101907d9ffde02570640bd22ef44b2d7e3912250a230a14080112100915007caa9b5931b76a3a85f046523e10011a09393837363534333231180120002a0c313838363738373430350000

View File

@@ -1 +0,0 @@
0a4c08001248000000020000101907d9ffde13aa95c122678053362136bdf8408f8276e4c2d87ec52b61aa1b9f646e58734930acebe899b3e464189a14a87202fb02574e70640bd22ef44b2d7e3912250a230a14080112100915007caa9b5931b76a3a85f046523e10011a09393837363534333231180120002a0c313838363738373430350000

View File

@@ -1,13 +1,17 @@
# OEMCRYPTO Fuzzing
Refer to [Setting up Clusterfuzz](build_clusterfuzz.md) if you are interested
in setting up a local instance of cluster fuzz to run fuzzing on your own
OEMCrypto implementations on linux.
## Objective
* Run fuzzing on OEMCrypto public APIs on linux using google supported
clusterfuzz infrastructure to find security vulnerabilities.
Design Document - https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit?usp=sharing
Fuzzing at google -
Fuzzing at google -
[go/fuzzing](https://g3doc.corp.google.com/security/fuzzing/g3doc/fuzzing_resources.md?cl=head)
## Monitoring
### Cluster fuzz statistics
@@ -18,7 +22,7 @@
The options to select are `Job type: libfuzzer_asan_oemcrypto` and `Fuzzer:
fuzzer name you are looking for`
Example: [load_license_fuzz](https://clusterfuzz.corp.google.com/fuzzer-stats?group_by=by-day&date_start=2020-07-11&date_end=2020-07-17&fuzzer=libFuzzer_oemcrypto_load_license_fuzz&job=libfuzzer_asan_oemcrypto)
### Issues filed by clusterfuzz - Fixing those issues
@@ -76,13 +80,31 @@
$ export PATH_TO_CDM_DIR=..
$ gyp --format=ninja --depth=$(pwd) oemcrypto/oemcrypto_unittests.gyp
$ ninja -C out/Default/
$ ./out/Default/oemcrypto_unittests --generate_corpus
$ mkdir oemcrypto/test/fuzz_tests/corpus/<fuzzername>_seed_corpus
# Generate corpus by excluding buffer overflow tests.
$ ./out/Default/oemcrypto_unittests --generate_corpus \
--gtest_filter=-"*Huge*"
```
* To avoid uploading huge binary files to git repository, the corpus files
will be saved in fuzzername_seed_corpus.zip format in blockbuster project's
oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need permissions
for blockbuster project, contact widevine-engprod@google.com.
* There can be lot of duplicate corpus files that are generated from unit
tests. We can minimize the corpus files to only a subset of files that
cover unique paths within the API when run using fuzzer. Run following
command to minimize corpus.
```shell
$ cd /path/to/cdm/repo
# build fuzzer binaries
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
$ mkdir /tmp/minimized_corpus
# minimize corpus
$ ./out/Default/<fuzz_target_binary> -merge=1 /tmp/minimized_corpus \
<FULL_CORPUS_DIR>
```
* To avoid uploading huge binary files to git repository, the minimized corpus
files will be saved in fuzzername_seed_corpus.zip format in blockbuster
project's oemcrypto_fuzzing_corpus GCS bucket using gsutil. If you need
permissions for blockbuster project, contact widevine-engprod@google.com.
```shell
$ gsutil cp gs://oemcrypto_fuzzing_corpus/<fuzzername_seed_corpus.zip> \
@@ -104,17 +126,13 @@
oemcrypto_fuzztests.gypi cflags_cc in order to generate additional debug
information locally.
* Build and test fuzz scripts locally using:
* Build and test fuzz scripts locally using following commands. The build
script builds fuzz binaries for both oemcrypto reference implementation
as well as opk implementation.
```shell
$ export CXX=clang++
$ export CC=clang
$ export GYP_DEFINES="clang=1"
$ cd /path/to/cdm/repo
$ export PATH_TO_CDM_DIR=.
$ gyp --format=ninja --depth=$(pwd) \
oemcrypto/test/fuzz_tests/oemcrypto_fuzztests.gyp
$ ninja -C out/Default/
$ cd PATH_TO_CDM_DIR
$ ./oemcrypto/test/fuzz_tests/build_oemcrypto_fuzztests
$ mkdir /tmp/new_interesting_corpus
$ ./out/Default/fuzzer_binary /tmp/new_interesting_corpus \
/path/to/fuzz/seed/corpus/folder
@@ -127,8 +145,9 @@
$ ./out/Default/fuzzer_binary crash_input_file
```
## Adding a new OEMCrypto fuzz script
* In order to fuzz a new OEMCrypto API in future, a fuzz script can be added to
oemcrypto/test/fuzz_tests folder which ends with _fuzz.cc.
* In order to fuzz a new OEMCrypto API in future, a fuzz script can be added
to oemcrypto/test/fuzz_tests folder which starts with oemcrypto and ends
with fuzz.cc(GCB build script for oemcrypto fuzzers expects the format).
* In the program, define the function LLVMFuzzerTestOneInput with the following signature:
```
@@ -164,50 +183,14 @@
### Adding a new fuzz script to the build script:
* In order to update build script such as adding a new fuzzer to build script,
we need to update the build script in docker image from cloud repository.
[Build script.](https://widevine-internal.googlesource.com/cloud/+/refs/heads/master/docker
/cloud_build/oemcrypto/release/ubuntu/fuzz/build.sh)
* As long as a new fuzz script is added which starts with oemcrypto and ends
with fuzz, the build command can be added to build_oemcrypto_fuzztests.
GCB script uses build_oemcrypto_fuzztests script to build fuzz binaries
and make them available for clusterfuzz to run continuously.
Add the new fuzz script name to fuzzers variable and follow steps in README
to upload new docker image. Make sure you update the tag to be higher than
latest version in GCR.
Run the following command from your machine to update the docker image tag
in the git trigger.
```shell
stubby call --rpc_creds_file=/tmp/mint.txt \
blade:alphasource-ci-proctor-metadata-service-prod \
ProctorMetadataService.UpdateTrigger --proto2 <<EOF
trigger {
cloud_project_number: 257246079067
name: "cdm-git-trigger"
id: "e8939c9a-d971-4c05-91b5-e0544abf872b"
state: LIVE
git_trigger {
url: "https://widevine-internal.googlesource.com/cdm"
branch_name: "master"
}
build_configs {
build {
steps {
name: "gcr.io/google.com/blockbuster-1154/
cloud-build-oemcrypto-release-ubuntu-fuzz:LATEST_TAG_VERSION"
}
}
}
result_config {
email_config {
notify_condition {
condition: ON_FAILURE
}
to_address: "wideving-engprod@google.com"
}
}
}
EOF
```
* If the new fuzzer cannot follow the naming convention OR GCB script needs
to be updated for any other reason, refer to [this section](https://docs.google.com/document/d/1mdSV2irJZz5Y9uYb5DmSIddBjrAIZU9q8G5Q_BGpA4I/edit#heading=h.bu9yfftdonkg)
section.
## Generate code coverage reports locally
@@ -244,4 +227,4 @@
* Once the build job is successful, latest code coverage reports can be
downloaded from [GCS](https://pantheon.corp.google.com/storage/browser/oemcrypto_fuzzing_code_coverage_reports;tab=objects?forceOnBucketsSortingFiltering=false&project=google.com:blockbuster-1154&prefix=).
The coverage report folder uploaded to GCS is appended with timestamp.
The coverage report folder uploaded to GCS is appended with timestamp.

View File

@@ -0,0 +1,174 @@
# OEMCRYPTO Fuzzing - Build clustefuzz and run fuzzing
## Objective
* Run fuzzing on OEMCrypto public APIs on linux by building open sourced
clusterfuzz source code in order to find security vulnerabilities.
[Clusterfuzz][1]
* Partners who implement OEMCrypto can follow these instructions to build
clusterfuzz, the fuzzing framework and run fuzzing using fuzzer scripts
provided by the Widevine team at Google.
## Glossary
* Fuzzing - Fuzzing is a methodology where random, interesting, unexpected
inputs are fed to APIs in order to crash those, thereby catching any
security vulnerabilities with the code.
* Fuzzing engines - [libfuzzer][4], afl, honggfuzz are the actual fuzzing
engines that get the coverage information from API, use that to generate
more interesting inputs which can be passed to fuzzer.
* Seed corpus - Fuzzing engine trying to generate interesting inputs from an
empty file is not efficient. Seed corpus is the initial input that a fuzzer
can accept and call the API with that. Fuzzing engine can then mutate this
seed corpus to generate more inputs to fuzzer.
* Clusterfuzz - ClusterFuzz is a scalable fuzzing infrastructure that finds
security and stability issues in software. Google uses ClusterFuzz to fuzz
all Google products. Clusterfuzz provides us with the capability, tools to
upload fuzz binaries and make use of the fuzzing engines to run fuzzing,
find crashes and organizes the information. Clusterfuzz framework is open
sourced, the source code can be downloaded and framework can be built
locally or by using google cloud.
* Fuzzing output - Fuzzing is used to pass random inputs to API in order to
ensure that API is crash resistant. We are not testing functionality via
fuzzing. Fuzz scripts run continuously until they find a crash with the API
under test.
## Building fuzz scripts
This section outlines the steps to build fuzz binaries that can be run
continuously using clusterfuzz.
> **Note:** All the directories mentioned below are relative to cdm repository
> root directory.
1. Fuzz scripts for OEMCrypto APIs are provided by the Widevine team at Google
located under `oemcrypto/test/fuzz_tests` directory.
> **Note:** Prerequisites to run the following step are [here][10]. We also need
> to install ninja.
2. Build a static library of your OEMCrypto implementation.
* Compile and link your OEMCrypto implementation source with
`-fsanitize=address,fuzzer` flag as per these [instructions][9] when
building a static library.
* Run `./oemcrypto/test/fuzz_tests/build_partner_oemcrypto_fuzztests
<oemcrypto_static_library_path>` script from cdm repository root
directory.
* This will generate fuzz binaries under the `out/Default` directory.
> **Note:** Alternatively, you can use your own build systems, for which you
> will need to define your own build files with the OEMCrypto fuzz source files
> included. You can find the the fuzz source files in
> `oemcrypto/test/fuzz_tests/partner_oemcrypto_fuzztests.gyp` and
> `oemcrypto/test/fuzz_tests/partner_oemcrypto_fuzztests.gypi`.
3. Seed corpus for each fuzz script can be found under
`oemcrypto/test/fuzz_tests/corpus` directory. Some fuzzers are simple and do
not have seed corpus associated with them.
4. Create a zip file `oemcrypto_fuzzers_yyyymmddhhmmss.zip` with fuzz binaries
and respective seed corpus zip files. Structure of a sample zip file with
fuzzer binaries and seed corpus would look like following:
```
* fuzzerA
* fuzzerA_seed_corpus.zip
* fuzzerB
* fuzzerB_seed_corpus.zip
* fuzzerC (fuzzerC doesn't have seed corpus associated with it)
```
## Building clusterfuzz
* OEMCrypto implementation can be fuzzed by building clusterfuzz code which is
open sourced and using it to run fuzzing. Use a Linux VM to build
clusterfuzz.
> **Note:** You may see some issues with python modules missing, please install
> those modules if you see errors. If you have multiple versions of python on
> the VM, then use `python<version> -m pipenv shell` when you are at [this][3]
> step.
* Follow these [instructions][2] in order to download clusterfuzz repository,
build it locally or create a continuous fuzz infrastructure setup using
google cloud.
## Running fuzzers on local clusterfuzz instance
* If you prefer to run fuzzing on a local machine instead of having a
production setup using google cloud, then follow these [instructions][6] to
add a job to the local clusterfuzz instance.
> **Note:** Job name should have a fuzzing engine and sanitizer as part of it. A
> libfuzzer and asan jobs should have libfuzzer_asan in the job name.
* Create a job e:g:`libfuzzer_asan_oemcrypto` and upload previously created
`oemcrypto_fuzzers_yyyymmddhhmmss.zip` as a custom build. Future uploads of
zip file should have a name greater than current name. Following the above
naming standard will ensure zip file names are always in ascending order.
* Once the job is added and clusterfuzz bot is running, fuzzing should be up
and running. Results can be monitored as mentioned [here][6].
* On a local clusterfuzz instance, only one fuzzer is being fuzzed at a time.
> **Note:** Fuzzing is time consuming. Finding issues as well as clusterfuzz
> regressing and fixing the issues can take time. We need fuzzing to run at
> least for a couple of weeks to have good coverage.
## Finding fuzz crashes
Once the clusterfuzz finds an issue, it logs crash information such as the
build, test case and stack trace for the crash.
* Test cases tab should show the fuzz crash and test case that caused the
crash. Run `./fuzz_binary <test_case>` in order to debug the crash locally.
More information about different types of logs is as below:
* [Bot logs][7] will show information related to fuzzing, number of crashes
that a particular fuzzer finds, number of new crashes, number of known
crashes etc.
* [Local GCS][8] in your clusterfuzz checkout folder will store the fuzz
binaries that are being fuzzed, seed corpus etc.
* `local_gcs/test-fuzz-logs-bucket` will store information related to fuzz
crashes if any were found by the fuzzing engine. It will store crash
information categorized by fuzzer and by each day. It will also store test
case that caused the crash.
* `/path/to/my-bot/clusterfuzz/log.txt` will have any log information from
fuzzer script and OEMCrypto implementation.
## Fixing issues
* Once you are able to debug using the crash test case, apply fix to the
implementation, create `oemcrypto_fuzzers_yyyymmddhhmmss.zip` with latest
fuzz binaries.
* Upload the latest fuzz binary to the fuzz job that was created earlier.
Fuzzer will recognize the fix and mark the crash as fixed in test cases tab
once the regression finishes. You do not need to update crashes as fixed,
clusterfuzz will do that.
[1]: https://google.github.io/clusterfuzz/
[2]: https://google.github.io/clusterfuzz/getting-started/
[3]: https://google.github.io/clusterfuzz/getting-started/prerequisites/#loading-pipenv
[4]: https://llvm.org/docs/LibFuzzer.html
[5]: https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/
[6]: https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/#checking-results
[7]: https://google.github.io/clusterfuzz/getting-started/local-instance/#viewing-logs
[8]: https://google.github.io/clusterfuzz/getting-started/local-instance/#local-google-cloud-storage
[9]: https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/#libfuzzer
[10]: https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/#prerequisites

View File

@@ -0,0 +1,24 @@
#!/bin/bash
# Also, if you are fixing this script, it should probably be moved to the
# jenkins directory, so that it is next to all the other scripts that LUCI
# runs.
set -ex
export CXX=clang++
export CC=clang
export GYP_DEFINES="$GYP_DEFINES clang=1"
export PATH_TO_CDM_DIR=.
export PYTHONPATH="$PYTHONPATH:$PATH_TO_CDM_DIR/third_party"
python3 $PATH_TO_CDM_DIR/third_party/gyp/__init__.py --format=ninja \
--depth=$(pwd) oemcrypto/test/fuzz_tests/oemcrypto_fuzztests.gyp
ninja -C out/Default
# oemcrypto_opk_fuzztests.gypi has flags to instrument all the gyp targets
# with fuzzer flags.
python3 $PATH_TO_CDM_DIR/third_party/gyp/__init__.py --format=ninja \
--depth=$(pwd) \
--include=oemcrypto/test/fuzz_tests/oemcrypto_opk_fuzztests.gypi \
oemcrypto/test/fuzz_tests/oemcrypto_opk_fuzztests.gyp
ninja -C out/Default

View File

@@ -0,0 +1,15 @@
#!/bin/bash
set -ex
# For use by partners to generate fuzz binaries for their OEMCrypto
# implementation on linux.
export CXX=clang++
export CC=clang
export GYP_DEFINES="$GYP_DEFINES clang=1 oemcrypto_static_library=$1"
export PATH_TO_CDM_DIR=.
export PYTHONPATH="$PYTHONPATH:$PATH_TO_CDM_DIR/third_party"
python3 $PATH_TO_CDM_DIR/third_party/gyp/__init__.py --format=ninja \
--depth=$(pwd) oemcrypto/test/fuzz_tests/partner_oemcrypto_fuzztests.gyp
ninja -C out/Default

View File

@@ -0,0 +1,94 @@
// Copyright 2020 Google LLC. All Rights Reserved. This file and proprietary
// source code may only be used and distributed under the Widevine Master
// License Agreement.
#include "OEMCryptoCENC.h"
#include "log.h"
#include "oemcrypto_fuzz_helper.h"
#include "oemcrypto_fuzz_structs.h"
namespace wvoec {
// Free dynamic memory allocated by fuzzer script.
void FreeOutputBuffers(OEMCrypto_SESSION session_id,
OEMCrypto_DestBufferDesc& output_descriptor,
int* secure_fd) {
switch (output_descriptor.type) {
case OEMCrypto_BufferType_Clear: {
delete[] output_descriptor.buffer.clear.clear_buffer;
break;
}
case OEMCrypto_BufferType_Secure: {
OEMCrypto_FreeSecureBuffer(session_id, &output_descriptor, *secure_fd);
break;
}
case OEMCrypto_BufferType_Direct: {
break;
}
}
}
bool InitializeOutputBuffers(OEMCrypto_SESSION session_id,
OEMCrypto_DestBufferDesc& output_descriptor,
int* secure_fd, size_t input_buffer_size) {
switch (output_descriptor.type) {
case OEMCrypto_BufferType_Clear: {
output_descriptor.buffer.clear.clear_buffer =
new OEMCrypto_SharedMemory[input_buffer_size];
return true;
}
case OEMCrypto_BufferType_Secure: {
OEMCryptoResult sts = OEMCrypto_AllocateSecureBuffer(
session_id, input_buffer_size, &output_descriptor, secure_fd);
return sts == OEMCrypto_SUCCESS;
}
case OEMCrypto_BufferType_Direct: {
return true;
}
}
}
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
// Redirect printf and log statements from oemcrypto functions to a file to
// reduce noise
RedirectStdoutToFile();
uint8_t subsample_flags;
// OEMCrypto_DestBufferDesc and a buffer from which data needs to be copied
// are expected as inputs to copy buffer API.
// Input fuzzed data is interpreted as
// (OEMCrypto_DestBufferDesc | subsample_flags | input_buffer)
if (size <= sizeof(OEMCrypto_Copy_Buffer_Fuzz)) {
return 0;
}
OEMCrypto_Copy_Buffer_Fuzz fuzzed_structure;
// Fuzz dest_buffer_desc.
memcpy(&fuzzed_structure, data, sizeof(fuzzed_structure));
ConvertDataToValidEnum(OEMCrypto_BufferType_MaxValue,
&fuzzed_structure.dest_buffer_desc.type);
OEMCryptoLicenseAPIFuzz license_api_fuzz;
Session* session = license_api_fuzz.session();
// Fuzz input buffer to be copied.
size_t input_buffer_size = size - sizeof(fuzzed_structure);
int secure_fd = 0;
// Create output buffer pointers. If secure buffer is not supported, we
// explicitly convert to clear buffer and fuzz.
if (!InitializeOutputBuffers(session->session_id(),
fuzzed_structure.dest_buffer_desc, &secure_fd,
input_buffer_size)) {
LOGI(
"[OEMCrypto decrypt CENC fuzz] Secure buffers are not supported. Use "
"clear buffer instead.");
fuzzed_structure.dest_buffer_desc.type = OEMCrypto_BufferType_Clear;
InitializeOutputBuffers(session->session_id(),
fuzzed_structure.dest_buffer_desc, &secure_fd,
input_buffer_size);
}
OEMCrypto_CopyBuffer(session->session_id(), data + sizeof(fuzzed_structure),
input_buffer_size, &fuzzed_structure.dest_buffer_desc,
subsample_flags);
FreeOutputBuffers(session->session_id(), fuzzed_structure.dest_buffer_desc,
&secure_fd);
return 0;
}
} // namespace wvoec

Some files were not shown because too many files have changed in this diff Show More