Running CATS
Blackbox Mode
Blackbox mode means that CATS doesn't need any specific context. You just need to provide the service URL, the OpenAPI spec and most probably authentication headers.
cats --contract=openapi.yaml --server=http://localhost:8080 --headers=headers.yml --blackbox
In blackbox mode CATS will only report errors if the received HTTP response code is a 5XX (except 501).
Any other mismatch between what the Fuzzer expects vs what the service returns (for example service returns 400 and CATS expects 200) will be reported as success.
The blackbox mode is similar to a smoke test. It will quickly tell you if the application has major bugs that must be addressed immediately.
--blackbox is actually equivalent to --ignoreResponseCodes=2XX,4XX,501. If you want your final report to only contain the 5XX errors, you can use --skipReportingForIgnored.
Context Mode
The second mode is called Context Mode. Each Fuzzer has an expected HTTP response code based on the scenario under test and will also check if the response is matching the schema defined in the OpenAPI spec corresponding to that response code. This will allow you to tweak either your OpenAPI spec or service behaviour in order to create good quality APIs and documentation and also to avoid possible serious bugs.
Running CATS in context mode usually implies providing a --refData file with pre-existing resource identifiers or values that needs to be fixed (or from a limited set). CATS cannot create data on its own (yet), so it's important that any request field or query param that requires pre-existence of those entities/resources to be created in advance and provided through the reference data file.
cats --contract=openapi.yaml --server=http://localhost:8080 --headers=headers.yml --refData=referenceData.yml
Continuous Fuzzing Mode
The first two modes are deterministic. Running CATS multiple times with the same configuration will (most likely) produce the same output: same number of tests, same number of failing tests with the same reason.
CATS can also do continuous fuzzing i.e. fuzz until a certain stop condition is met, either time based, failing tests or number of tests.
Continuous fuzzing has more randomness when executing: random selection of field to fuzz, fuzzing data is either generated on the fly or randomly picked
from a set of data. When doing continuous fuzzing you must also supply a matchXXX condition. This is used to flag the tests as error when that condition is met.
This is a typical continuous fuzzing run:
cats random --contract=openapi.yaml --server=http://localhost:8080 -H "API-KEY=$token" --mc 500 --path "/users/auth" -X POST --stopAfterTimeInSec 10
This will run continuous fuzzing for method POST on path /users/auth, flag any 500 response code as error and stop after 10 seconds.
As continuous fuzzing do not rely on pre-defined fuzzers, the matchXXX condition offers control on how each test case is flagged as error or not.
Continuous fuzzing is also more targeted, this is why supplying the --path and http method -X is required.
It's recommended to supply a --stopXXX argument in order for CATS to stop the execution when the supplied condition is met.
Otherwise, it will run indefinitely and won't produce a summary report. It will still produce individual .json and .html test reports.
Continuous fuzzing is based on mutators.
Notes on Skipped Tests
You may notice a significant number of tests marked as skipped. CATS tries to apply all Fuzzers to all fields, but this is not always possible.
For example the RandomStringInBooleanFieldsFuzzer cannot be applied to string fields. This is why that test attempt will be marked as skipped.
It was an intentional decision to also report the skipped tests in order to show that CATS actually tries all the Fuzzers on all the fields/paths/endpoints.
Skipped tests are not included in the final report.
Notes on Console Output
When running with --verbosity DETAILED CATS produces a significant amount of logging.
If the output is redirected to a file please make sure you do proper cleanup when not needing it.
A simple run can produce tens of MB of data.
You have several options to control the logging level using: --log, --skipLog, --onlyLog arguments.
For example --log=error will only log errors. Or you can keep the default --verbosity SUMMARY which has a very condensed output.
Additionally, CATS support a lot more arguments that allows you to restrict the number of fuzzers, provide timeouts, limit the number of requests per minute and so on.