Storage

Disk

This page outlines configuration options relevant to using the disk storage feature of OPA. Configuration options are to be found in the configuration docs.

info

The persistent disk storage enables OPA to work with data that does not fit into the memory resources granted to the OPA server. It is not supposed to be used as the primary source of truth for that data.

The on-disk storage should be considered ephemeral: you need to secure the means to restore that data. Backup and restore, or repair procedures for data corruption are not provided at this time.

Partitions

Partitions determine how the JSON data is split up when stored in the underlying key-value store. For example, this table shows how an example document would be stored given different configured partitions:

{
  "users": {
    "alice": { "roles": ["admin"] },
    "bob": { "roles": ["viewer"] }
  }
}

Partitions	Keys	Values
(1) none	`/users`	`{"alice": {"roles": ["admin"]}, "bob": {"roles": ["viewer"]}}}`
---	---	---
(2) `/users`	`/users/alice`	`{"roles": ["admin"]}`
	`/users/bob`	`{"roles": ["viewer"]}`
---	---	---
(3) `/users/*`	`/users/alice/roles`	`["admin"]`
	`/users/bob/roles`	`["viewer"]`

Partitioning has consequences on performance: in the example above, the number of keys to retrieve from the database (and the amount of data of its values) varies.

Query	Partitions	Number of keys read
`data.users`	(1)	1
	(2)	2
	(3)	2
---	---	---
`data.users.alice`	(1)	1 with `bob` data thrown away
	(2)	2
	(3)	2

For example, retrieving the full extent of data.users from the disk store will require a single key fetch with the partitions of (1). With (2), the storage engine will fetch two keys and their values.

Retrieving a single user's data, e.g. data.users.alice, will require reading a single key and all the users data with (1); but throw away most of it: all the data not belonging to alice.

There is no one-size-fits-all setting for partitions: good settings depend on the actual usage, and that comes down to the policies that are used with OPA. Commonly, you would optimize the partition settings for those queries that are performance critical.

To figure out suboptimal partitioning, please have a look at the exposed metrics.

OPA stores some internal values (such as bundle metadata) in the data store, under /system. Partitions for that part of the data store are managed by OPA, and providing any overlapping partitions in the config will raise an error.

Partitioned values must be objects

A partition splits the value it points at into one key per member, so that value has to be an object. In the example above, /users works because data.users is an object keyed by user name, and /users/* works because each user is an object keyed by attribute name.

Had data.users.alice been an array instead, /users/* would be rejected:

value at /users/alice cannot be partitioned: expected object, found array

The fix is to partition one level higher — /users rather than /users/* — so that the array is stored whole, under the key /users/alice. Arrays inside a partitioned value are fine; only the level that a partition splits has to be an object.

Large bundles

Badger, the embedded key-value store, caps how much a single transaction can hold. Since opa build merges all of a bundle's data files into one data.json, a bundle can easily carry more data than fits into one transaction.

OPA splits such a bundle across as many transactions as it needs, so bundles served by a bundle service are not limited by the transaction size. Note that this means activating a large bundle is not atomic: if OPA is killed part way through, the store is left holding part of the new bundle.

Bundles passed to opa run on the command line are written in a single transaction instead, so a bundle larger than one transaction still fails to load that way. Serve it from a bundle service, or raise Badger's memtablesize (see below).

The one thing that cannot be split is a single key: if the value stored under one key is larger than a transaction can hold, OPA reports

value at /users/alice is too large to store: Txn is too big to fit into one request

Adding a partition below that path, so the value is spread over more keys, is usually the right answer. Raising Badger's memtablesize super flag also raises the transaction size limit.

Metrics

Using the REST API, you can include the ?metrics query string to gain insights into the disk storage access related to a certain OPA query.

$ curl 'http://localhost:8181/v1/data/tenants/acme1/bindings/user1?metrics' | opa eval -I 'input.metrics' -fpretty
{
  "counter_disk_read_bytes": 339,
  "counter_disk_read_keys": 3,
  "counter_server_query_cache_hit": 1,
  "timer_disk_read_ns": 40736,
  "timer_rego_external_resolve_ns": 251,
  "timer_rego_input_parse_ns": 656,
  "timer_rego_query_eval_ns": 66616,
  "timer_server_handler_ns": 117539
}

The timer_disk_*_ns timers give an indication about how much time was spent with the different disk operations.

Available timers are

timer_disk_read_ns
timer_disk_write_ns
timer_disk_commit_ns

Also note the counter_disk_* counters in the metrics:

counter_disk_read_keys: number of keys retrieved
counter_disk_written_keys: number of keys written
counter_disk_deleted_keys: number of keys deleted
counter_disk_read_bytes: bytes retrieved

Suboptimal partition settings can be spotted when the amount of keys and bytes retrieved for a query is disproportional to the actual data returned: the query likely had to retrieve a giant JSON object, and most of it was thrown away.

Debug Logging

Pass --log-level debug to opa run to see all the underlying storage engine's logs.

When debug logging is enabled, the service will output some statistics about the configured disk partitions and their key sizes.

[DEBUG] partition /tenants/acme3/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme4/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme8/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme9/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme0/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme2/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme6/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)

Note that this process will iterate over all database keys. It only happens on startup, when debug logging is enabled.

Fine-tuning Badger settings (super flags)

While partitioning should be the first thing to look into to tune the memory usage and performance of the on-disk storage engine, this configurable gives you the means to change many internal aspects of how Badger uses memory and disk storage.

danger

To be used with care!

Any of the Badger settings used by OPA can be overridden using this feature. There is no validation happening for configurable options set using this flag.

When the embedded Badger version changes, these options could change too.

The configurable options correspond to Badger options that can be set on the library's Options struct.

The following options can not be overridden:

dir
valuedir
detectconflicts

Aside from conflict detection, Badger in OPA uses the default options you can find here.

Conflict detection is disabled because the locking scheme used within OPA does not allow for having multiple concurrent writes.

Example

storage:
  disk:
    directory: /tmp/disk
    badger: nummemtables=1; numgoroutines=2; maxlevels=3

Disk​

Partitions​

Partitioned values must be objects​

Large bundles​

Metrics​

Debug Logging​

Fine-tuning Badger settings (super flags)​

Example​