Today we're excited to announce that the creators of Open Policy Agent (along with many team members from Styra) have joined Apple to continue our decade-long mission of delivering an open source solution to unifying policy across the cloud-native stack.

Apple is an enthusiastic OPA user, utilizing it as a key component of its authorization infrastructure to manage a vast portfolio of global-scale cloud services. Today's announcement demonstrates Apple's commitment to the OPA project by making a larger investment in the technology and the community.

Open Policy Agent has a rich and vibrant community of end-user organizations, vendors, and individuals each contributing ideas, integrations, docs, and code so anyone in the world can use OPA to enforce the policies they care about. We've been fortunate to see such a community grow over the last decade, and look forward to continuing our contributions to the project as active community members.

We've compiled a set of FAQs below to address questions.

What does this mean for the Open Policy Agent project? Open Policy Agent remains a CNCF graduated open source project and there are no changes to the project governance or licensing.

What does this mean for Open Policy Agent maintainers? There is no change to the list of maintainers, except for an organization change from Styra to Apple for the maintainers that are joining Apple.

What will happen to the tools I use under the Styra GitHub? We've initiated the community process for these repositories to be included in the CNCF OPA GitHub organization with the goal of deeper collaboration with the open source community:

• Styra's commercial distribution of OPA, EOPA: an optimized version of OPA designed for data heavy workloads with data-filtering functionality that was previously only available to enterprise customers.
• OPA Control Plane: a new control plane for OPA capable of building bundles from git and additional datasources and deploying them to S3 on AWS, GCP, and Azure.
• SDKs: SDKs for integrating with OPA including TypeScript, React, UCAST-Prisma, C#, ASP.NET, Java, and Springboot.
• Regal: a linter for OPA's policy language Rego (partly written in Rego itself).

Other tools will be evaluated for contribution in the future and remain publicly available for the community.

What will happen to the OPA website and Rego Playground? The OPA website continues to remain available and managed by the CNCF and broader OPA community. The Rego Playground continues to be operated by Styra with no changes to current functionality.

What is the planned roadmap for OPA? We're excited to continue the development of OPA with the same monthly release schedule. The 2025 OPA roadmap includes the following categories of work:

• Language extensions (and/or/not, keywords in refs, string interpolation, partial-set-functions, ellipsis)
• Type checking improvements
• Tooling improvements (streaming OPA test results, debugger attach to trace, rule tracing)
• Partial evaluation improvements (redundant expression elimination, 'in' handling)
• Performance (multiple expression indexing, faster loading of dependency-free bundles)
• Decision API and logging (congestion back-pressure, logging metadata, logging to disk)

Swift OPA builds on the robust foundation of OPA's Intermediate Representation (IR) plans. Introduced in 2022, the OPA toolchain gained the ability to compile Rego policy into IR, precisely defining the concrete steps necessary to evaluate the policy. Swift OPA then interprets these IR plans, allowing you to leverage OPA's rich policy natively within Swift applications and services.

Until today, integrating OPA with a Swift application or service required one of the following approaches:

• Inter-process communication to a standalone OPA instance
• Evaluating Rego policy compiled into a Web Assembly (WASM) binary

Each of these approaches represent different ways of using OPA. For most of the OPA project's history, it's been used with a client/server architecture where communication takes place over the network. The embedded approach taken by WASM, on the other hand, avoids this step to enable in-process evaluation.

With its native language support, Swift OPA works more closely to the embedded approach described: evaluation occurs within the process boundary of your service or application. When developing Swift applications, this reduces latency, as well as complexity and operational overhead by simplifying builds and deployments.

Current Status​

Swift OPA development is currently focused on expanding support for OPA's built-in functions, starting with over 80 built-ins and a roadmap to add many more in the future. A rigorous conformance test suite has also been published to GitHub. A community discussion has been initiated about how to best publish the transformed data used by the test suite, with the goal of ensuring consistent behavior across different language implementations of OPA including Swift OPA.

It's exciting to add Swift support to the OPA community, and you're invited to start exploring Swift OPA today.

How to get involved​

You can explore Swift OPA now at https://github.com/open-policy-agent/swift-opa, and join the community on the Open Policy Agent Slack channel: #swift-opa. Your feedback and ideas are welcome - and there are plenty of opportunities to contribute to the project!

We are excited to announce OPA 1.0, a milestone release consolidating an improved developer experience for the future of Policy as Code. After nearly 10 years of innovations and contributions from over 450 developers, OPA 1.0 is finally here. The release makes new functionality designed to simplify policy writing and improve the language's consistency the default. This release marks the beginning of a new era for our project and represents a robust foundation for Policy as Code projects in the years ahead.

Since the project's CNCF graduation at the start of 2021, our work has been focused on Rego's developer experience and consistency. While much of the new functionality has been around for some time, OPA 1.0 will make new features the default — warranting the SemVer bump to v1.0.0. At this time, we would like to take the chance to highlight the following key changes to the defaults in Rego v1:

• Using if for all rule definitions and contains for multi-value rules is now mandatory, not just when using the rego.v1 import.
• Other new keywords (every, in) are available without any imports.
• Previously requirements that were only run in "strict mode" (like opa check --strict) are now the default. Duplicate imports and imports which shadow each other are no longer allowed.
• OPA 1.0 comes with a range of backwards compatibility features to aid your migrations, please see the v0 compatibility guide if you must continue to support v0 Rego.

Many users have already started using the new syntax (we started adding the new keywords three years ago back in v0.34.0) but for those who haven't; in order to get the best of OPA 1.0; and to remain abreast of follow on updates; users are encouraged to upgrade as soon as possible (using the backwards compatibility functionality if required). Most users will be able to update their Rego quickly as the process can be largely automated using the Rego tools already built into OPA. The process can be summarized as follows:

• Download and install an OPA 1.0 binary before running the following commands.
• opa check --v0-v1: Find parser and compiler errors that might be present in old code.
• opa check --v0-v1 --strict: Find problems in Rego code no longer permitted in OPA 1.0.
• opa fmt --write --v0-v1: Automatically update code to the OPA 1.0 syntax.
• regal lint: using Regal, the linter for Rego is also recommended to find bugs and performance issues.

For users looking for more detailed information, please see the following resources: OPA Documentation on upgrading to 1.0; the Renovating Rego post for older projects which digs into the above process in detail; the Maintainer Track presentation from KubeCon NA 2024 for a video overview.

Those integrating with OPA's Go packages — both via the SDK and the low-level Rego package — are encouraged to update their applications to use the new v1 packages. This is also documented in the upgrading documentation and is a straightforward process. In some cases, users or integrators will need to support both v0 and v1 Rego simultaneously in the same application. This is generally only applicable to those offering OPA as part of a managed offering where the Rego is controlled by end users. Those who do have this use case, please review the v0 compatibility guide to review the most suitable option for your application.

One last thing, while OPA 1.0 is primarily about consolidating the Rego developer experience, the release also comes with some significant improvements to performance. Check the release notes to dig into the details.

And that's a wrap! OPA 1.0 is here, it's a milestone release for our project and we're proud of the effort this release represents. We're excited for you to upgrade and experience these improvements firsthand. As you do, please report any issues via Slack or GitHub Discussions, and feel free to open bug reports or feature requests to help shape future releases. Also, don't forget to use the Regal language server to ensure your Rego code is efficient & error-free too.

We want to extend our gratitude to our incredible community for always testing the latest versions, reporting bugs, contributing code and supporting fellow community members on their Rego journeys. We're not done yet and look forward to working together for years to come.

Happy holidays — the OPA team

December 28th marked the 8th anniversary of the first commit in the Open Policy Agent project. 5000+ commits from more than 400 contributors later, we're starting to prepare for OPA 1.0.

Following the rules of semantic versioning, one would be excused to think of 1.0 as the first "stable" version. That's not really the case for this project. Since the first public releases of OPA, great care has been taken to ensure new changes don't break existing policy. Thousands of organizations have come to rely on OPA for policy enforcement across the whole stack — often for critical production use cases.

While many features have been added to OPA and the Rego language since 2015, few have ever been removed. All this means that almost any policy written eight, five or three years ago still evaluates using the very latest version OPA, just as it did when it was written! But to keep adding features without ever being able to remove things that might not have worked out as we imagined — or simply didn't age well — also comes at a price. An increased cost of maintenance for sure, but more importantly, old ideas, language constructs and built-in functions all add to the cost of learning Rego, compared to having policy look and act consistently using only modern equivalent features.

With OPA 1.0, we're aiming to fix this.

TL;DR​

If you have a busy day ahead and want to get right to something actionable — here's what you can do. From OPA v0.59.0 and onwards, you can start to prepare for the changes in the upcoming 1.0 release following these steps:

• Use import rego.v1 in each of your Rego files. This replaces all future.keywords imports from previous versions, and is all you need to import until OPA 1.0. Use opa fmt --rego-v1 to format your policy with automatic additions of OPA v1.0 constructs, like if, contains and more. This will also replace any future.keywords imports with the rego.v1 import.
• Use opa check --rego-v1 to ensure your policy is compatible with "Rego 1.0" mode.

We'll get back to practical concerns by the end of the blog, but before that, let's see what changes are planned for the first major OPA version.

Changes to Rego coming in OPA 1.0​

Note: Below follows a non-comprehensive list subject to change. While these features are planned — and most of them even implemented already! — updates may be made before the final 1.0 release. I'll do my best to keep this blog post up to date, as we will with the OPA 1.0 documentation.

The future is now — no more import future.keywords​

Several keywords (in, every, if and contains*)* have been added to OPA since the start. Introducing new keywords means there's always a risk that existing policy might break in case identifiers, like rule names or variables, have been named in a way that clashes with the new keywords. In order to prevent this, access to these keywords have required an import of future.keywords. We now live in that future.

OPA v1.0 makes import future.keywords a no-op, as all keywords are now made available everywhere. In the time before OPA 1.0 is released, the new rego.v1 should be used in place of future.keywords imports.

The if keyword made mandatory​

The if keyword helps explain the "inverted if … then" nature of Rego rules, and makes rules easier to read. Additionally, any rule body with only a single expression, like:

allow {
    "admin" in user.roles
}

May with the help of if be expressed as a one-liner, with the curly brackets removed:

allow if "admin" in user.roles

OPA 1.0 makes if a natural — and mandatory — part of every rule's anatomy.

Tip: Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your rules with if added to the rule head.

The contains keyword made mandatory​

The contains keyword helps express multi-value (or as they're often called, partial) rules — i.e. rules that build a set of values. It also helps avoid ambiguities around certain classes of rules, like the fairly recently introduced "nested" rule type:

users.names contains name if {
    # ...
    name := sprintf("%s, %s", [first_name, last_name])
}

The contains keyword was already mandatory for nested rules. OPA v1.0 makes the use of this consistent across all multi-value rules by making contains a requirement.

Tip: Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your multi-value rules with contains added to the rule head.

Strict mode made (mostly) the default​

Most of the rules that have existed in OPA strict mode will be made the default in OPA 1.0. This will help users catch mistakes early, and have them fixed right away. If you have been running opa check --strict as part of your policy build pipeline, you're already in the clear here.

The rules from strict mode that will be made default in OPA 1.0 are:

No duplicate imports​

Duplicate imports should realistically not be a problem in any repo, and the fix is simply to have them removed.

package policy

import data.authz
import data.authz # this is now an error

No deprecated built-in functions​

Deprecated built-in functions will be removed in OPA 1.0, and most of them are trivial to replace using a single line of Rego, or a different built-in function.

package policy

# simply change to use `true in {input.foo, input.bar}`
# using `in` additionally has the benefit that it can be used
# to check for any type of value, and not just boolean "true"
a := any([input.foo, input.bar])

# change to use `every` keyword, e.g.
# every x in [input.foo, input.bar] {
#     x == true
# }
# just like `in` may be used for much more, `every` can be
# used to evaluate complex expressions
e := all([input.foo, input.bar])

# simply use the minus (`-`) operator instead, e.g.
# s3 := s1 - s2
s3 := set_diff(s1, s2)

# simply change to use regex.match instead
r := re_match(..)

# simply change to use net.cidr_intersects
n := net.cidr_overlap(..)

# cast_array, cast_set, cast_string, cast_boolean, cast_null, cast_object
# use the "is_x" equivalent built-in function in their place
s := is_string("yes")

input and data now reserved keywords​

OPA 1.0 prohibits the use of input or data as identifiers:

# this is not allowed in OPA 1.0
input := "overloaded"

# and neither is this
data := {}

Overloading input has mostly been common in tests. Do note that with input as {..} remains valid. If you're using assignment to input however, (input := {..}), you'll just need to change the name to something like inp instead.

Other changes coming in OPA 1.0​

See the 1.0 tag in the OPA backlog for a list of all issues related to OPA 1.0. Do note though that not all issues marked 1.0 might be picked for inclusion, and new issues may pop up before the release!

Other notable changes include:

import rego.v1​

As previously mentioned — beginning with OPA v0.59.0 a new handy import to help with the 1.0 transition is made available. By adding import rego.v1 to a Rego policy, you can tell OPA to treat the policy just as it will handle it once version 1.0 is released.

• Since rego.v1 implies all the (no longer) future keywords, the importing future.keywords is no longer needed when import rego.v1 is present, and will in fact be an error.
• Just as in OPA 1.0, the use of if and contains will be enforced
• The strict mode requirements brought in OPA 1.0 will be checked automatically

Bind server to localhost interface by default​

In OPA 1.0, the server will bind to the localhost interface by default, and not 0.0.0.0 (all interfaces). This change is needed in order to avoid accidentally exposing OPA to the internet, which while uncommon (as OPA normally runs behind firewalls and gateways) still happens, and we should aim to provide a secure default. Should you still want to bind against 0.0.0.0, or some other interface, you can use the --addr flag of the opa run command, like opa run --server --addr 0.0.0.0:8181. The impact of this change is expected to be small, but good to keep in mind.

Documentation​

The OPA docs have been updated to cover much of what's mentioned in this blog in greater detail. It also covers more of the technical reasons some of these changes are needed. See the docs on OPA 1.0 for more information.

How to prepare​

As we covered in the TL;DR section, we're providing a number of tools to help with the transition starting from OPA v0.59.0 already. These tools will likely be extended and improved in following releases, but starting to use them today will ensure as smooth transition as possible.

To summarize​

• OPA 1.0 planned for release this year, including some backwards incompatible changes
• Starting now, you should use import rego.v1 in all of your policies (this replaces future.keywords) imports
• Use opa check --rego-v1 for testing compliance against 1.0
• Use opa fmt --rego-v1 to have your Rego code updated for 1.0 compliance
• Run the OPA server with the --v1-compatible flag for OPA 1.0 compliance

Also worth pointing out — following guides like the Rego Style Guide, and using tools like Regal, is an excellent way to ensure not just compliance with future changes to Rego, but that your current policy repo is continuously kept in the best possible condition.

If you have any questions, concerns or would like to provide feedback around the upcoming 1.0 release, or the tools made available to help you transition smoothly — don't hesitate to reach out using any of the below channels:

• The OPA community's discussion board
• The OPA Slack
• The OPA project's backlog (to file an issue or feature request)

As 2023 draws to a close, the time has come to reflect on another important year for Open Policy Agent (OPA). Now more than two years deep into CNCF Graduated status, OPA continues to see accelerated growth in production deployments — and across a diverse range of use cases. Such use cases demand both performance and stability, while user base growth depends on learning resources and ease of use. This year, the OPA community has worked hard and delivered on all fronts, for new and experienced users alike. This post takes time to share how this was achieved; highlight prominent events and updates; celebrate input from the wider community and set the scene for a historic year of OPA in 2024.

OPA 'Away From Keyboard'​

While OPA users and maintainers predominantly collaborate online, there were a good number of occasions where OPA existed very much in the physical realm this year too.

KubeCon EU enabled a few OPA events in Amsterdam early this summer. For the first time ever, an OPA-themed ContribFest session was held, where OPA, Conftest and OPA Gatekeeper maintainers worked with new contributors to the different OPA projects. In Amsterdam we also saw an OPA meet-up where speakers from Miro, Bankdata and Styra presented. At this KubeCon EU there were four OPA talks:

• The Compliance Business Case for Kubernetes in the EU: Anders Eknert
• Open Policy Agent. (OPA) Intro & Deep Dive — Charlie Egan, Rita Zhang
• Calling OPA from eBPF, Through WASM, in the Kernel? You've Gone Mad! — Nandor Kracser
• Scratching an Itch: Running Policy in Hard to Reach Places with WASM & OPA — Charlie Egan

Rolling forward a few months, OPA also had a strong presence in Chicago at KubeCon NA. KubeCon is a huge event and it was great to get so many eyes on OPA as part of the graduated projects update in the keynote session. On top of that, the OPA kiosk in the project pavilion was an important meeting place for maintainers and users at the event. Discussions covered all sorts of use cases from authorization of applications, Kubernetes admission, IAC policy and beyond. Don't forget to check out the OPA project update from the conference's maintainer track.

From Strength to Strength​

OPA grows in so many different ways each year it's sometimes hard to know how to quantify it. Here are some highlighted figures which illustrate OPA's trajectory as we enter 2024.

2700 Contributors. Nearly 3000 people have helped make OPA into the project it is today. Contributors help make OPA better by making changes to docs and code; by participating in GitHub discussions and by filing bugs. What's equally impressive is how these contributors are from over 450 different companies. OPA is a general purpose, domain agnostic policy engine so it's vital the project is guided by such a varied contributor base.

9 years spent by users reading the documentation on the OPA website. This year OPA contributors worked hard and made over 160 updates to the docs; and so it's reassuring to look back at the end of the year and see just how many users benefited from the hard work.

2000 Go repositories build on OPA. Integrating with OPA has always been a priority so it's fantastic to see that just so many different projects are adding policy functionality in this way. With the OPA SDK, it's possible to bring all the best parts of OPA right into your Go application making it a powerful tool when standardizing your policy as code stack.

1.5 million Playground Runs. The Rego Playground is for every OPA user, it's there as a learning tool, as a collaborative scratch pad and now also integrates the output from Regal, the new linter for Rego. On average, every 20s someone clicks the 'Evaluate' button on the playground, all day long, all year long. One of the major uses of the playground is for users and maintainers collaborating on support in the OPA Slack, if you're interested in getting help within your team or on the Slack, creating a minimal example on the playground is a place to start.

It's not just OPA's community that's moving forward in leaps and bounds, OPA itself has been keeping pace and has received loads of great updates this year too. Let's dig into that now.

New Features​

General references in rule heads​

The single most important addition to Rego this year was arguably general references in rule heads. Simply put, it is now possible to include variables in rule names (or "references"), making it possible to build complex, nested map structures which would previously require multiple rules distributed over several packages.

Example using dynamic policy composition to collect informative notices from all "rules" policies, and have them organized by category and title.

grouped_notices[category][title] contains notice if {
    some category, title
    rules_to_run[category][title]

    some notice in data.rules[category][title].notices
}

Output would be a nested structure, as expected:

{
  "grouped_notices": {
    "testing": {
      "file-missing-test-suffix": ["ignored"]
    },
    "custom": {
      "naming-convention": ["ignored"],
      "one-liner-rule": ["obsolete", "ignored"]
    }
  }
}

For more examples and information, see the OPA docs on the topic.

Default keyword on functions​

The default keyword has been around since forever, and is considered idiomatic for scenarios where a "fallback" value is needed, should rule evaluation fail in other rules sharing the same name. A long requested feature has been to extend support for default to cover custom functions, and 2023 was the year it happened.

package functions

default first_name(_) := "unknown"

first_name(full_name) := split(full_name, " ")[0]

New built-in functions​

Seven new built-in functions were added to Rego this year. The json.verify_schema and json.match_schema functions are both recent additions for evaluating policy against JSON schemas — a use case that's been increasingly common in recent times. The time.format function will help policy authors present dates and time using either a custom format, or one of the supported constants for datetime formats that were also added this year. Three new crypto functions were added: crypto.hmac.equal, crypto.x509.parse_keypair, and crypto.parse_private_keys. Finally, the new numbers.range_step function, which works just as numbers.range, but with a configurable step value.

Package scoped annotations​

A metadata annotation using the scope of package is now truly scoped to the package, and not just the file in which it is declared. This allows for some interesting opportunities to separate metadata declarations from "implementing" packages, and build things like lightweight frameworks leveraging Rego's metadata annotations. Additionally, it'll allow defining package scoped annotations for "packages" created via general references in rule heads, where the package scope isn't directly allowed on the rule itself.

Debugging​

The Swiss army knife of OPA also known as "opa eval" got a new flag to help debugging policy this year. Using the --show-builtin-errors flag, policy authors may now get a list of all errors produced by built-in functions as part of evaluation, making it much faster to identify certain types of problems.

Performance​

OPA keeps getting faster, and in 2023 we saw some great improvements in this area. The json.patch built-in function was remodeled entirely and now performs extremely well even when provided with a huge list of changes. Performance isn't entirely in the hands of OPA though. Some optimizations can only be performed at the level of an actual Rego policy, and OPA provides several tools to help policy authors with this. The profiler (opa eval --profile) is one such tool, and from this year it'll now also include the number of generated expressions in evaluation. This helps policy authors better understand why some expressions are evaluated more times than what one might expect.

Server​

Several improvements to the server (and by extension, the OPA SDK) landed in OPA this year. Bundle fetching now works with AWS Signing Version 4A, allowing bundles hosted on AWS to be distributed across different geographical regions. Also, a new shorthand format for quickly running the server pointed at a remote bundle was introduced. The OCI downloader saw several new authentication methods added. Finally, instance labels may now be added via discovery allowing for greater flexibility in runtime re-configuration of long running OPAs.

Monitoring​

Given the large number of OPA instances running in production, having a good story around monitoring is essential. The status API provides a way for any OPA deployed to report its current status to a centralized control plane or monitoring system. In 2023, several new metrics got added to the status reports, including most notably the request count for unauthorized calls to the OPA REST API when an authentication/authorization policy is in use, as well as errors that might have happened in decision logging.

Security​

2023 was the year the OPA docker images finally made rootless the default, and the special "-rootless" images that previously existed for this purpose are now obsolete. If you're still using them, make sure to remove the suffix from the image on your next version upgrade!

Ecosystem​

Gatekeeper​

The OPA Gatekeeper project had a busy 2023, with many improvements landing this year. The external data feature allows users to connect with external data sources as part of policy evaluation. This year it gained support for caching of responses from external data providers for both audit and admission. A new AssignImage mutator which enables mutation of image registry or tag was also made available. The new PubSub feature (currently in alpha) enables users to subscribe to pubsub services to consume a large number of audit violations. Additionally, observability statistics for admission, audit and gator CLI are now available!

Speaking of the Gator CLI — the tool now prints violating object names on test output, and additionally supports trace and image flags. It may now also be provided an AdmissionReview object for verification.

Using the new (experimental) Kubernetes Native Validation feature, users can now write CEL (Common Expression Language) based rules in addition to Rego rules in constraint templates, similar to Kubernetes ValidatingAdmissionPolicy. Finally, the ExpansionTemplate feature, which enables validation of workload resources, has graduated to beta.

Conftest​

The Conftest project saw many improvements around tooling this year. A new --strict flag was added to the verify and test commands, which will enforce additional safety checks on the policies such as unused arguments, duplicate imports, and more. Two more flags got added: the --quiet flag to the verify command which will silence success notifications and only show errors, and a --config flag which allows users to specify where the config file to be tested lives. Test results may now also be emitted in a format compatible with Azure DevOps. On the topic of formats, a new input format was added to the already long list of supported ones, and textproto files may now be targeted for policy evaluation too. Finally, the Confest Docker images now also support both the linux/amd64 and linux/arm64 platforms.

OPA Ecosystem​

One goal of the OPA project is to build a domain agnostic policy engine. Being domain agnostic is achieved by simultaneously building generic core policy functionality, while also supporting a range of out-of-the-box integrations for different use cases. This year, the wider OPA community has wholeheartedly delivered on the latter and listed 22 new integrations on the website. The OPA Ecosystem also has a new home as a top level page, where integrations can be browsed by category, check it out!

Most new ecosystem additions this year have been with other open source tools, generally adding policy functionality to a larger tool or leaning on Rego to provide a solid foundation for a domain-specific policy tool. Some notable examples include:

• Source Code Management: Reposaur, a repository compliance tool; and Legitify, a repository security configuration scanner.
• Supply Chain Security: dependency-management-data, helps understand software dependency posture; and Enterprise Contract verifies supply chain security artifacts with Rego policy.
• Infrastructure CD checks: Torque, Spinnaker integrate Rego-based checks for continuous deployment while ccbr and ​​BrainIAC support a range of checks on existing IAC codebases.
• Extending Authorization with OPA: The data orchestration tool Alluxio now also supports delegation of permissions to OPA.

Digger, an open source CI/CD orchestrator for Terraform both integrates OPA for user RBAC and leaning into existing tooling by leveraging OPA project conftest for IAC policy.

Meanwhile, other integrations went deeper and applied Rego in previously unexplored ways. regocpp is a cutting-edge project from collaborators at Microsoft that aims to bring Rego to other environments, natively. Based on C++, regocpp supports a number of Rego built-ins and the grammar as of v0.55.0.

The aforementioned linter, Regal also pushes the boundaries of where Rego can be used to write policies. Using the JSON representation of the Rego abstract syntax tree, this project implements a range of linting rules… in Rego! Regal has already been deployed by a number of open source Rego policy libraries and now supports over 60 rules. Integrated with the Rego Playground the linter is already available to everyone. There's no doubt that this will be a great tool for OPA learners and long-timers alike while continuing to help scale the OPA community.

If you're interested in listing your OPA integration or project, please see the instructions or stop by the #ecosystem channel in the OPA slack if you have any questions.

Thanks​

2023 was an exciting year for OPA and its community. With so many projects using, integrating or extending OPA for all sorts of use cases — and so many users helping to contribute in all sorts of ways — this community is truly a great place to be. Thank you all who helped make it so! Your efforts are seen and appreciated.

There's a lot of great stuff lined up for next year already, so buckle up, and let's import future.2024!

Special thanks to Charlie Egan, Rita Zhang and John Reese for having helped contribute to this blog.

It's a new year, and once again it's time to look back and reflect on the year that passed in the world of Open Policy Agent! 2022 was OPA's first full year as a CNCF graduated project, and while it would be easy to think that things would slow down after reaching that point of maturity and recognition, things have rather sped up. In community growth as well as pace of development — both of which we'll take a closer look at here.

This year, tech communities like ours enjoyed finally getting to meet in person again, and major conferences in our space, like KubeCon / CloudNativeCon, saw thousands of attendees in both Europe and North America. We also saw meetups, hackathons and other smaller gatherings move back from virtual to in-person events. On the topic of events — let's start our review of the year 2022 for Open Policy Agent there.

Notable Events​

2022 User Survey​

While we queried our nearest OPA instance for policy decisions, we also queried the OPA community to learn about their experience of interacting with the project, documentation and tooling. The survey results provided us valuable insights into where we might want to spend some extra time and effort in 2023 — richer documentation with more examples was requested by many. We're also seeing a need to better highlight the benefits of the various management capabilities in OPA, like decision logging and monitoring. An interesting trend which continues from 2022 is the increasing number of OPA use cases inside organizations. While infrastructure policies (including Kubernetes) still come out on top, we're seeing more organizations embrace OPA for policy across their whole stack. A standardized way of working with policy was always a goal of the project, so it's really exciting to see more organizations seeing the benefits of this approach!

Open Policy Day with OPA​

This year we were excited to see an entire event dedicated to OPA — the Open Policy Day with OPA co-located with KubeCon North America. During the course of the day, attendees got to hear end-user stories on using OPA in production, with speakers from organizations like Nvidia, T-Mobile. Capital One, Chime and Snowflake. If you couldn't attend in person, all the talks are up on YouTube!

Conferences and Meetups​

In the cloud-native space, OPA was represented in talks at both KubeCon / CloudNativeCon Europe as well as North America. As the interest in policy as code grew over the year, we saw a number of talks, workshops and events focused on the topic, and many discovered the benefits of unified policy management across tech stacks and organizations.

Additionally — while OPA has been, and continues to be, a popular topic at cloud-native, security and DevOps themed meetups, it's great to see new meetup groups dedicated entirely to the topic of OPA. Several OPA meetups took place in 2022, and we'd love to see that trend continue in the new year! If you'd like to host your own, let us know.

It was great to see so much buzz around OPA at these events, and much of that is thanks to the amazing work going on within the OPA projects — let's look into what's been going on there next!

New Features​

OPA and Rego saw a record number of new features added in 2022.

New Keywords​

A few new keywords made a big difference both to the aesthetics of Rego, as well as its functionality. First off, the new every keyword elegantly helps solve the problem of expressing "for all" type of queries:

import future.keywords.every

only_dev_servers {
    some site in sites
    site.name == "dev"

    every server in site.servers {
        endswith(server.name, "-dev")
    }
}

Next, the new if keyword helps policy authors express their rules in the same way as they normally should be read — as conditional assignments: "allow is true if conditions x, y and z are true". As an added bonus, the if keyword allows skipping the braces around single-line rule bodies, leading to rule constructs that read more like plain English:

allow {
    "admin" in input.user.roles
}

May now be written as:

allow if "admin" in input.user.roles

Similarly, the new contains keyword allows set-building partial rules to be expressed as "the set contains x if conditions x, y and z are true":

deny[message] {
    input.user.security_clearance_level < 2
    message := "Security clearance level 2 or higher required"
}

May now be written as:

deny contains message if {
    input.user.security_clearance_level < 2
    message := "Security clearance level 2 or higher required"
}

Refs in Rule Heads​

It is said that one of the hardest things in computer science is naming things… "refs in rule heads" seems to support that claim! Don't let the name intimidate you though, this is a great addition to Rego! Dynamically creating deeply nested objects would previously often require the use of nested packages, with each package provided in its own file. This is no longer the case, as nesting can now be expressed in one place:

package policy

claims.user.name := concat(" ", [input.user.first_name, input.user.last_name])

request.method := input.request.method
request.valid := validate(input.request)

Evaluating the policy package will now provide something like the below result:

{
  "claims": {
    "user": {
      "name": "John Doe"
    }
  },
  "request": {
    "method": "PUT",
    "valid": true
  }
}

Built-in Function Metadata and Metadata Introspection​

Another exciting addition to Rego this year was the introduction of metadata annotations. Previously, policy authors would have to devise custom methods and formats for annotating their policies, packages and rules with metadata in the form of comments. Native support for annotations now provides a unified way not only for authoring structured annotations, but also for parsing them using either the "opa inspect" command, programmatically from Go, or even from Rego policies themselves via the new built-in rego.metadata functions. Having a standardized way of annotating packages and rules with metadata should benefit both human consumers as well as tooling.

Testing​

A major advantage of treating policy as code is that code is testable. Unit testing provides effective guardrails around policies and rules, and allows frequent updates without the risk of breaking things. While test-driven development has been considered a best practice for Rego since the start, one feature that many found missing was the ability to mock functions. Function mocking allows replacing built-in functions, like http.send, or custom functions, with different implementations, commonly free of side-effects, all using the same with keyword familiar to test authors.

Policy and Data Distribution​

Policy is commonly only half of the equation when OPA makes decisions — having access to up-to-date data related to users, endpoints or resources is often just as important. Data tends to be more dynamic in nature than policy, and certain types of deployments require a lot of data. The combination of huge datasets and frequently changing data would previously require continuous transfer of all required data in a bundle — even when only a few attributes had been updated. The introduction of delta bundles solves this by providing a new bundle type containing only the changes to data made since the last fetch, i.e. the delta. A much-awaited feature, and one that will help ensure OPA covers even the most complex of use cases going forward.

Another introduction this year was support for OCI bundle registries. In an increasingly containerized world, distribution of applications is no longer the only use case for containers, and providing policy and data using the same channels as you would for e.g. Docker images, simplify things considerably in many environments.

Disk Storage​

While delta bundles may solve the problem of distributing large volumes of data, that data must eventually still be stored somewhere for OPA to make use of it. Up until this year, only a single option for storage was provided: in-memory. While this is normally the best place to store it for fast access, large datasets distributed in-memory across a large number of running instances quickly adds up, and many are those who learnt the hard way that the old slogan of "memory is cheap" isn't always true at scale. The disk based storage option now provides a balance between performance and costs, and is a welcome addition to OPA in many types of integrations.

Compiler Strict mode​

The Topdown compiler in OPA was enhanced with a new strict mode option, allowing policy authors to catch common mistakes before deploying their policy to production. Unused imports and variables, or use of deprecated built-in functions — now all flagged by the compiler with strict mode turned on. Together with JSON schema-based type checking, developers are provided some powerful tools to ensure the robustness of their policy. Guard rails around the guard rails!

Intermediate Representation (IR)​

The OPA project has the ambitious goal of standardizing policy across the full stack, because of this it's sometimes necessary to consider alternative deployment models to the traditional standalone service. This year we saw OPA provide a new intermediate representation format, allowing custom implementations to parse and execute evaluation plans. An implementation for the JVM and Javascript has already been made available with the Jarl project, and hopefully we'll see more to follow in 2023!

Built-in functions​

20 new built-in functions got added in 2022 — more than any year before! We saw new functions in almost every existing category, and OPA's exciting new GraphQL capabilities create a category of their own.

GraphQL: graphql.is_valid, graphql.parse, graphql.parse_and_verify, graphql.parse_query, graphql.parse_schema, graphql.schema_is_valid

Strings: indexof_n, regex.replace, strings.any_prefix_match, strings.any_suffix_match

Objects: object.subset, object.union_n, object.keys

Crypto: crypto.hmac.md5, crypto.hmac.sha1, crypto.hmac.sha256, crypto.hmac.sha512

Misc: providers.aws.sign_req, net.cidr_is_valid, graph.reachable_paths

Performance Improvements​

Some improvements are more understated, though no less noteworthy. As a mature software project, deployed in production in thousands of organizations across the world, OPA needs to be both robust and performant. This year saw the following exciting improvements in OPA performance:

• The — optimize flag now works for more commands (previously only available for opa build)
• Lazy objects optimization allows delaying evaluation of attributes until needed
• Built-in function optimizations for object.get, in, object.union_n, and others
• Two new highly optimized built-in functions for doing prefix and suffix matching en masse: strings.any_prefix_match and strings.any_suffix_match
• Internal optimizations to set element addition, object insertion and set union.

Ecosystem and integrations​

Gatekeeper​

A whole lot of great things landed in the Gatekeeper project this year! Following recent developments, Gatekeeper was made compatible with the Kubernetes v1.25 shift from Pod Security Policies to Pod Security Admission. On the topic of Kubernetes workloads, the new Validation of Workload Resources feature allows writing rules that apply to any Pod spec, whether deployed as a standalone pod or embedded in a parent resource, like a Deployment. Similarly, Gatekeeper now also allows validating subresources. The external data feature, which, as the name implies, allows Gatekeeper to interface with various external data sources for validation and mutation, moved to beta this year, and the next feature to do so is Gator, which allows testing of Gatekeeper ConstraintTemplates and Constraints in a local environment. Finally, the mutation feature is now considered stable.

In addition to all the features listed above, Gatekeeper is now faster than ever before! Some of the most notable improvements include reduced time for template compilation, adding and evaluating constraints, a whopping ~20X reduction in persistent audit memory usage, reduced request duration for policies with replicated data and reduced CPU time when adding data to OPA storage.

The ecosystem around Gatekeeper also saw improvements this year, where the most notable ones were the new Gatekeeper Policies website, and the inclusion of Gatekeeper policies on ArtifactHub.

Conftest​

Conftest saw a rapid pace of development this year, with 12 releases pushed — from version 0.29.0, and ending in version 0.37.0! The project added support for policy authoring using a number of additional file types — like env, hcl, jsonc, CycloneDX, and SPDX — as input. Possibly even more exciting is the addition of several new built-in functions exclusive to Conftest, like parse_config, parse_config_file, and parse_combined_config. These functions all allow policy authors to pull in configuration to test from inside of a policy, allowing a greater deal of flexibility in how config tests are executed.

The tooling around Conftest improved as well: when using the --version flag, the version of OPA used by Conftest will now also be displayed. Additionally, the new --quiet flag allows excluding anything but errors in the output, which should help in quickly identifying issues.

Integrations​

OPA would not be what it is without its massive ecosystem of tools, integrations and useful and fun projects. The year started out with some great news in the infrastructure space, with AWS opening up for the possibility of externalizing compliance checks of CloudFormation templates via hooks, and it did not take long for the AWS CloudFormation hook for OPA to arrive on the scene. Later this year, Hashicorp announced support for OPA in their Terraform Cloud offering. A Pulumi integration was also added to the ecosystem. The message seems clear — the tool to use for infrastructure as code (IaC) compliance is OPA, and the language to define IaC policies is Rego!

Outside of the infrastructure space, we saw a number of interesting integrations being built by the community, like Alfred, a Rego Playground you can self host, a CircleCI integration for CI/CD pipeline policies, fig support for command line auto-completion goodness, self-sovereign identity (SSI) integrations, and even policy-driven access to remote systems via SansShell. OPA-powered policy enforcement even made it to the desktop this year, with the CISA-developed ScubaGear project using Rego for validating M365 tenant configurations!

Finally, a much awaited addition to the OPA ecosystem — the Rego Style Guide now offers policy authors a comprehensive set of rules and best practices for authoring Rego.

For a more comprehensive list of OPA integrations, check out the OPA ecosystem page, and the Awesome OPA list.

Credits​

None of the above would have been made possible without the amazing community around OPA. In 2022, we saw an incredible number of people contribute to the project in all imaginable ways — code, documentation, bug reports, support discussions, integrations and tools. OPA is being used more and more widely around the world, and in different domains. With this growth, it'd be easy to overlook the huge effort in growing a community to support the project. We know how hard the community has worked to get to where we are and for that we are immensely grateful. Thank you all for getting us to where we are today and laying the foundation for another fantastic year with Open Policy Agent.

November has arrived and we are looking forward to the holiday season!

Thanks to all of the community members that stopped by the booth at Kubecon, it was a pleasure meeting you!

User Survey​

We are looking for input from the community to see how everyone is using OPA. Take 5 minutes to fill out this 7 question survey to help out the community!

Take the Survey

Ecosystem Updates​

Open Policy Agent v0.46.1​

• New language feature: refs in rule heads
• Entrypoint annotations in rule metadata
• New Built-in Function: graphql.schema_is_valid
• New Built-in Function: net.cidr_is_valid

Gatekeeper 3.10.0​

• Kubernetes v1.25+, removal of Pod Security Policies and migration to Pod Security Admission 🔐
• Mutation is promoted to stable 🦠
• Introducing Validation of Workload Resources as alpha 🚀
• Performance improvements 🏃

Contributor Shout Outs​

Thanks to all of the contributors that participated in these releases, the OPA community wouldn't be here without you!

• @mattfarina
• @jaspervdj
• @ricardomaraschini
• @byronic
• @philipaconrad
• @pjbgf
• @caldwecr
• @hzliangbin
• @peterchenadded
• @phantlantis
• @ericjkao
• @TheLunaticScripter
• @humbertoc-silva
• @Juneezee
• @vinhph0906
• @aholmis
• @Joffref
• @olegroom
• @iamatwork
• @fredallen-wk
• @bartandacc
• @max0ne
• @OpenSourceZombie
• @JAORMX
• @Boojapho
• @ethanrange
• @stp-bsh
• @qa-ship-it
• @salaxander
• @boatmisser
• @gracedo
• @meons
• @mariusblarsen

Community Tools​

circle-policy-agent​

The policy-agent is essentially a CircleCI-flavored wrapper library around the Open Policy Agent (OPA), which will allow the users to write the policy documents in CircleCI terminology.

Star on GitHub

custom-opa-spicedb​

This experiment adds support for querying relations from Authzed / SpiceDB via GRPC to check resource level permissions as custom builtin commands for Open Policy Agent.

Star on GitHub

Videos 🎥​

Policy as Code with Open Policy Agent — Anders Eknert, Styra​

Should user Alice be allowed to read credit reports? Should a cloud instance be deployable without basic security configuration in place? Should service X be allowed to query the database? Policy defines the rules of our systems, but how do we ensure our policies are enforced consistently in increasingly distributed and diverse tech stacks? In this talk we'll explore the benefits of decoupling policy from our applications, deployment pipelines and platforms, and how Open Policy Agent (OPA) can help unify the way we work with policy across the stack.

Securing kubernetes with opa and gatekeeper​

Starts at 3:23:20 as part of the Kubehuddle Edinburgh event.

Blogs​

• I have a plan! Exploring the OPA Intermediate Representation (IR) format
• 5 Application Authorization Best Practices for Better Cybersecurity
• Intro to sets in Rego
• OPA into WASM
• Opa for k8s
• Spring Security Authorization with OPA
• Programming Your Policies: Justin Cormack at QCon San Francisco 2022

Let us know how we did​

The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of. Reach out using one of the links below.

October is here, the leaves are changing colors, and weather is starting to become cool.

The OPA community will be at KubeCon NA, so don't forget to register for Cloud Native Policy Day with OPA!

Register Today!

Community Updates​

The OPA community now has over 300 GitHub contributors!

Our friends over at Postman connected with us to setup an OPA webpage to help our community members explore the OPA API.

The OPA API on Postman

Ecosystem Updates​

Open Policy Agent v0.45.0​

• Feature: Improved Decision Logging with nd_builtin_cache
• New builtin: regex.replace for regex-based search/replace on strings
• Optimization: object.union_n builtin implementation to use a more efficient merge algorithm

Community Tools​

Capua​

A Kafka Policy engine that will help you validate your resources and artefact creations with style.

Like on GitHub

Videos 🎥​

rq: Datalog for your shell pipelines​

rq brings the full power of Rego, a Datalog dialect created for Open Policy Agent (OPA) to your shell pipelines. It allows you to easily transform and query data in a variety of commonly used formats using Rego expressions.

Speaker: Charles Daniels, Backend Software Engineer, Styra Inc.

Blogs​

• How DoorDash Ensures Velocity and Reliability through Policy Automation
• Deploying Gatekeeper policies as OCI artifacts, the GitOps way
• Rego — The unified policy language for better policy management
• Creating Custom OPA Policies with Azure Policy
• Use OPA Gatekeeper to prohibit specific IAM users from creating resources in a specific Namespace
• [Copy and paste OK] Procedure for linking Open Policy Agent with Python

Events 📆​

Cloud Native Policy Day with OPA, Oct 25th​

Cloud Native Policy Day with OPA hosted by Styra, the creators of Open Policy Agent, will bring together the OPA community for a day of sharing and discussing policy-as-code best practices, key learnings and creative use cases for OPA. Project maintainers will be on hand to field 1:1 questions and provide live-coding demos — and you'll see proven real-world implementations from various OPA adopters during each of the sessions.

Whether you're looking to start down your policy journey, or are an OPA adopter with Rego skills to share, join the community for sharing, learning and socializing.

Attendees are invited to come for the full day with lunch provided or to stop by just for the sessions that interest them most. To register for the event, add Cloud Native Policy Day with OPA from the co-located event list selections when registering for KubeCon + CloudNativeCon NA 2022 or add it to your existing registration by selecting "modify" on your confirmation page or clicking the "modify" link in your confirmation email.

👉 Register Today! 👈

Let us know how we did​

The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of.

Happy September Everyone! This month's edition is coming in a little late, but don't worry, it's still packed with great information.

Don't forget to register for Cloud Native Policy Day with OPA! More info at bottom.

Register Today!

Community Updates​

The Rego Playground now has a "Format" button! 🎉

This button auto-formats your policy code in the editor, as well as your input/data JSON documents.

Ecosystem Updates​

Open Policy Agent v0.44.0​

• security fixes, which mitigate CVE-2022-36085 in OPA itself, and CVE-2022-27664 and CVE-2022-32190 in our Go build tooling.
• Linear performance scaling for sets up into the 500k key range and beyond
• The union builtin is now about 15-30% faster than the equivalent operation in pure Rego.
• This release introduces two new builtins: strings.any_prefix_match, and strings.any_suffix_match.

NPM-OPA-WASM v1.8.0​

• New Feature: add loadPolicySync by @elliots in #255

We will discuss these new features in the September 20th Office Hours. Sign up today and send in your questions.

Join OPA Office Hours

Community Tools​

Goast​

Go AST (Abstract Syntax Tree) based static analysis tool with Rego.

Like on GitHub

Java App with OPA Policies​

Motivation for this code and application was to try to understand and implement the Hexagonal Architecture — also called Port and Adapter Architecture.

Test it out

OPA Support for Go Fiber​

Open Policy Agent support for Fiber.

Note: Requires Go 1.16 and above

Try it

Blogs​

Read up on how the OPA community is using OPA.

• Control User Access and Permissions in CVAT with Open Policy Agent
• What Exposed OPA Servers Can Tell You About Your Applications
• Using XACML with OPA and Rego: The Best of Both Worlds
• Authorize REST API with OPA (Japanese)
• Controlling Kafka Data Flows using Open Policy Agent
• Introduction of Open Policy Agent / Rego to realize Policy as Code (Japanese)
• Collaborating on Access Control Policies with Open Policy Agent

Events 📆​

Cloud Native Policy Day with OPA, Oct 25th​

Cloud Native Policy Day with OPA hosted by Styra, the creators of Open Policy Agent, will bring together the OPA community for a day of sharing and discussing policy-as-code best practices, key learnings and creative use cases for OPA. Project maintainers will be on hand to field 1:1 questions and provide live-coding demos — and you'll see proven real-world implementations from various OPA adopters during each of the sessions.

Whether you're looking to start down your policy journey, or are an OPA adopter with Rego skills to share, join the community for sharing, learning and socializing.

Attendees are invited to come for the full day with lunch provided or to stop by just for the sessions that interest them most. To register for the event, add Cloud Native Policy Day with OPA from the co-located event list selections when registering for KubeCon + CloudNativeCon NA 2022 or add it to your existing registration by selecting "modify" on your confirmation page or clicking the "modify" link in your confirmation email.

Register Today!

Let us know how we did​

The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of.

It isn't an overstatement to say that the versatility of Open Policy Agent (OPA) is a key factor in its success. As a general purpose policy engine, OPA needs to handle inputs from a disparate set of systems — Terraform, Kubernetes, CI/CD pipelines or custom applications, to name a few — and deliver decisions in a format understood by that particular system. Providing an agnostic approach to input and output data — anything that is or can be modeled hierarchically in a JSON or YAML document is potentially subject to policy — allows integrating OPA with all kinds of systems, applications and tech stacks.

But how does the data get passed between client application and OPA? Running OPA as a standalone service, and querying OPA for decisions over its REST API, is by far the most common way to integrate OPA, and for good reasons! Running OPA as a separate component provides a nice, unified interface for communication, which commonly involves writing only a few lines of code in most modern programming languages. Additionally, many technologies allow extending the built-in functionality for e.g. authorization by utilizing webhooks, which are commonly REST requests with a JSON-encoded payload. Perfect integration point for OPA!

There are however some scenarios where the standalone REST model proves to be challenging:

• Resource-constrained environments like embedded systems. While OPA is fairly light-weight, some environments simply don't have the resources required to run a standalone OPA service, or networking capabilities for querying.
• Distributed deployments with tight latency budgets, where every millisecond counts.
• Environments constrained by other limitations on what type of software can be run, like web browsers.

To accommodate these requirements, OPA provides a few alternatives to the standalone service model:

• Applications written in Go may integrate directly with the OPA Go API, or through the high-level Go SDK, alleviating the need for OPA to run as a separate service.
• Policy may be compiled into Wasm modules, which can then be evaluated in any Wasm runtime. While the most famous Wasm runtime may be that included in web browsers, most programming languages today offer integrations with a Wasm runtime, allowing (at least the policy evaluation parts of) OPA to run "inside" of the application rather than outside of it. Additionally, OPA itself ships with a Wasm runtime, which makes it possible to have OPA pull down bundles including Wasm compiled policy for faster evaluation, and potentially other benefits.

Wasm​

Rego policies compiled to Wasm modules offers a flexible, highly performant alternative to "regular" policy evaluation, with runtimes available for a wide array of languages, frameworks and platforms. As such, it should be considered an option for any OPA integration where the standalone server model falls short of the requirements. However, as ubiquitous as Wasm runtimes may be, they are not available everywhere. Embedded environments, exotic architectures or specialized hardware all constitute examples of environments where we're unlikely to encounter a Wasm runtime. But even with one available, Wasm itself is not without limitations, even by design!

With the goals of providing a safe, sandboxed environment, originally targeting web browsers, Wasm has several restrictions on what can and can't be done in the confines of the runtime. Interacting with the host system, or for that matter, other host systems — whether through system calls, network requests, or file system operations, is generally prohibited. While the WebAssembly System Interface (WASI) aims to offer an API for this exact purpose, and could potentially be used for certain features of Rego (like the http.send built-in function) in the future, relying on WASI means that a policy evaluated in one runtime might not work in another, as currently only a certain subset of the WASI API is implemented in any given runtime, and Wasm runtimes like those provided by web browsers likely have no interest in supporting interactions with the host system at all. Last, while some great progress has been made around WASI recently, it is still nowhere near the maturity of Wasm.

Intermediate Representation (IR)​

OPA v0.37.0, released early 2022, brought two major enhancements to OPA: compiler strict mode and delta bundles. While those two features might have stolen the show of the release, the changelog additionally provides us with this:

The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

Interesting! Now, what does it mean? As alluded to in the last sentence, the low-level building blocks, or the evaluation plan, that eventually becomes Wasm, is now made available for consumption by other implementations. What would another implementation look like? That's up to you! While OPA may provide us with the low-level, step-by-step plan, for the evaluation of a query, it'll be on us to parse and evaluate that plan. Ever wanted to have your policy decisions served right inside of your Python app? Doable. Can't run the OPA server on your tiny microcontroller? You no longer need to. No runtime, no restrictions. What's the catch?

Bring Your Own OPA​

Using your programming language of choice to implement the full set of instructions included in the intermediate representation format isn't something you'd pull off in just a few hours. A robust implementation is likely going to necessitate quite some effort, and even if you decide to invest the days — or possibly, weeks — required for a greenfield implementation, implementing the IR instructions is only half of the story. OPA provides an impressive number of built-in functions, requiring corresponding implementations in the platform you choose to target. You probably won't need every single built-in to accommodate your use case though, so starting with the ones known to be relevant for you is likely a smart idea. Rego modules compiled to Wasm, on the other hand, ship with native implementations of many of the built-ins. If you plan to build an IR compiler or evaluator in C or C++, leveraging those would give you a head start!

Another aspect to consider is the management features provided by OPA. Similarly to Wasm, the scope of the IR format is limited to policy evaluation. Fetching bundles from remote endpoints, sending decision logs, or providing metrics and status reports is left as an exercise to the implementation. However, just like with the built-in functions, you likely won't need to support the full set of management capabilities shipped with OPA, but can pick and choose the parts that make sense to you. More interestingly, you're free to implement your own management features. Rather than pulling bundles from an S3 bucket, why not stream your permissions data from a Kafka topic? Or build a direct integration against that tool your organization uses for health checks, and so on. As laborious as a custom implementation may be, it opens up for some very interesting opportunities!

We're getting ahead of ourselves though. Before we dash off to write our own, next generation, OPA implementation in whatever the hottest programming language is these days, we should probably start by getting familiar with the IR format, and how to make sense of evaluation plans.

Making Plans​

Let's create a simple policy, and build a plan from that. The below policy contains two rules — is_admin to check if the "admin" role is included in the list of roles provided in the input for a user, and allow, which in this case simply is true if is_admin is true, but presumably would be extended to include more checks in future iterations of our policy.

package policy

import future.keywords.if

import future.keywords.in

allow if is_admin

is_admin if "admin" in input.user.roles

Simple enough, right? Let's see what a plan might look like! In order to build one, we'll use the aptly named opa build command. This command is used to build bundles, and the --target flag allows us to say that rather than just copying Rego and data files into the bundle, we want OPA to compile either a plan, or a wasm and put that in the bundle for us too. When building a plan, we'll additionally need to provide an entrypoint — this would be the path to either a package or a rule, from which the plan should be built. The path to the entry points (more than one is allowed) will later be used to query an implementation capable of parsing and evaluating our plan. Let's build a bundle with the plan target, and the entrypoint set to that of our allow rule:

opa build --target plan --entrypoint policy/allow .

This will create a bundle.tar.gz file in the current directory, with our plan inside of it. Since we're only interested in the plan for now, let's extract it from the bundle:

tar -zxvf bundle.tar.gz /plan.json

The plan.json file​

We now have a plan to work with! Let's see what's in that plan.json file. The first thing you'll notice is that the plan file contains three top level attributes — static, plans and funcs. The static object is fairly straightforward:

{
  "static": {
    "strings": [{ "value": "result" }, { "value": "user" }, { "value": "roles" }, { "value": "admin" }],
    "builtin_funcs": [
      {
        "name": "internal.member_2",
        "decl": {
          "args": [{ "type": "any" }, { "type": "any" }],
          "result": { "type": "boolean" },
          "type": "function"
        }
      }
    ],
    "files": [{ "value": "policy.rego" }]
  }
}

The strings array contains references to all strings included in the plan, and will be referenced whenever needed in evaluation. We'll recognize the "user", "roles" and "admin" strings from our policy, while the "result" string has been added by the plan builder, to be used as a key in the result set from plan evaluation. Thanks, plan builder! The builtin_funcs array provides a list of all the built-in functions used in our policy, along with the types expected for their arguments and return values. While "internal.member_2" might look unfamiliar, it's the internal name used for the built-in function representing the in operator used in our policy! Finally, the files array contains a list of all files used to build the plan, which in our case is only policy.rego.

The next attribute is the actual plans, and here's where things start to turn a bit cryptic. But don't worry, I'll walk you through it!

{
  "plans": {
    "plans": [
      {
        "name": "policy/allow",
        "blocks": [
          {
            "stmts": [
              {
                "type": "CallStmt",
                "stmt": {
                  "func": "g0.data.policy.allow",
                  "args": [{ "type": "local", "value": 0 }, { "type": "local", "value": 1 }],
                  "result": 2
                }
              },
              {
                "type": "AssignVarStmt",
                "stmt": {
                  "source": { "type": "local", "value": 2 },
                  "target": 3
                }
              },
              {
                "type": "MakeObjectStmt",
                "stmt": {
                  "target": 4
                }
              },
              {
                "type": "ObjectInsertStmt",
                "stmt": {
                  "key": { "type": "string_index", "value": 0 },
                  "value": { "type": "local", "value": 3 },
                  "object": 4
                }
              },
              {
                "type": "ResultSetAddStmt",
                "stmt": {
                  "value": 4
                }
              }
            ]
          }
        ]
      }
    ]
  }
}

Functions, statements, blocks​

For each entrypoint provided, we'll find a corresponding plan, which represents the planned evaluation path for that entrypoint. The name of the plan is our entrypoint ("policy/allow") and the blocks attribute contains the statements to be evaluated in order to "run" the plan. Quite literally in order too, as each statement block and statement will be evaluated in an entirely procedural fashion. Quite a contrast to the Rego code that produced it!

The first statement is a CallStmt, which means we'll need to evaluate the function (i.e. func) corresponding to the provided name in the funcs object — in our case this has been named "g0.data.policy.allow" (mapped from our allow rule) — and we'll take a closer look at the funcs part in a minute. The args provided to the function is "local" value 0, and "local" value 1. These represent the global input and data variables that you're likely familiar with from your Rego policies, and "value" in this case is rather the "name" — or pointer — to the values in the local scope. Who said naming was one of the hardest problems in computer science? Not the OPA plan compiler!

Where the Locals Go​

Since you'll see a lot of references to local throughout compiled plans, learning how it's used is imperative to understanding the steps involved in plan execution. When a function is invoked, like our "g0.data.policy.allow" above, a local object is created to represent the inputs to that function. The statements that comprise the function may in turn both read from the local object, as well as write to it, effectively making it a bearer of local state.

If a statement inside of a function involves calling another function, a new, "inner" local object will be created for the scope of that function, and the result of the function evaluation will be stored in the "outer" local, and so on.

Back to our CallStmt! We now know that it'll be invoked with the input (local 0) and data (local 1) as its arguments. The next attribute in the statement simply says "return": 2, meaning that whatever value is returned by the function should be stored in the next position — i.e. 2 — in the local state. Next up, the AssignVarStmt is used to assign the value at position 2 — that's our return value — to a new local value at position 3 (as defined by the "target" attribute). Moving on, we'll see a MakeObjectStmt used to create an empty object, which is placed at local value 4. In the next step, a key value pair is inserted into the object (ObjectInsertStmt) where the key is the "string_index" at position 0. Remember the strings attribute from the static object from before? This is it. The first item in that array is "result", so it looks like we're building a result set over here! The value associated with the result is good ol' local value 3, which we might recall was the result of invoking the "g0.data.policy.allow" function. Finally ResultSetAddStmt signals that we're done here. We have a result from the plan, and we're now ready to return it.

Following Procedure​

What about "g0.data.policy.allow" then? I promised we'd get back to the "funcs" object in a minute, and wow, time really flies when describing procedural instructions of an evaluation plan! A quick glance at the funcs object reveals that it contains not just the "allow" function, but also our "is_admin" function, here in the form of "g0.data.policy.is_admin". Since the allow function merely mirrors the result of is_admin, let's zoom in on the latter to learn how an implementation would evaluate the statements step by step, and how the state of the local object is updated in (almost) each step along the way. Rather than describing the steps in words, let's use a table for demonstration. In the left column you'll see a simplified version of each statement called, and in the right you'll see the local state after the statement has been applied. Note how each step procedurally builds up the final state, which is eventually returned to the caller. Beautiful, isn't it?

Evaluation plan statements and local state. As Medium does not have embeddable tables, you may find the spreadsheet from the image above in this Google Sheets document.

To learn more about the different statements an evaluator implementation may encounter, consult the OPA docs on the topic.

Planning Ahead​

Every aspect and instruction of the IR format would be too much for a single blog to cover, but the process of untangling an evaluation plan has hopefully been made clearer by now. While creating a full-fledged "OPA" native to your language or platform of choice might be a huge undertaking, even a basic implementation, with only a handful of built-in functions implemented, gets you surprisingly far towards something that actually feels usable. Open source implementations of course would have the benefit of others being able to contribute the parts that make the project usable to them. On the topic of open source implementations, are there any of those out there yet? In fact, there is!

Introducing Jarl​

For the past few months, fellow OPA maintainer Johan Fylling and I have spent some of our spare time hacking away on an IR implementation for the Java Virtual Machine (JVM) called Jarl. A jarl was a chieftain in the age of vikings, and given the viking theme of OPA itself, we figured it would be a good name for the project. And of course, it has that "J" in there too, which seems almost mandatory for JVM-based software.

We chose to use Clojure for our implementation, and while we are both rather novice Clojure coders, it's been a lot of fun to work with! Not only that, but using Clojure means getting access to the broader JVM ecosystem, both in terms of libraries available, and that applications written in Java, Kotlin or Scala will be able to use Jarl eventually. As an added bonus, Clojurescript allows the library to be compiled into Javascript as well, allowing us to target deployments in Node, or web browsers. Quite a versatile platform to build on!

While we still have a long way to go before Jarl is anywhere near production readiness, we're already at a point where it's usable for evaluation of many types of common policies. As we wanted to ensure conformance against OPA from the start, we ported the OPA compliance test suite to the IR format, which has proven to be tremendously useful for testing not just plan evaluation, but also the behavior of built-in function implementations. This code should be useful for anyone building their own implementation, so if that's you, make sure to check it out. As for the built-in functions, we currently have most of them ported, but some work remains to be done before we're able to have all tests from OPA pass. Then awaits the management features…

If the project sounds interesting to you, we'd love to hear from you! Reach out on the OPA Slack, or just try it out and report back on any issues, feature requests or ideas you might have. Or if you'd rather work on your own implementation, we'd love to help you get started.

Wrapping Up​

With the introduction of the intermediate representation format, another integration option for OPA has been made available. While it might be a bit of a niche — and certainly not the first choice to consider for most applications — it opens the door for many new and interesting use cases where an OPA integration using the standalone server model might not have been the best fit, or possible at all. Although perhaps not a well-known option until now, I hope this blog may contribute to changing that, and I'm looking forward to seeing how this alternative is leveraged in the future. Interesting times ahead!

August is here and we are feeling the heat! Reply back to this email and let the OPA team know how you're beating the heat this year!

Community Updates​

Office Hours and the Bi-weekly meeting have converged to 1 weekly meeting. Each Office Hours is an open format meeting you can use to ask any questions you'd like.

But now we are reserving the Office Hours Session following a new OPA release to showcase the new features from that release. Join the next session on August 9th to hear about the v0.43.0 release

Sign Up for Office Hours

You can also watch the replays from the OPA Office Hours and the Gatekeeper Weekly meetings on YouTube. Check out the release announcement for v0.42.0 below.

YouTube Videos 🎥​

Our community has posted lots of good stuff on YouTube this month, check out these cool videos.

Feature Release Videos​

Keywords, Contains and If​

How to incorporate the new keywords contains and if into your policies.

Watch Now

Builtin, Object.subset​

This new builtin allows you to check if a set, array, or object is a subset of another item.

Watch Now

Events 📆​

• Cloud Native Computing Meetup | August 25, 2022
• OPA Office Hours | August 9th, 2022

Blogs​

• Open Policy Agent (OPA) For Kubernetes
• High-Performance OPA
• How to Shape OPA Data for Policy Performance

Ecosystem​

Open Policy Agent v0.43.0​

• Large Object Performance Improvements
• GraphQL Tutorial

Gatekeeper v3.9.0​

• Constraint schema validation testing
• Make gatekeeper validate subresources

Conftest v0.34.0​

• Add parse_config and parse_config_file Rego functions to allow unit testing using config file snippets

Calling all OPA End-Users​

OPA Summit is officially scheduled, are you ready to share your OPA development journey? Let me us know how you're using OPA and we will help you craft a presentation. This event will be colocated at Kubecon in Detroit this October.

👉 Speak at OPA Summit 👈

Let us know how we did​

The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of. Reach out using one of the links below.

It's that time of year again! We have polled the Open Policy Agent (OPA) community to learn a bit more about what members are working on, their goals and how we can improve the project in the future. This year we had over 240 respondents, from various industries ranging from Software, Finance, E-commerce, Security, and more. With this new data set, we can learn if OPA usage has changed from the previous year, what features and tools are utilized the most and how to improve the OPA project as a whole for the community. To start, let's compare last year's survey results to this year's to see how things have changed or remained consistent:

Last Year's Survey

Year-over-year numbers​

Within a couple of percentage points, the number of use cases and respondents' implementation goals show similar results to last year. Of the users that responded that they have over four use cases, 70% of those reporting have used OPA for a year or longer. This shows us that as OPA usage matures in an organization, users gain confidence in adding additional use cases, helping them achieve their higher-level goals.

Almost 43% of respondents are in production with their OPA usage. This is a noticeable improvement from last year. With the addition of the Evaluating option, we can assume those users would have chosen experimentation given last year's choices, making the other two possibilities a few percentage points lower than the current year.

The last metric we highlighted in last year's survey is the time to production, showing that 40% of users reach production within six months. This year we are seeing about 27% of users in the production phase by this point. However, 50% of the users in this time frame are in a pre-production phase, which is a substantial amount.

Policy libraries​

We've seen a slight increase in usage for the Gatekeeper policy library from 57% to 62%, for the respondents that indicated they're using OPA for Kubernetes Admission Control. However, overall we are seeing 50% of respondents indicating that they are not using any external policy libraries. As policy libraries grow around specific use cases we can expect this number to increase.

Feedback​

Last year's request for better debugging tools led to creating two issues, rule-level tracing, and the print function. The Print Function was released in v0.34.0 and happily adopted by the community. Rule-level tracing still needs assistance from the community; perhaps you can help the community and submit a PR?

As we did with the previous year's survey, we asked the community for feedback to see what improvements would improve their OPA experience. The number one request was for more examples; nearly 33% of respondents asked for examples of specific or complex configurations and tutorials/sample data to go with them. About 12% of respondents asked for more integrations with AWS, such as the AWS CloudFormation integration that came out in June. And another 10% of users asked for additional debugging capabilities.

Learning tools​

The official OPA documentation is the most used resource by the community, with over 90% of respondents using it, followed by the Rego Playground at 66%. The OPA docs are consistently evolving and receive updates as new features roll out, but as with most open source projects, we need the community's help to keep the docs up to date. As for the Rego playground, we maintain this tool in the hopes that it helps users debug problems and collaborate on new policies. If you see any way that we can improve it, please let us know by creating a feature request.

Monitoring​

One surprising discovery from this year's responses is that 36% of users don't track OPA decisions, and 39% don't monitor their OPA status. While these metrics are accessible via OPA's management APIs, perhaps the docs can be spruced up with some new tutorials on configuring monitoring and logging!

Wrap up​

To sum it up, we saw consistency in the implementation goals and number of use cases for OPA with a slight uptick in the overall number of users in production. The utilization of policy libraries seems to have dropped to half of what it was last year. Debugging remains a high-priority area where users wish to see additional improvements, along with more examples and tutorials for the documentation. The OPA Docs and Rego playground take home the gold for most valuable resources, but they could use a few more examples to help community members configure monitoring and logging.

Thanks for your participation in this year's OPA User Survey. If you've sent us your mailing address, you can expect your t-shirt to arrive in your mailbox soon!

photo credit Kayla

Happy OPA 2022 Survey from Charlie!

Intro​

Hello everyone and welcome to the very first edition of the OPA Monthly Newsletter! We are excited to bring you all the happenings in the OPA ecosystem. You can expect to find a bit of everything in this newsletter, some community updates, a bit of contributor news, a handful of release notes, and any interesting content we've found on the internet this month.

Slack Updates​

Our Slack Org now hosts over 5,150 OPA community members!! The OPA team has been hard at work revamping the space to make it functional and valuable for all of our members. A little while ago you may remember we announced a Slack Reorg to consolidate and update channel names and descriptions. This effort was to give everyone a clear understanding of what's going on and where to go.

To continue to improve the Slack experience for our members we've added 2 new channels. For everyone interested in contributing to the OPA project you can now hang out in the #development channel to speak directly with other contributors and maintainers.

We've also added a #vendor channel to allow members to reach out directly to our rich ecosystem of vendors that are building products on top of OPA. Jump into the channel today and ask questions about how to improve your OPA management.

News Highlights​

One of our community members @boranx shared with the community that Conftest has made it into the Technology Radar by ThoughtWorks

GitHub Updates​

The OPA project wouldn't be the same without all of the contributions from the community. As such we would like to send a big thank you to all of the contributors from the v0.34 release.

• Edward Paget has contributed (#3826 SDK Feat) & (#3863 Bundles Fix)
• Kirk Patton a long time contributor added (#3773 Fix for exit statuses)
• GitHub User @0xAP first time contributor added (#3860 Bundles improvement)
• Andreas Brehmer first time contributor added (#3836 Fmt fix)
• Florian Gasc first time contributor added (#3879 Storage fix)
• Omolola Olamide has landed (#3910 Tutorial Updates)

Twitter Highlights​

For those not active on Twitter, we've collected some of the highlights and OPA shoutouts here:

https://twitter.com/that_tech_tea/status/1451930146835861504

https://twitter.com/nusairat/status/1458815340985520130

https://twitter.com/nmeisenzahl/status/1458419364433117184

Check out the slides and demos that Nico Meisenzahl created:

• enhance-your-compliance-and-governance-with-policy-based-cicd
• demo-opa-terraform-validation
• demo-opa-cicd-validation

Ecosystem Updates​

The OPA Project is always changing, check out the latest updates and features for OPA and some of the sub-projects.

OPA Release v0.35.0​

• Early Exit Optimization improves performance in many policy types
• New net.lookup_ip_addr built-in function to resolve host IP addresses
• Massive performance improvement in decision logging compression

OPA Release v0.34.0​

• A new in operator for checking membership and for iteration
• New print function for debugging
• New opa inspect command for quickly checking contents of a bundle

Gatekeeper Release v3.7.0​

• Mutation has graduated to Beta! 🎉
• Added ModifySet mutator 📐

Conftest Release v0.28.3​

• The OPA print function is now supported in Conftest!

Kube-mgmt Release v3.1.0​

• Support extra environment variables in opa and kube-mgmt containers

Community Spotlights​

• The one and only Developer-Guy has been working tirelessly to add OPA policy functionality to Cosign, Check out the PR to see the awesome work to connect the two projects.

What happened this month?​

• Meetup: OPA London Meetup
• Meetup: OPA Stockholm Meetup
• Talk: WTF is Cloud Native talk
• Talk: API Authorization with Open Policy Agent
• Blog: Connecting OPA with AWS Lambda
• Blog: Automated Manifest File Validation Using Open Policy Agent and GitHub Actions

What's coming up next month?​

A list of community meetings, meetups, and conferences.

OPA Bi-Weekly​

• Dec 7th at 10 AM PT
• Dec 21st at 10 AM PT

Gatekeeper Weekly​

• Dec 2nd, 2 PM PT
• Dec 8th, 9 AM PT
• Dec 15th, 2 PM PT
• Dec 22nd, 9 AM PT

Let us know how we did​

This was our very first edition of the OPA Newsletter, we really hope you enjoyed it! While we tried our best to find all the latest and greatest activities in the community we surely missed a lot as well. Want to share some cool content, have an OPA shoutout to make, want to speak at a conference, or host a meetup? Let us know by sending an email to: opa_newsletter@styra.com.

If you're new to OPA or to the community check out these community resources to get started.

• Chat with the community on Slack
• Ask for help and support on GitHub Discussions

One of the key takeaways from the Open Policy Agent 2021 Survey, was the need to improve the OPA debugging experience. Simply put, we need to make it easier to know what's going on when policies and rules are evaluated.

However, whenever someone talks about an "experience," it's rarely a small task and a checkbox to be checked once completed. Rather, it's all the little things that when combined provide a great improvement to the greater goal. If the OPA project used JIRA, it would probably be a safe bet to classify the "improve debugging experience" story as an "epic." With some improvements made, many new ideas and feature requests are likely to emerge along the way, and it would be rather optimistic to think that such a story ever got done, in the sense that no new improvements could be made.

To make things more complicated — and certainly more interesting — the OPA debugging experience isn't isolated to OPA itself. Improving the debugging experience for OPA entails not just looking at where things can be made better in OPA, but just as much in the tools commonly used when authoring Rego policies. These include tools like VS Code, IntelliJ IDEA and all the other editors commonly used for policy authoring.

So, where do we start?

Debugging with OPA eval​

Evaluating rules and variables has traditionally been done using the aptly named opa eval command. Commonly referred to as the "Swiss army knife of OPA," opa eval allows a policy author to quickly evaluate either standalone expressions like:

opa eval --format raw 1+1

Or, more commonly, with policy, data and input provided through command line arguments, and the path to the rule or variable of interest.

opa eval --data policy.rego --input input.json data.policy.main

While truly a versatile tool, debugging with opa eval has a couple of drawbacks. Having to create a new file to provide input might feel a little clunky, but hardly a terrible experience. But what if you want to evaluate the value of some variable inside of a rule, a test or a comprehension?

The trace built-in has to some extent been used for this purpose, but requires additional parameters passed to OPA to actually print something, and while occasionally useful, it was often perceived as somewhat clunky for the purpose of simply printing something.

Another problem frequently mentioned in the context of debugging is how sometimes opa eval, or the trace built-in comes back with just… nothing.

This brings us right into another topic often considered tricky with regards to debugging — OPA's handling of undefined. When OPA encounters undefined values, policy evaluation normally halts. Considering how Rego is a declarative query language, there isn't a whole lot more to do when a query comes back with nothing, just like a query language like SQL wouldn't have a whole lot more to do with an empty resultset. Since Rego rules are often compositions of many statements or other rules, it can sometimes be pretty difficult to tell where the undefined value that halted policy evaluation was introduced.

What to do?

Introducing the print built-in​

To tackle this, OPA v0.34.0 introduces a new print function to its ever growing list of built-ins. The print function does exactly what you'd expect it to do — prints any provided values to the console. Consider a rule like the one below.

allow {
    print("Entering allow")
    role := input.user.roles[_]
    print("Found role", role)
    role == "admin"
}

Running OPA eval would produce the following result.

$ opa eval -f raw -d policy.rego -i input.json 'data.policy.allow'
Entering allow
Found role developer
Found role sysadmin
Found role dba
Found role admin
true

The print function takes any number of arguments (static values, variables, input, data, etc) and prints each one (separated by whitespace) to the console.

While simple on the surface, a whole lot of thought has been put into its design, and unlike other built-in functions (which are often trivial to add into OPA) the print function required changes to the internal compiler. How come?

Varargs​

One of the design goals of the new print function was to allow a variable number of values or variables (i.e. varargs) to be passed as arguments, without resorting to the use of an array for the arguments, as is done by sprintf and other built-ins. Simply put we wanted something intuitive like:

print("x", input.x, "y", input.y)

To work just as expected. Sounds easy, right? Well, not really.

One of the more obscure (and hence, not encouraged) features of Rego can be traced back to its Datalog roots. Any built-in function can have it's return value expressed as the last argument to the function. Meaning that:

x := concat(".", ["a", "b", "c"])

Could alternatively be written as:

concat(".", ["a", "b", "c"], x)

With the last argument "reserved" for the return value, how would an implementation of varargs work? The answer was a new type of void function, where there simply is no return value to take into account. Since Rego functions are generally free from side effects, a void type of function hasn't really made sense previously, but with print having no purpose other than the desired side effect of printing to the console, adding a void type made sense.

Printing undefined​

The next problem to tackle in order for print to work nicely as a debugging tool was how to deal with undefined. Since we can expect print to be used to debug variables from input and data that might not be defined, it would be kind of a bummer if calling print itself halted policy evaluation! Ideally we'd be able to call the print function and have it print something even if some of the arguments provided pointed at undefined values. That way we could use the function to try and help also with the problem of identifying where in a policy undefined values have been introduced.

This requirement meant some internal assumptions of how Rego is parsed had to change, and the end result is a print function that prints undefined values as <undefined>, without halting policy evaluation.

allow {
    print(input.user.email, input.user.roles)
    input.user.roles[_] == "admin"
    endswith(input.user.email, "@acmecorp.com")
}

Evaluating the above allow rule with user.email missing from the input would now output something like this to the console:

$ opa eval -f raw -d policy.rego -i input.json data.policy.allow
<undefined> ["developer", "admin"]

Using print​

OPA supports many different modes of operation, from opa eval and opa test, to the OPA REPL and of course running as a standalone server. Both opa eval and the REPL will always print to the console (stderr, specifically) as expected. When running as a server, OPA will print any output from print function calls at the info log level. This makes print useful for debugging at the default info level or below. When configured to run with log level error (the generally recommended log level for production), OPA erases any calls to print from policies as they are loaded. Print calls left in the policy at that point will thus not impact performance whatsoever.

When running opa test, the print function by default will print to the console on test failures. Should you want to print output also for successful tests, the — verbose (short form -v) will do the trick. One case I've found particularly useful in tests is to use print in combination with the with … as mocking construct, to quickly see what exactly the result of a rule evaluation returns, like:

test_decison_allowed {
    result := decision with input as {
        "user": {
            "id": "abc123"
        },
        "request": {
            "method": "POST",
            "path": "/users"
        }
    }
    print(result)
    print(expectedResult)

    result == expectedResult
}

decision {
    [_, payload, _] := io.jwt.decode(input.user.token)
    print(payload)
    ...
}

While printing the outcome of rule evaluation in tests like this is valuable, the decision rule itself might be composed of multiple rules, and being able to add a few print lines in those to understand why our result isn't what we expect can help us quickly pinpoint the problem.

Wrapping up​

Rego as a policy language isn't a general purpose programming language, and shouldn't be treated as one. However, making the OPA and Rego debugging experience as smooth as possible means we sometimes might need to adapt concepts familiar from the programming languages policy authors normally work with. That the print function pull request added almost 2000 lines of code — half of them however from added test cases! — to the codebase is an interesting case study in how all the "little" details — like backwards compatibility, usability and performance — need to be considered when adding new functionality to a mature open source project like OPA.

I hope that you'll find the print function a useful addition to OPA, and a small improvement to the OPA/Rego debugging experience. Expect a lot more to come out in this space in future releases, and as always, make your voice heard in the OPA Slack if you have ideas, questions or feature requests you'd like to see incorporated into OPA!

Open Policy Agent (OPA) provides policy-based control for cloud native environments. It's commonly used alongside massive projects like Kubernetes and Envoy, and has dozens of other integrations and related projects in its ecosystem. Recent updates to the project aim to better integrate OPA with serverless architectures and other infrastructure with intermittent compute.

AWS Lambda is one such serverless solution. When a Lambda function is invoked, its execution environment only stays up until the function responds, at which point the runtime is frozen and all active processes/threads are paused. They are resumed once the next invocation is received, and the cycle repeats. OPA's plugin architecture wasn't designed to handle this freezing process, which resulted in unexpected behavior for things like bundle retrieval and log shipping.

Why is Manual Better than Automatic?​

However, thanks to a great new feature released in Open Policy Agent v0.32.0, plugins can now be triggered manually instead of automatically. Now, you may ask: "Wait, why is manual better than automatic?" Well, in many cases it's not, but for environments like AWS Lambda, it's not only useful, but critical to the stability and functionality of the OPA process.

First, let's briefly review the default behavior of plugins in OPA. Most of the standard plugins — Discovery, Bundles, and Decision Logs — operate by using a loop running in a goroutine that listens to various signals, one of which is a timer. When the timer delay elapses, the plugins will do stuff in the background, e.g. ship logs, check for new bundles to download, etc. OPA calls this a "periodic" trigger.

For most use cases, periodic triggers are exactly what we need. We don't really care precisely when a plugin is doing something in the background, just that it happens roughly within the interval period we've specified. The non-deterministic nature of periodic plugin triggers works well for a majority of applications. But what happens when that non-determinism becomes a problem? What if we need control over exactly when a plugin both starts and finishes a run of its primary task? Well, now we can get that control, thanks to the addition of Manual Trigger Support.

Introducing Manual Trigger support​

Plugins now support an optional "manual" trigger mode that can be set directly in a plugin's configuration. Additionally, if manual triggers are set on the Discovery plugin, all other plugins will inherit that setting. When a plugin's trigger is set to manual, the plugin's background loop will either pause until the Trigger func is called, or it will never start, depending on the plugin. The Trigger func will run a plugin's primary task and return only when the task is complete, or the provided context ends.

Now that we understand what manual triggers do, let's look at how we can use them in AWS Lambda. Lambda's on-demand compute is excellent for saving on compute costs, but it presents a challenge for code that has indeterminate start and end points, i.e., OPA plugins with periodic triggers. We don't want to interrupt the OPA process by freezing the execution environment when it's in the middle of downloading a bundle or shipping logs. And we don't want to wait around for a background loop to kick off those behaviors. Manual triggers give us the deterministic start and end points we need to properly coordinate OPA plugins in Lambda.

Try it for Yourself​

For those that are interested in running OPA in AWS Lambda, GoDaddy has recently open-sourced a small OPA plugin to make this easier, which uses manual triggers to operate OPA as a Lambda Extension. Lambda Extensions are deeply integrated into the lifecycle of a Lambda function, allowing you to perform background tasks without impacting the response time of your functions. OPA and Lambda Extensions are a great match, allowing you to plug in OPA's policy enforcement to your serverless infrastructure without complex installation or configuration management.

…happy OPA 2021 survey from Cal [credit: @eileen_kemp]

Last month we surveyed the OPA community to learn more about user adoption and help us plan and improve the project. We received over 300 responses from users across financial services, healthcare, public sector, automotive, cloud technology providers and more. This post highlights some of the survey results.

Use Cases and Adoption​

OPA adoption driven by authorization use cases across the stack​

Like last year, we used the survey to gauge use case adoption among respondents. We're interested in understanding where and why companies are deploying OPA because it helps us steer the project's long-term roadmap in the right direction. This year we asked respondents about the high-level goals they're trying to achieve by using OPA. We found that implementation of internal compliance and governance rules was the most common goal, however, nearly 60% of respondents indicated two or more goals being highly relevant.

In terms of use cases (e.g., Kubernetes admission control, Microservice authorization, etc.), the results were similar to the previous year with 50% of respondents indicating they use OPA for two or more use cases:

Kubernetes admission control continues to be the most common use case for OPA with 54% of respondents indicating they run OPA or OPA Gatekeeper to enforce various policies on their clusters:

From experiments to production in 6 months (or less)​

The survey showed the distribution of respondents OPA usage maturity was roughly equal:

What was more interesting was that about half of respondents indicated they had only been using OPA since January 2021. Of those users, nearly 40% had already reached production. Furthermore, the survey results show that most respondents reached production within 6 months. Beyond that, the percentage of users that are still in experimental stages drops to single digits:

These results are encouraging and also give us high-level metrics to improve on — ideally the time to production with OPA will continue to decrease as we improve the user experience and harden the project.

The survey results also highlighted a range of deployment sizes for production users. The following chart breaks down the deployment size responses by use case:

Policy library adoption is growing​

The survey asked users about various features in OPA and one of the most encouraging bits of information was that policy library adoption is growing within platform authorization use cases, like Kubernetes admission control and Terraform plan validation. Specifically, we found that nearly 60% of Kubernetes admission control users rely on the official gatekeeper-library policies that implement various best practices as well as PSP. We also found that nearly 30% of users that run OPA to validate Terraform plans rely on various open source policy libraries.

OPA Feedback​

In addition to gauging adoption we also used the survey to solicit feedback about the project.

Debugging needs some love​

After poring over the feedback comments, we found that the most common area for improvement is debugging. As with all surveys, some comments were non-specific, however multiple respondents requested better tracing modes and explanation presentation formats. Improved debug output support was another common request, and respondents also mentioned a desire for an interactive debugger similar to what you find in typical programming languages.

SDKs for various languages​

Aside from debugging, the next most common request was better SDK support for OPA in various languages. Several respondents indicated interest in Wasm-based SDKs for OPA, and others requested regular SDKs for Java, NodeJS and other languages. One of the reasons we haven't developed SDKs for OPA yet is because the OPA API is extremely simple (e.g., you can query OPA for decisions with a single HTTP POST request). However, with the Wasm compiler in OPA improving with every release, and the Wasm ecosystem growing rapidly, it feels like it's time to invest into language-specific integration libraries.

Wrap Up​

Thanks to everyone who completed the survey! The OPA t-shirts for completing the survey will be shipped soon. If you have not filled out the survey but would like to do so, you can still complete the OPA survey. As always, if you have questions or feedback, we're available on Slack, GitHub, etc.

Celebrating the growth of OPA community with a little cleanup​

The OPA community now has over 4,600 members in Slack! This is a tremendous milestone and we are so excited to have all of the new members join us. With this explosion of new members, the total number of Slack channels has crept up on us. The OPA team has noticed having so many channels has created confusion for our new and existing members on where to post about specific topics or themes. To make Slack easier to navigate we are rolling out a new set of channel names and descriptions. We hope this new structure will make Slack a bit easier for everyone. However, if you have a suggestion on how to make it better we would love to know.

Primary / Default Channels​

Any new members joining the OPA community will want to hang out here to start! Read about the latest announcements, introduce yourself in the community chit-chat and ask questions in the help channel.

#announcements​

Previous channel name: #general

The general channel is now the announcements channel. General was our busiest channel by far! It is now our announcements channel so that big news like releases and community events can stick around longer. New channels have been created for chit-chat and general help questions. The announcements channel is still open for anyone to post, but now think about posting things you want the entire community to know.

#chit-chat​

Previous channel name: #random

Everyone loves a good random channel and the OPA community is no different. Just because we changed the name you shouldn't feel the need to change what you're posting. Drop in your OPA memes and funny web links just like before. But now we also want to include community introductions and general chatter here as well.

#help​

Previous channel names:

• #rego
• #openpolicyagent
• #questions_and_answers
• #feasibility-question

We want to make it as easy as possible for you to find help while learning about OPA. So we've combined the channels we noticed new members were looking for help into a single channel. We hope that this will make it easier for everyone to know where to go when they need help and where to go when they feel like helping others.

Integrations​

For OPA users that have been around for a while you are probably well aware of the Conftest and Gatekeeper projects. These two OPA projects have gained a lot of traction and provide amazing contributions to the OPA community. If you have specific questions about the projects these channels are the best place to go. While the maintainers do hang around these channels, we love seeing community members showing off their OPA knowledge answering questions for each other.

#conftest​

Conftest is a must have in your policy toolkit. Write tests against structured configuration files including JSON, YAML, XML, Dockerfile, HCL, and more.

#gatekeeper​

Gatekeeper helps you safeguard your Kubernetes clusters by defining OPA-based admission control policies that are enforced via webhooks. Gatekeeper also helps you audit your Kubernetes clusters to detect policy violations.

Topics​

Trimmed down from the myriad of channels that existed before, the OPA team has chosen 3 channels that contained the most buzz from the community. Terraform, Envoy and WebAssembly are the 3 topics we noticed everyone likes to chat about. We hope that this buzz continues to grow. Also be on the lookout for programs in the future to be recognized as OPA experts in these areas.

#terraform​

Terraform lets you describe the infrastructure you want and automatically creates, deletes, and modifies your existing infrastructure to match. OPA makes it possible to write policies that test the changes Terraform is about to make before it makes them.

#envoy​

OPA-Envoy plugin extends OPA with a gRPC server that implements the Envoy External Authorization API. You can use this version of OPA to enforce fine-grained, context-aware access control policies with Envoy without modifying your microservices.

#wasm​

OPA is able to compile Rego policies into executable Wasm modules that can be evaluated with different inputs and external data. This is not running the OPA server in Wasm, nor is this just cross-compiled Golang code. The compiled Wasm module is a planned evaluation path for the source policy and query.

Archived Channels​

• #intros
• #bosun
• #feedback
• #registry
• #intellij-extension
• #gsoc19

You may notice some of the lesser used channels have been archived, we picked these channels based on a number of factors such as frequency of posts and average rate of responses. Ultimately we feel consolidating these conversations into the primary channel #chit-chat will increase participation and response rates.

Bot Channels​

• #bot-github
• #bot-rss

These are not the bots you're looking for…or maybe they are! Going forward any channels with the bot- prefix will be used for channels that include Slack bots or automated tools. Currently, the OPA team uses these channels to keep up to date with external sources like Stack Overflow, Reddit, and GitHub.

Wrapping Up​

Whether you're an OPA power user or looking to write your first Rego policy, we want the OPA Slack community to be your home for all things policy related. The OPA team realizes that sometimes new ideas need their own space to flourish. If you're interested in creating a new channel, reach out to @peteroneilljr or @tsandall on Slack and join us in our mission to solve policy enforcement across the stack. In addition to these Slack updates, you should also be on the look out for our new GitHub Discussions forum that will officially launch in the next couple of weeks. For a sneak peak check out the link at the bottom of this article!

Join the community on:

• Slack
• Twitter
• GitHub
• Discussions (Launching Soon!!)

"Community Spotlight" is a new series of blogs where we talk to people in and around the OPA community: users, integrators and contributors. Our first "spotlight" is Grant Shively, principal software engineer at GoDaddy.

Okay, so let's start with an introduction.

Sure! My name is Grant Shively and I've been working at GoDaddy for the last 13 years. For the last six years or so I've been a principal engineer on what we call our Care Platform, which is essentially our CRM system.

On the infrastructure side, we've moved from bare metal legacy servers with big monolithic ASP.NET web applications to a cloud-native, microservice architecture. Meanwhile we've transitioned much of the application platform towards Node.js and .NET Core. Recently, I've been doing a lot of work migrating things from our internal cloud to AWS.

For many years I've been involved in projects where we've needed complex authorization policies. And so we've built all these systems to deal with that, but there's never been a company-wide solution for that kind of thing.

I've been pushing for an authorization platform for the company for a few years, and last year it finally gained traction. We completed two company-wide initiatives with leaders from each of the major organizations where we all got together to talk about how we wanted to solve for authorization holistically at GoDaddy.

During the initiatives, we did a buy vs. build analysis where we evaluated different vendors, and open source projects. And that's when I discovered OPA. I watched the Netflix talk, which showed how they used OPA to solve many of the issues we are currently experiencing.

And so we did a proof of concept at the end of last year to prove the value and efficacy of the project using OPA, and we got buy-in on it, and now we're here, building out the final multi-tenant solution! We are hoping to have the next team running in production by the end of Q2, and we already have a number of teams lined up to onboard after that.

That was a great introduction! So, OPA adoption starting from a platform team, and growing from there?

I've definitely evangelized OPA within the company, since I'm a big fan of the project. There's someone from another team helping to contribute to this platform we're building so I guess that makes us two teams at the moment. My team is focused on the systems that support our customer service guides, while the other team is working on our domain control center. Initially, they're looking to implement this authorization platform for some of the high risk scenarios we have around domains, such as transfer of ownership and things like that.

You said you were working mainly with .NET and Node.js?

Yep. As a company we work in a number of languages, including Python, Go, Node.js, .NET Core, and Java… I think those are most of the blessed languages, but there are probably some other languages used for one-offs as well.

My team, we were traditionally C#, and then we brought in Node.js during our transition to microservices. With the arrival of .NET Core, we were comfortable with continuing to support C# as well. Personally, after working with both languages for a number of years, I prefer Node.js for implementing our small, single-purpose APIs.

You've made quite a few contributions to OPA, which is written in Go. Did you have experience working with that before?

No, I hadn't touched it before! Go is definitely approachable. And I think the OPA code base is very clean and logical to explore. You don't have to look too much to figure out where something's at, once you get used to it.

Go feels like a language that minimizes syntactic sugar. There's really only a couple of "right" ways to do things, I feel. Sometimes it's frustrating too, like with the lack of map/filter/reduce operations. Coming from more functional paradigms it feels frustrating at times.

Agreed. In this process of finding OPA, what other alternatives did you consider? Building your own? Some commercial options?

We looked at Athenz. One of our senior architects had worked on that project and it looked interesting. It did some of the stuff we wanted, but not all. And then we looked at a whole bunch of products from various vendors. We briefly thought about building our own until we ran into OPA. OPA completely negated any reason for us to try and build our own because it did exactly what we needed, and in such an elegant way that it would be very difficult for us to replicate.

Compared to the other options you considered, what was the main appeal of OPA? What was it that won you over?

One of the things we needed was the ability for multiple engineering teams to work with policy, in a self-service manner, while at the same time leveraging globally-managed policy and data, like authentication token expiration rules and identity attributes.

Like everybody else, we have a few home-grown systems, and we definitely needed to be able to integrate with those. Also, we have an extremely distributed architecture. Hundreds of AWS accounts, and whatever solution we found was going to need to run in pretty much all of them.

I knew that there were going to be integration points that would be very difficult for most vendor products. One thing we really didn't want was some sort of centralized authorization API for the whole company. We've been bitten by such architectures in the past. Our apps are highly distributed, and we need our authorization decision points to be highly distributed too, while still being able to manage and distribute policy from a central location. And a lot of the bigger vendors had these products that just felt a little too heavy for large-scale decentralization, you know?.

Right.

And we really liked Rego. We've talked about this before, but a lot of the vendors are based on XACML, which is just really tedious to work with. Compared to that, Rego was a breath of fresh air. Another factor we considered was the buy-in we saw from some of the bigger companies like Netflix, etc. Rapid adoption of a project is usually a great sign.

I'm sure I'll have more opinions on Rego once we try and onboard more teams onto our platform. Certain teams at GoDaddy have more operational experience than others, and we'd really like to build a simplified UI policy builder on top of our platform and Rego. That way, they can just fill out some simple input boxes, hit enter, and it publishes a policy for them. That kind of thing.

Interesting!

Yeah, I'm both looking and not looking forward to tackling that problem, haha!

Maybe you don't remember anymore, but did you have any such moments where you got stuck while learning Rego? I know one thing that felt a little odd to me when starting out was how Rego handled undefined values. How rule evaluation just stops when they are encountered.

Some of the syntax around iteration was a little confusing at first. I think it's one of those things that makes total sense once you understand it, but it wasn't like anything I'd worked with before.

The biggest issue though — and we still kind of have it—is understanding how a decision was made, and how to surface that in logs. We have a couple of open issues about that, and we have been running OPA internally with a few patches we want to upstream that help address some of this.

It's really important to our security and engineering teams that we can look at a somewhat human readable, but hopefully not super verbose, way of saying that "this authorization request was approved or denied because of these reasons".

Another related issue would be how to best propagate obligations up through layers of policy. Some obligations can come from a really low level, like policy checking the claims of a JWT, and if some property doesn't exist, then we want to propagate an obligation up from that lower level rule. Solving some of those problems has felt a little clunky.

On the other side, what did you really like?

I really liked that you could create custom built-in functions and other things to extend OPAs functionality. That has been really useful to us as we've built things around our requirements, like custom key signing, custom decision logging, and so on.

There have been a few cases where we needed to extend portions of the OPA code, only to find that they were hardcoded around a particular implementation, without interfaces for us to leverage. I've been very pleased with the responsiveness of and guidance from the OPA maintainers. We've been able to contribute a number of small changes to make the OPA code more flexible for our extensions.

On the topic of flexibility, one of the things that really appealed to me was the number of integrations available. Even if you're never going to use all of them, it really shows what the general purpose nature of OPA enables.

Yeah, that brings to mind another consideration. Netflix talked about how they used OPA for authorization across their entire stack—infrastructure, forward facing code, back end. When I saw OPA, I knew our platform could grow to be much more than just, you know, a simple authorization engine—there are many other policy-based scenarios that don't necessarily have to do with authorization. So that may be a far future thing, but that was also very enticing about OPA.

I think some of the flexibility really pays off at scale. Having one unified language to describe policy across the stack, having one place to go for decision logs, and so on.

Yep. We're definitely looking at increasing the number of integrations. So many of the systems deployed today work only with Active Directory, and AD group-based authorization, so it'll be interesting to see how we can integrate with systems like that.

About integrations. Are there any of the existing ones in the ecosystem you are using or are planning to use?

I'm only familiar with the Envoy one. Oh, and Kubernetes. What else is there? We're just starting to use Envoy, so I think there will likely be some places where we use that integration. And, we'll probably want to look into a Kubernetes integration too at some point, if AWS EKS supports that.

Looking at the list now… Wow, there's really a bunch of them here! I'll need to look into some of these. What we're really doing a lot of is AWS integrations. Kind of in the same vein as the Envoy integration, extending OPA and all that, you know? We're doing the same thing with OPA and AWS, and I hope we'll be able to eventually open source that.

Things like key signing using the AWS Key Management Service (KMS) or shipping decision logs directly to AWS Kinesis without having to go through HTTP endpoints in between.

I know you've mentioned AWS Lambda functions too.

Yeah. A lot of our APIs are lambda-based, and we'll obviously want to use this platform we're building for that too. We're currently trying to figure out how to make OPA work as well for on-demand serverless functions as it does for traditional compute environments.

Definitely a hot topic! I've seen some questions on that on the OPA Slack as well. What are the challenges there?

So, the way OPA currently works is that many of the internal "plugins", like the client downloading bundles, or the one uploading decision logs-they all work on a time-based loop. So you configure them to upload or download or to do whatever they are meant to do at a certain interval, like every thirty seconds or something.

This doesn't really work for serverless functions though as they don't have continuous compute available; functions freeze after serving a response, so nothing can run in the background. What we'd need in this context is rather something based on other types of triggers. We're waiting for a feature called AWS Lambda Extensions, which should reach general availability in May. They should make it possible to use OPA really efficiently even in a lambda context.

Anything you have found missing from OPA or would like to see on the roadmap?

I mentioned it before, but yeah, the big thing would be figuring out how to do the tracing in a more efficient way. I have come to understand that the way "full" tracing is done today is apparently pretty expensive. We just need something like rule level tracing, which has been discussed a bit in GitHub issues in the past. With that in OPA proper we wouldn't have a reason to run our own modifications at all, except for plugins.

Awesome! Finally, what are your future plans for OPA at GoDaddy?

What we're starting out with is authorization for internal systems. Once we nail that and it's working well we're looking to expand it to authorization for customer systems, and at some point possibly our infrastructure too.

With what OPA can do, I feel like the sky's the limit in terms of what you want to throw into it. We just have to make sure we fully support the platform we're building as we continue growing. I have high hopes for it!

IBM Research & Styra

What's happened?​

In a previous Medium blog, a feature released in OPA v0.27.0 was introduced that lets OPA's static type checker take JSON schemas for input documents into account, improving how errors from misused data are caught during policy authoring.

The article explains that opa eval can take a schema for the input document via the --schema(-s) flag, applied globally across the module, allowing OPA's checker to catch issues like undefined objects.

What's new?​

This section covers extending type checking to support multiple JSON schema files for both input and data documents.

Example Rego code (based on a Kubernetes admission review input) is shown with a typo bug:

package kubernetes.admission

deny[msg] {
  input.request.kind.kinds == "Pod"                       # This line has a typo, should be input.request.kind.kind
  image := input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

input.request.kind.kinds

which should be:

input.request.kind.kind

Running:

% opa eval --format pretty -i admission-input.json -d policy.rego -s schemas/admission-input.json

returns:

1 error occurred: policy.rego:4: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^
                     have: "kinds"
                     want (one of): ["kind" "version"]

The article notes that a similar typo in input.request.object.spec.containers[_].image would go undetected, since the admission review schema leaves input.request.object generically typed.

To solve this, opa eval now supports a directory of schema files via the same --schema (-s) flag, while still supporting single schema files. New Rego Metadata blocks allow specifying schema annotations and scope, improving bug detection for undefined fields.

An override feature is also introduced, described as letting users merge existing schemas and subschemas "for more precise type checking."

Additionally, schema loading is enabled for opa eval — bundle, supporting type checking of data documents across a bundle — useful for "batch type analysis of Rego policies as part of any CI/CD pipelines."

These features are available in OPA v0.28.0.

What's the big deal you say?​

Example policies and schemas are available in the opa-schema-examples repository.

The Kubernetes Admission Review example is revisited to demonstrate annotations and schema overriding. The object field in an Admission Review can contain any Kubernetes resource, and its schema leaves that field generically typed.

Annotations associate a Rego expression with an input or data schema loaded via opa eval -s, within a given scope.

Annotations use METADATA comment blocks in YAML syntax, where "every line in the block must start at Column 1."

Example schema directory structure:

mySchemasDir/
├── input.json
└── kubernetes
└──────pod.json

Loading the schema directory can be done via:

% opa eval data.kubernetes.admission --format pretty -i opa-schema-examples/kubernetes/input.json -d opa-schema-examples/kubernetes/policy.rego -s opa-schema-examples/kubernetes/mySchemasDir

% opa eval data.kubernetes.admission -format pretty -i opa-schema-examples/kubernetes/input.json -b opa-schema-examples/bundle.tar.gz -s opa-schema-examples/kubernetes/mySchemasDir

In this example, input is associated with the Admission Review schema (input.json), and input.request.object is set to the Kubernetes Pod schema (pod.json), with the second annotation overriding the first. The order of annotations is stated to matter "for overriding to work correctly."

Notes on schema reference syntax:

• Relative paths inside mySchemasDir are used, omitting the .json suffix
• The global variable schema represents the top level of the directory
• schema.input is valid; schema.pod-schema is invalid due to the hyphen — the correct syntax is schema["pod-schema"]

Combining annotations and overriding allows catching type errors in input.request.object.spec.containers[_].image:

package kubernetes.admission

# METADATA
# scope: rule
# schemas:
#   - input: schema["input"]
#   - input.request.object: schema.kubernetes["pod"]
deny[msg] {
  input.request.kind.kinds == "Pod"                       # This line has a typo, should be input.request.kind.kind
  image := input.request.object.spec.containers[_].images # This line has a typo, should be input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

2 errors occurred:
policy.rego:9: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^ have: "kinds"
                     want (one of): ["kind" "version"]
policy.rego:10: rego_type_error: undefined ref: input.request.object.spec.containers[_].images
  input.request.object.spec.containers[_].images
                                          ^ have: "images"
                                          want (one of): ["args" "command" "env" "envFrom" "image" "imagePullPolicy" "lifecycle" "livenessProbe" "name" "ports" "readinessProbe" "resources" "securityContext" "stdin" "stdinOnce" "terminationMessagePath" "terminationMessagePolicy" "tty" "volumeDevices" "volumeMounts" "workingDir"]

A second example checks whether an operation is allowed for a user, given an ACL data document. In the first allow rule, input uses input.json and data.acl uses acl-schema.json; an invalid expression like data.acl.typo would trigger a type error.

package policy

import data.acl

default allow = false

# METADATA
# scope: rule
# schemas:
#   - input: schema["input"]
#   - data.acl: schema["acl-schema"]
allow {
        access = data.acl["alice"]
        access[_] == input.operation
}

allow {
        access = data.acl["bob"]
        access[_] == input.operation
}

The article clarifies that this annotation "does not constrain other paths under data" — only the type of data.acl is statically known.

The second allow rule in the same example has no schema annotations, so it isn't type-checked against any loaded schema. Different rules in the same module can use different input schemas.

Annotations can also apply at different scopes via the scope field in Metadata, defaulting to the following statement if omitted. Supported scope values:

• rule - applies to the individual rule statement
• document - applies to all of the rules with the same name in the same package
• package - applies to all of the rules in the package
• subpackages - applies to all of the rules in the package and all subpackages (recursively)

More details: Annotation scopes documentation

What's Next?​

The article closes by noting future plans to extend schema support to additional JSON Schema features such as additionalProperties, with more updates promised in upcoming OPA releases.

Further reading: OPA schemas documentation

Links​

• Documentation: OPA schemas documentation
• Examples: opa-schema-examples repository
• JSON to JSON schema online tool: jsonschema.net
• JSON schema reference: Understanding JSON Schema reference
• Related blog: /blog/type-checking-your-rego-policies-with-json-schema-in-opa-5f7ac4c8a958

IBM Research & Styra

The Open Policy Agent (OPA) is an open-source engine that unifies policy enforcement across the cloud native stack. It provides a query language called Rego that lets the user specify policy as code and an engine that evaluates the queries given input data.

Rego is a powerful declarative language that does not require the user to specify a query strategy. This is achieved by the OPA runtime, leaving users free to reason about policies at a higher level.

Rego has a gradual type system meaning that types can be partially known statically. For example, an object could have certain fields whose types are known and others that are unknown statically. OPA type checks what it knows statically and leaves the unknown parts to be type checked at runtime. However, the input handed to OPA could by design be any JSON value and hence gradual type-checking has no way of catching policy authoring mistakes, even when the policy author knows the intended schema.

In this article, we introduce a new feature that enhances OPA's ability to statically type check Rego code by taking into account schemas for input documents. This improves programmer productivity and helps Rego programmers catch errors earlier.

To achieve this, we enable opa eval to take the schema for the input document, specified in JSON Schema format. The input schema is passed with the flag --schema (-s). Armed with this information, OPA's enhanced type checker can detect bugs stemming from incorrect usage of the input.

This feature is now available in OPA v0.27.0. Let's check it out!

Rego type checking demo

Consider the following Rego code, which assumes as input a Kubernetes admission review. For resources that are Pods, it checks that the image name starts with a specific prefix.

package kubernetes.admission


deny[msg] {
  input.request.kind.kinds == "Pod"
  image := input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

Notice that this code has a typo in it: input.request.kind.kinds is undefined and should have been input.request.kind.kind.

Consider the following input document:

{
  "kind": "AdmissionReview",
  "request": {
    "kind": {
      "kind": "Pod",
      "version": "v1"
    },
    "object": {
      "metadata": {
        "name": "myapp"
      },
      "spec": {
        "containers": [
          { "image": "nginx", "name": "nginx-frontend" },
          { "image": "mysql", "name": "mysql-backend" }
        ]
      }
    }
  }
}

% opa eval -format pretty -i admission-review.json -d pod.rego
[]

The empty value returned is indistinguishable from a situation where the input did not violate the policy. This error is therefore causing the policy not to catch violating inputs appropriately.

If we fix the Rego code and change input.request.kind.kinds to input.request.kind.kind, then we obtain the expected result:

[
  "image 'nginx' comes from untrusted registry"
  "image 'mysql' comes from untrusted registry"
]

With this feature, it is possible to pass a schema to opa eval, written in JSON Schema.

Consider the Kubernetes admission review input schema. We can pass this schema to the evaluator as follows:

% opa eval -format pretty -i admission-review.json -d pod.rego -s admission-schema.json

With the erroneous Rego code, we now obtain the following type error:

1 error occurred: pod.rego:5: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^
                     have: "kinds"
                     want (one of): ["kind" "version"]

This indicates the error to the Rego developer right away, without having the need to observe the results of runs on actual data, thereby improving productivity.

With this new feature, Rego developers will be able to provide JSON Schemas for their input documents and get the most out of static type checking. If you don't have a JSON Schema handy, you can easily obtain one from a sample JSON file using online tools (see links below). In the future, we will add to this feature the support to allow users to specify a directory of schemas and assign schemas to data documents, as well. This will be done via annotations (global, rule-level, rule output) that will indicate what schema is associated with what Rego path expression.

Happy Rego type checking!

For more information and limitations, see documentation and examples.