August Edition!

August is here and we are feeling the heat! Reply back to this email and let the OPA team know how you're beating the heat this year!

Community Updates​

Office Hours and the Bi-weekly meeting have converged to 1 weekly meeting. Each Office Hours is an open format meeting you can use to ask any questions you'd like.

But now we are reserving the Office Hours Session following a new OPA release to showcase the new features from that release. Join the next session on August 9th to hear about the v0.43.0 release

Sign Up for Office Hours

You can also watch the replays from the OPA Office Hours and the Gatekeeper Weekly meetings on YouTube. Check out the release announcement for v0.42.0 below.

YouTube Videos 🎥​

Our community has posted lots of good stuff on YouTube this month, check out these cool videos.

Feature Release Videos​

Keywords, Contains and If​

How to incorporate the new keywords contains and if into your policies.

Watch Now

Builtin, Object.subset​

This new builtin allows you to check if a set, array, or object is a subset of another item.

Watch Now

Events 📆​

• Cloud Native Computing Meetup | August 25, 2022
• OPA Office Hours | August 9th, 2022

Blogs​

• Open Policy Agent (OPA) For Kubernetes
• High-Performance OPA
• How to Shape OPA Data for Policy Performance

Ecosystem​

Open Policy Agent v0.43.0​

• Large Object Performance Improvements
• GraphQL Tutorial

Gatekeeper v3.9.0​

• Constraint schema validation testing
• Make gatekeeper validate subresources

Conftest v0.34.0​

• Add parse_config and parse_config_file Rego functions to allow unit testing using config file snippets

Calling all OPA End-Users​

OPA Summit is officially scheduled, are you ready to share your OPA development journey? Let me us know how you're using OPA and we will help you craft a presentation. This event will be colocated at Kubecon in Detroit this October.

👉 Speak at OPA Summit 👈

Let us know how we did​

The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of. Reach out using one of the links below.

It's that time of year again! We have polled the Open Policy Agent (OPA) community to learn a bit more about what members are working on, their goals and how we can improve the project in the future. This year we had over 240 respondents, from various industries ranging from Software, Finance, E-commerce, Security, and more. With this new data set, we can learn if OPA usage has changed from the previous year, what features and tools are utilized the most and how to improve the OPA project as a whole for the community. To start, let's compare last year's survey results to this year's to see how things have changed or remained consistent:

Last Year's Survey

Year-over-year numbers​

Within a couple of percentage points, the number of use cases and respondents' implementation goals show similar results to last year. Of the users that responded that they have over four use cases, 70% of those reporting have used OPA for a year or longer. This shows us that as OPA usage matures in an organization, users gain confidence in adding additional use cases, helping them achieve their higher-level goals.

Almost 43% of respondents are in production with their OPA usage. This is a noticeable improvement from last year. With the addition of the Evaluating option, we can assume those users would have chosen experimentation given last year's choices, making the other two possibilities a few percentage points lower than the current year.

The last metric we highlighted in last year's survey is the time to production, showing that 40% of users reach production within six months. This year we are seeing about 27% of users in the production phase by this point. However, 50% of the users in this time frame are in a pre-production phase, which is a substantial amount.

Policy libraries​

We've seen a slight increase in usage for the Gatekeeper policy library from 57% to 62%, for the respondents that indicated they're using OPA for Kubernetes Admission Control. However, overall we are seeing 50% of respondents indicating that they are not using any external policy libraries. As policy libraries grow around specific use cases we can expect this number to increase.

Feedback​

Last year's request for better debugging tools led to creating two issues, rule-level tracing, and the print function. The Print Function was released in v0.34.0 and happily adopted by the community. Rule-level tracing still needs assistance from the community; perhaps you can help the community and submit a PR?

As we did with the previous year's survey, we asked the community for feedback to see what improvements would improve their OPA experience. The number one request was for more examples; nearly 33% of respondents asked for examples of specific or complex configurations and tutorials/sample data to go with them. About 12% of respondents asked for more integrations with AWS, such as the AWS CloudFormation integration that came out in June. And another 10% of users asked for additional debugging capabilities.

Learning tools​

The official OPA documentation is the most used resource by the community, with over 90% of respondents using it, followed by the Rego Playground at 66%. The OPA docs are consistently evolving and receive updates as new features roll out, but as with most open source projects, we need the community's help to keep the docs up to date. As for the Rego playground, we maintain this tool in the hopes that it helps users debug problems and collaborate on new policies. If you see any way that we can improve it, please let us know by creating a feature request.

Monitoring​

One surprising discovery from this year's responses is that 36% of users don't track OPA decisions, and 39% don't monitor their OPA status. While these metrics are accessible via OPA's management APIs, perhaps the docs can be spruced up with some new tutorials on configuring monitoring and logging!

Wrap up​

To sum it up, we saw consistency in the implementation goals and number of use cases for OPA with a slight uptick in the overall number of users in production. The utilization of policy libraries seems to have dropped to half of what it was last year. Debugging remains a high-priority area where users wish to see additional improvements, along with more examples and tutorials for the documentation. The OPA Docs and Rego playground take home the gold for most valuable resources, but they could use a few more examples to help community members configure monitoring and logging.

Thanks for your participation in this year's OPA User Survey. If you've sent us your mailing address, you can expect your t-shirt to arrive in your mailbox soon!

photo credit Kayla

Happy OPA 2022 Survey from Charlie!

November Edition

Intro​

Hello everyone and welcome to the very first edition of the OPA Monthly Newsletter! We are excited to bring you all the happenings in the OPA ecosystem. You can expect to find a bit of everything in this newsletter, some community updates, a bit of contributor news, a handful of release notes, and any interesting content we've found on the internet this month.

Slack Updates​

Our Slack Org now hosts over 5,150 OPA community members!! The OPA team has been hard at work revamping the space to make it functional and valuable for all of our members. A little while ago you may remember we announced a Slack Reorg to consolidate and update channel names and descriptions. This effort was to give everyone a clear understanding of what's going on and where to go.

To continue to improve the Slack experience for our members we've added 2 new channels. For everyone interested in contributing to the OPA project you can now hang out in the #development channel to speak directly with other contributors and maintainers.

We've also added a #vendor channel to allow members to reach out directly to our rich ecosystem of vendors that are building products on top of OPA. Jump into the channel today and ask questions about how to improve your OPA management.

News Highlights​

One of our community members @boranx shared with the community that Conftest has made it into the Technology Radar by ThoughtWorks

GitHub Updates​

The OPA project wouldn't be the same without all of the contributions from the community. As such we would like to send a big thank you to all of the contributors from the v0.34 release.

• Edward Paget has contributed (#3826 SDK Feat) & (#3863 Bundles Fix)
• Kirk Patton a long time contributor added (#3773 Fix for exit statuses)
• GitHub User @0xAP first time contributor added (#3860 Bundles improvement)
• Andreas Brehmer first time contributor added (#3836 Fmt fix)
• Florian Gasc first time contributor added (#3879 Storage fix)
• Omolola Olamide has landed (#3910 Tutorial Updates)

Twitter Highlights​

For those not active on Twitter, we've collected some of the highlights and OPA shoutouts here:

https://twitter.com/that_tech_tea/status/1451930146835861504

https://twitter.com/nusairat/status/1458815340985520130

https://twitter.com/nmeisenzahl/status/1458419364433117184

Check out the slides and demos that Nico Meisenzahl created:

• enhance-your-compliance-and-governance-with-policy-based-cicd
• demo-opa-terraform-validation
• demo-opa-cicd-validation

Ecosystem Updates​

The OPA Project is always changing, check out the latest updates and features for OPA and some of the sub-projects.

OPA Release v0.35.0​

• Early Exit Optimization improves performance in many policy types
• New net.lookup_ip_addr built-in function to resolve host IP addresses
• Massive performance improvement in decision logging compression

OPA Release v0.34.0​

• A new in operator for checking membership and for iteration
• New print function for debugging
• New opa inspect command for quickly checking contents of a bundle

Gatekeeper Release v3.7.0​

• Mutation has graduated to Beta! 🎉
• Added ModifySet mutator 📐

Conftest Release v0.28.3​

• The OPA print function is now supported in Conftest!

Kube-mgmt Release v3.1.0​

• Support extra environment variables in opa and kube-mgmt containers

Community Spotlights​

• The one and only Developer-Guy has been working tirelessly to add OPA policy functionality to Cosign, Check out the PR to see the awesome work to connect the two projects.

What happened this month?​

• Meetup: OPA London Meetup
• Meetup: OPA Stockholm Meetup
• Talk: WTF is Cloud Native talk
• Talk: API Authorization with Open Policy Agent
• Blog: Connecting OPA with AWS Lambda
• Blog: Automated Manifest File Validation Using Open Policy Agent and GitHub Actions

What's coming up next month?​

A list of community meetings, meetups, and conferences.

OPA Bi-Weekly​

• Dec 7th at 10 AM PT
• Dec 21st at 10 AM PT

Gatekeeper Weekly​

• Dec 2nd, 2 PM PT
• Dec 8th, 9 AM PT
• Dec 15th, 2 PM PT
• Dec 22nd, 9 AM PT

Let us know how we did​

This was our very first edition of the OPA Newsletter, we really hope you enjoyed it! While we tried our best to find all the latest and greatest activities in the community we surely missed a lot as well. Want to share some cool content, have an OPA shoutout to make, want to speak at a conference, or host a meetup? Let us know by sending an email to: opa_newsletter@styra.com.

If you're new to OPA or to the community check out these community resources to get started.

• Chat with the community on Slack
• Ask for help and support on GitHub Discussions

One of the key takeaways from the Open Policy Agent 2021 Survey, was the need to improve the OPA debugging experience. Simply put, we need to make it easier to know what's going on when policies and rules are evaluated.

However, whenever someone talks about an "experience," it's rarely a small task and a checkbox to be checked once completed. Rather, it's all the little things that when combined provide a great improvement to the greater goal. If the OPA project used JIRA, it would probably be a safe bet to classify the "improve debugging experience" story as an "epic." With some improvements made, many new ideas and feature requests are likely to emerge along the way, and it would be rather optimistic to think that such a story ever got done, in the sense that no new improvements could be made.

To make things more complicated — and certainly more interesting — the OPA debugging experience isn't isolated to OPA itself. Improving the debugging experience for OPA entails not just looking at where things can be made better in OPA, but just as much in the tools commonly used when authoring Rego policies. These include tools like VS Code, IntelliJ IDEA and all the other editors commonly used for policy authoring.

So, where do we start?

Debugging with OPA eval​

Evaluating rules and variables has traditionally been done using the aptly named opa eval command. Commonly referred to as the "Swiss army knife of OPA," opa eval allows a policy author to quickly evaluate either standalone expressions like:

opa eval --format raw 1+1

Or, more commonly, with policy, data and input provided through command line arguments, and the path to the rule or variable of interest.

opa eval --data policy.rego --input input.json data.policy.main

While truly a versatile tool, debugging with opa eval has a couple of drawbacks. Having to create a new file to provide input might feel a little clunky, but hardly a terrible experience. But what if you want to evaluate the value of some variable inside of a rule, a test or a comprehension?

The trace built-in has to some extent been used for this purpose, but requires additional parameters passed to OPA to actually print something, and while occasionally useful, it was often perceived as somewhat clunky for the purpose of simply printing something.

Another problem frequently mentioned in the context of debugging is how sometimes opa eval, or the trace built-in comes back with just… nothing.

This brings us right into another topic often considered tricky with regards to debugging — OPA's handling of undefined. When OPA encounters undefined values, policy evaluation normally halts. Considering how Rego is a declarative query language, there isn't a whole lot more to do when a query comes back with nothing, just like a query language like SQL wouldn't have a whole lot more to do with an empty resultset. Since Rego rules are often compositions of many statements or other rules, it can sometimes be pretty difficult to tell where the undefined value that halted policy evaluation was introduced.

What to do?

Introducing the print built-in​

To tackle this, OPA v0.34.0 introduces a new print function to its ever growing list of built-ins. The print function does exactly what you'd expect it to do — prints any provided values to the console. Consider a rule like the one below.

allow {
    print("Entering allow")
    role := input.user.roles[_]
    print("Found role", role)
    role == "admin"
}

Running OPA eval would produce the following result.

$ opa eval -f raw -d policy.rego -i input.json 'data.policy.allow'
Entering allow
Found role developer
Found role sysadmin
Found role dba
Found role admin
true

The print function takes any number of arguments (static values, variables, input, data, etc) and prints each one (separated by whitespace) to the console.

While simple on the surface, a whole lot of thought has been put into its design, and unlike other built-in functions (which are often trivial to add into OPA) the print function required changes to the internal compiler. How come?

Varargs​

One of the design goals of the new print function was to allow a variable number of values or variables (i.e. varargs) to be passed as arguments, without resorting to the use of an array for the arguments, as is done by sprintf and other built-ins. Simply put we wanted something intuitive like:

print("x", input.x, "y", input.y)

To work just as expected. Sounds easy, right? Well, not really.

One of the more obscure (and hence, not encouraged) features of Rego can be traced back to its Datalog roots. Any built-in function can have it's return value expressed as the last argument to the function. Meaning that:

x := concat(".", ["a", "b", "c"])

Could alternatively be written as:

concat(".", ["a", "b", "c"], x)

With the last argument "reserved" for the return value, how would an implementation of varargs work? The answer was a new type of void function, where there simply is no return value to take into account. Since Rego functions are generally free from side effects, a void type of function hasn't really made sense previously, but with print having no purpose other than the desired side effect of printing to the console, adding a void type made sense.

Printing undefined​

The next problem to tackle in order for print to work nicely as a debugging tool was how to deal with undefined. Since we can expect print to be used to debug variables from input and data that might not be defined, it would be kind of a bummer if calling print itself halted policy evaluation! Ideally we'd be able to call the print function and have it print something even if some of the arguments provided pointed at undefined values. That way we could use the function to try and help also with the problem of identifying where in a policy undefined values have been introduced.

This requirement meant some internal assumptions of how Rego is parsed had to change, and the end result is a print function that prints undefined values as <undefined>, without halting policy evaluation.

allow {
    print(input.user.email, input.user.roles)
    input.user.roles[_] == "admin"
    endswith(input.user.email, "@acmecorp.com")
}

Evaluating the above allow rule with user.email missing from the input would now output something like this to the console:

$ opa eval -f raw -d policy.rego -i input.json data.policy.allow
<undefined> ["developer", "admin"]

Using print​

OPA supports many different modes of operation, from opa eval and opa test, to the OPA REPL and of course running as a standalone server. Both opa eval and the REPL will always print to the console (stderr, specifically) as expected. When running as a server, OPA will print any output from print function calls at the info log level. This makes print useful for debugging at the default info level or below. When configured to run with log level error (the generally recommended log level for production), OPA erases any calls to print from policies as they are loaded. Print calls left in the policy at that point will thus not impact performance whatsoever.

When running opa test, the print function by default will print to the console on test failures. Should you want to print output also for successful tests, the — verbose (short form -v) will do the trick. One case I've found particularly useful in tests is to use print in combination with the with … as mocking construct, to quickly see what exactly the result of a rule evaluation returns, like:

test_decison_allowed {
    result := decision with input as {
        "user": {
            "id": "abc123"
        },
        "request": {
            "method": "POST",
            "path": "/users"
        }
    }
    print(result)
    print(expectedResult)

    result == expectedResult
}

decision {
    [_, payload, _] := io.jwt.decode(input.user.token)
    print(payload)
    ...
}

While printing the outcome of rule evaluation in tests like this is valuable, the decision rule itself might be composed of multiple rules, and being able to add a few print lines in those to understand why our result isn't what we expect can help us quickly pinpoint the problem.

Wrapping up​

Rego as a policy language isn't a general purpose programming language, and shouldn't be treated as one. However, making the OPA and Rego debugging experience as smooth as possible means we sometimes might need to adapt concepts familiar from the programming languages policy authors normally work with. That the print function pull request added almost 2000 lines of code — half of them however from added test cases! — to the codebase is an interesting case study in how all the "little" details — like backwards compatibility, usability and performance — need to be considered when adding new functionality to a mature open source project like OPA.

I hope that you'll find the print function a useful addition to OPA, and a small improvement to the OPA/Rego debugging experience. Expect a lot more to come out in this space in future releases, and as always, make your voice heard in the OPA Slack if you have ideas, questions or feature requests you'd like to see incorporated into OPA!

Open Policy Agent (OPA) provides policy-based control for cloud native environments. It's commonly used alongside massive projects like Kubernetes and Envoy, and has dozens of other integrations and related projects in its ecosystem. Recent updates to the project aim to better integrate OPA with serverless architectures and other infrastructure with intermittent compute.

AWS Lambda is one such serverless solution. When a Lambda function is invoked, its execution environment only stays up until the function responds, at which point the runtime is frozen and all active processes/threads are paused. They are resumed once the next invocation is received, and the cycle repeats. OPA's plugin architecture wasn't designed to handle this freezing process, which resulted in unexpected behavior for things like bundle retrieval and log shipping.

Why is Manual Better than Automatic?​

However, thanks to a great new feature released in Open Policy Agent v0.32.0, plugins can now be triggered manually instead of automatically. Now, you may ask: "Wait, why is manual better than automatic?" Well, in many cases it's not, but for environments like AWS Lambda, it's not only useful, but critical to the stability and functionality of the OPA process.

First, let's briefly review the default behavior of plugins in OPA. Most of the standard plugins — Discovery, Bundles, and Decision Logs — operate by using a loop running in a goroutine that listens to various signals, one of which is a timer. When the timer delay elapses, the plugins will do stuff in the background, e.g. ship logs, check for new bundles to download, etc. OPA calls this a "periodic" trigger.

For most use cases, periodic triggers are exactly what we need. We don't really care precisely when a plugin is doing something in the background, just that it happens roughly within the interval period we've specified. The non-deterministic nature of periodic plugin triggers works well for a majority of applications. But what happens when that non-determinism becomes a problem? What if we need control over exactly when a plugin both starts and finishes a run of its primary task? Well, now we can get that control, thanks to the addition of Manual Trigger Support.

Introducing Manual Trigger support​

Plugins now support an optional "manual" trigger mode that can be set directly in a plugin's configuration. Additionally, if manual triggers are set on the Discovery plugin, all other plugins will inherit that setting. When a plugin's trigger is set to manual, the plugin's background loop will either pause until the Trigger func is called, or it will never start, depending on the plugin. The Trigger func will run a plugin's primary task and return only when the task is complete, or the provided context ends.

Now that we understand what manual triggers do, let's look at how we can use them in AWS Lambda. Lambda's on-demand compute is excellent for saving on compute costs, but it presents a challenge for code that has indeterminate start and end points, i.e., OPA plugins with periodic triggers. We don't want to interrupt the OPA process by freezing the execution environment when it's in the middle of downloading a bundle or shipping logs. And we don't want to wait around for a background loop to kick off those behaviors. Manual triggers give us the deterministic start and end points we need to properly coordinate OPA plugins in Lambda.

Try it for Yourself​

For those that are interested in running OPA in AWS Lambda, GoDaddy has recently open-sourced a small OPA plugin to make this easier, which uses manual triggers to operate OPA as a Lambda Extension. Lambda Extensions are deeply integrated into the lifecycle of a Lambda function, allowing you to perform background tasks without impacting the response time of your functions. OPA and Lambda Extensions are a great match, allowing you to plug in OPA's policy enforcement to your serverless infrastructure without complex installation or configuration management.

…happy OPA 2021 survey from Cal [credit: @eileen_kemp]

Last month we surveyed the OPA community to learn more about user adoption and help us plan and improve the project. We received over 300 responses from users across financial services, healthcare, public sector, automotive, cloud technology providers and more. This post highlights some of the survey results.

Use Cases and Adoption​

OPA adoption driven by authorization use cases across the stack​

Like last year, we used the survey to gauge use case adoption among respondents. We're interested in understanding where and why companies are deploying OPA because it helps us steer the project's long-term roadmap in the right direction. This year we asked respondents about the high-level goals they're trying to achieve by using OPA. We found that implementation of internal compliance and governance rules was the most common goal, however, nearly 60% of respondents indicated two or more goals being highly relevant.

In terms of use cases (e.g., Kubernetes admission control, Microservice authorization, etc.), the results were similar to the previous year with 50% of respondents indicating they use OPA for two or more use cases:

Kubernetes admission control continues to be the most common use case for OPA with 54% of respondents indicating they run OPA or OPA Gatekeeper to enforce various policies on their clusters:

From experiments to production in 6 months (or less)​

The survey showed the distribution of respondents OPA usage maturity was roughly equal:

What was more interesting was that about half of respondents indicated they had only been using OPA since January 2021. Of those users, nearly 40% had already reached production. Furthermore, the survey results show that most respondents reached production within 6 months. Beyond that, the percentage of users that are still in experimental stages drops to single digits:

These results are encouraging and also give us high-level metrics to improve on — ideally the time to production with OPA will continue to decrease as we improve the user experience and harden the project.

The survey results also highlighted a range of deployment sizes for production users. The following chart breaks down the deployment size responses by use case:

Policy library adoption is growing​

The survey asked users about various features in OPA and one of the most encouraging bits of information was that policy library adoption is growing within platform authorization use cases, like Kubernetes admission control and Terraform plan validation. Specifically, we found that nearly 60% of Kubernetes admission control users rely on the official gatekeeper-library policies that implement various best practices as well as PSP. We also found that nearly 30% of users that run OPA to validate Terraform plans rely on various open source policy libraries.

OPA Feedback​

In addition to gauging adoption we also used the survey to solicit feedback about the project.

Debugging needs some love​

After poring over the feedback comments, we found that the most common area for improvement is debugging. As with all surveys, some comments were non-specific, however multiple respondents requested better tracing modes and explanation presentation formats. Improved debug output support was another common request, and respondents also mentioned a desire for an interactive debugger similar to what you find in typical programming languages.

SDKs for various languages​

Aside from debugging, the next most common request was better SDK support for OPA in various languages. Several respondents indicated interest in Wasm-based SDKs for OPA, and others requested regular SDKs for Java, NodeJS and other languages. One of the reasons we haven't developed SDKs for OPA yet is because the OPA API is extremely simple (e.g., you can query OPA for decisions with a single HTTP POST request). However, with the Wasm compiler in OPA improving with every release, and the Wasm ecosystem growing rapidly, it feels like it's time to invest into language-specific integration libraries.

Wrap Up​

Thanks to everyone who completed the survey! The OPA t-shirts for completing the survey will be shipped soon. If you have not filled out the survey but would like to do so, you can still complete the OPA survey. As always, if you have questions or feedback, we're available on Slack, GitHub, etc.

Celebrating the growth of OPA community with a little cleanup​

The OPA community now has over 4,600 members in Slack! This is a tremendous milestone and we are so excited to have all of the new members join us. With this explosion of new members, the total number of Slack channels has crept up on us. The OPA team has noticed having so many channels has created confusion for our new and existing members on where to post about specific topics or themes. To make Slack easier to navigate we are rolling out a new set of channel names and descriptions. We hope this new structure will make Slack a bit easier for everyone. However, if you have a suggestion on how to make it better we would love to know.

Primary / Default Channels​

Any new members joining the OPA community will want to hang out here to start! Read about the latest announcements, introduce yourself in the community chit-chat and ask questions in the help channel.

#announcements​

Previous channel name: #general

The general channel is now the announcements channel. General was our busiest channel by far! It is now our announcements channel so that big news like releases and community events can stick around longer. New channels have been created for chit-chat and general help questions. The announcements channel is still open for anyone to post, but now think about posting things you want the entire community to know.

#chit-chat​

Previous channel name: #random

Everyone loves a good random channel and the OPA community is no different. Just because we changed the name you shouldn't feel the need to change what you're posting. Drop in your OPA memes and funny web links just like before. But now we also want to include community introductions and general chatter here as well.

#help​

Previous channel names:

• #rego
• #openpolicyagent
• #questions_and_answers
• #feasibility-question

We want to make it as easy as possible for you to find help while learning about OPA. So we've combined the channels we noticed new members were looking for help into a single channel. We hope that this will make it easier for everyone to know where to go when they need help and where to go when they feel like helping others.

Integrations​

For OPA users that have been around for a while you are probably well aware of the Conftest and Gatekeeper projects. These two OPA projects have gained a lot of traction and provide amazing contributions to the OPA community. If you have specific questions about the projects these channels are the best place to go. While the maintainers do hang around these channels, we love seeing community members showing off their OPA knowledge answering questions for each other.

#conftest​

Conftest is a must have in your policy toolkit. Write tests against structured configuration files including JSON, YAML, XML, Dockerfile, HCL, and more.

#gatekeeper​

Gatekeeper helps you safeguard your Kubernetes clusters by defining OPA-based admission control policies that are enforced via webhooks. Gatekeeper also helps you audit your Kubernetes clusters to detect policy violations.

Topics​

Trimmed down from the myriad of channels that existed before, the OPA team has chosen 3 channels that contained the most buzz from the community. Terraform, Envoy and WebAssembly are the 3 topics we noticed everyone likes to chat about. We hope that this buzz continues to grow. Also be on the lookout for programs in the future to be recognized as OPA experts in these areas.

#terraform​

Terraform lets you describe the infrastructure you want and automatically creates, deletes, and modifies your existing infrastructure to match. OPA makes it possible to write policies that test the changes Terraform is about to make before it makes them.

#envoy​

OPA-Envoy plugin extends OPA with a gRPC server that implements the Envoy External Authorization API. You can use this version of OPA to enforce fine-grained, context-aware access control policies with Envoy without modifying your microservices.

#wasm​

OPA is able to compile Rego policies into executable Wasm modules that can be evaluated with different inputs and external data. This is not running the OPA server in Wasm, nor is this just cross-compiled Golang code. The compiled Wasm module is a planned evaluation path for the source policy and query.

Archived Channels​

• #intros
• #bosun
• #feedback
• #registry
• #intellij-extension
• #gsoc19

You may notice some of the lesser used channels have been archived, we picked these channels based on a number of factors such as frequency of posts and average rate of responses. Ultimately we feel consolidating these conversations into the primary channel #chit-chat will increase participation and response rates.

Bot Channels​

• #bot-github
• #bot-rss

These are not the bots you're looking for…or maybe they are! Going forward any channels with the bot- prefix will be used for channels that include Slack bots or automated tools. Currently, the OPA team uses these channels to keep up to date with external sources like Stack Overflow, Reddit, and GitHub.

Wrapping Up​

Whether you're an OPA power user or looking to write your first Rego policy, we want the OPA Slack community to be your home for all things policy related. The OPA team realizes that sometimes new ideas need their own space to flourish. If you're interested in creating a new channel, reach out to @peteroneilljr or @tsandall on Slack and join us in our mission to solve policy enforcement across the stack. In addition to these Slack updates, you should also be on the look out for our new GitHub Discussions forum that will officially launch in the next couple of weeks. For a sneak peak check out the link at the bottom of this article!

Join the community on:

• Slack
• Twitter
• GitHub
• Discussions (Launching Soon!!)

"Community Spotlight" is a new series of blogs where we talk to people in and around the OPA community: users, integrators and contributors. Our first "spotlight" is Grant Shively, principal software engineer at GoDaddy.

Okay, so let's start with an introduction.

Sure! My name is Grant Shively and I've been working at GoDaddy for the last 13 years. For the last six years or so I've been a principal engineer on what we call our Care Platform, which is essentially our CRM system.

On the infrastructure side, we've moved from bare metal legacy servers with big monolithic ASP.NET web applications to a cloud-native, microservice architecture. Meanwhile we've transitioned much of the application platform towards Node.js and .NET Core. Recently, I've been doing a lot of work migrating things from our internal cloud to AWS.

For many years I've been involved in projects where we've needed complex authorization policies. And so we've built all these systems to deal with that, but there's never been a company-wide solution for that kind of thing.

I've been pushing for an authorization platform for the company for a few years, and last year it finally gained traction. We completed two company-wide initiatives with leaders from each of the major organizations where we all got together to talk about how we wanted to solve for authorization holistically at GoDaddy.

During the initiatives, we did a buy vs. build analysis where we evaluated different vendors, and open source projects. And that's when I discovered OPA. I watched the Netflix talk, which showed how they used OPA to solve many of the issues we are currently experiencing.

And so we did a proof of concept at the end of last year to prove the value and efficacy of the project using OPA, and we got buy-in on it, and now we're here, building out the final multi-tenant solution! We are hoping to have the next team running in production by the end of Q2, and we already have a number of teams lined up to onboard after that.

That was a great introduction! So, OPA adoption starting from a platform team, and growing from there?

I've definitely evangelized OPA within the company, since I'm a big fan of the project. There's someone from another team helping to contribute to this platform we're building so I guess that makes us two teams at the moment. My team is focused on the systems that support our customer service guides, while the other team is working on our domain control center. Initially, they're looking to implement this authorization platform for some of the high risk scenarios we have around domains, such as transfer of ownership and things like that.

You said you were working mainly with .NET and Node.js?

Yep. As a company we work in a number of languages, including Python, Go, Node.js, .NET Core, and Java… I think those are most of the blessed languages, but there are probably some other languages used for one-offs as well.

My team, we were traditionally C#, and then we brought in Node.js during our transition to microservices. With the arrival of .NET Core, we were comfortable with continuing to support C# as well. Personally, after working with both languages for a number of years, I prefer Node.js for implementing our small, single-purpose APIs.

You've made quite a few contributions to OPA, which is written in Go. Did you have experience working with that before?

No, I hadn't touched it before! Go is definitely approachable. And I think the OPA code base is very clean and logical to explore. You don't have to look too much to figure out where something's at, once you get used to it.

Go feels like a language that minimizes syntactic sugar. There's really only a couple of "right" ways to do things, I feel. Sometimes it's frustrating too, like with the lack of map/filter/reduce operations. Coming from more functional paradigms it feels frustrating at times.

Agreed. In this process of finding OPA, what other alternatives did you consider? Building your own? Some commercial options?

We looked at Athenz. One of our senior architects had worked on that project and it looked interesting. It did some of the stuff we wanted, but not all. And then we looked at a whole bunch of products from various vendors. We briefly thought about building our own until we ran into OPA. OPA completely negated any reason for us to try and build our own because it did exactly what we needed, and in such an elegant way that it would be very difficult for us to replicate.

Compared to the other options you considered, what was the main appeal of OPA? What was it that won you over?

One of the things we needed was the ability for multiple engineering teams to work with policy, in a self-service manner, while at the same time leveraging globally-managed policy and data, like authentication token expiration rules and identity attributes.

Like everybody else, we have a few home-grown systems, and we definitely needed to be able to integrate with those. Also, we have an extremely distributed architecture. Hundreds of AWS accounts, and whatever solution we found was going to need to run in pretty much all of them.

I knew that there were going to be integration points that would be very difficult for most vendor products. One thing we really didn't want was some sort of centralized authorization API for the whole company. We've been bitten by such architectures in the past. Our apps are highly distributed, and we need our authorization decision points to be highly distributed too, while still being able to manage and distribute policy from a central location. And a lot of the bigger vendors had these products that just felt a little too heavy for large-scale decentralization, you know?.

Right.

And we really liked Rego. We've talked about this before, but a lot of the vendors are based on XACML, which is just really tedious to work with. Compared to that, Rego was a breath of fresh air. Another factor we considered was the buy-in we saw from some of the bigger companies like Netflix, etc. Rapid adoption of a project is usually a great sign.

I'm sure I'll have more opinions on Rego once we try and onboard more teams onto our platform. Certain teams at GoDaddy have more operational experience than others, and we'd really like to build a simplified UI policy builder on top of our platform and Rego. That way, they can just fill out some simple input boxes, hit enter, and it publishes a policy for them. That kind of thing.

Interesting!

Yeah, I'm both looking and not looking forward to tackling that problem, haha!

Maybe you don't remember anymore, but did you have any such moments where you got stuck while learning Rego? I know one thing that felt a little odd to me when starting out was how Rego handled undefined values. How rule evaluation just stops when they are encountered.

Some of the syntax around iteration was a little confusing at first. I think it's one of those things that makes total sense once you understand it, but it wasn't like anything I'd worked with before.

The biggest issue though — and we still kind of have it—is understanding how a decision was made, and how to surface that in logs. We have a couple of open issues about that, and we have been running OPA internally with a few patches we want to upstream that help address some of this.

It's really important to our security and engineering teams that we can look at a somewhat human readable, but hopefully not super verbose, way of saying that "this authorization request was approved or denied because of these reasons".

Another related issue would be how to best propagate obligations up through layers of policy. Some obligations can come from a really low level, like policy checking the claims of a JWT, and if some property doesn't exist, then we want to propagate an obligation up from that lower level rule. Solving some of those problems has felt a little clunky.

On the other side, what did you really like?

I really liked that you could create custom built-in functions and other things to extend OPAs functionality. That has been really useful to us as we've built things around our requirements, like custom key signing, custom decision logging, and so on.

There have been a few cases where we needed to extend portions of the OPA code, only to find that they were hardcoded around a particular implementation, without interfaces for us to leverage. I've been very pleased with the responsiveness of and guidance from the OPA maintainers. We've been able to contribute a number of small changes to make the OPA code more flexible for our extensions.

On the topic of flexibility, one of the things that really appealed to me was the number of integrations available. Even if you're never going to use all of them, it really shows what the general purpose nature of OPA enables.

Yeah, that brings to mind another consideration. Netflix talked about how they used OPA for authorization across their entire stack—infrastructure, forward facing code, back end. When I saw OPA, I knew our platform could grow to be much more than just, you know, a simple authorization engine—there are many other policy-based scenarios that don't necessarily have to do with authorization. So that may be a far future thing, but that was also very enticing about OPA.

I think some of the flexibility really pays off at scale. Having one unified language to describe policy across the stack, having one place to go for decision logs, and so on.

Yep. We're definitely looking at increasing the number of integrations. So many of the systems deployed today work only with Active Directory, and AD group-based authorization, so it'll be interesting to see how we can integrate with systems like that.

About integrations. Are there any of the existing ones in the ecosystem you are using or are planning to use?

I'm only familiar with the Envoy one. Oh, and Kubernetes. What else is there? We're just starting to use Envoy, so I think there will likely be some places where we use that integration. And, we'll probably want to look into a Kubernetes integration too at some point, if AWS EKS supports that.

Looking at the list now… Wow, there's really a bunch of them here! I'll need to look into some of these. What we're really doing a lot of is AWS integrations. Kind of in the same vein as the Envoy integration, extending OPA and all that, you know? We're doing the same thing with OPA and AWS, and I hope we'll be able to eventually open source that.

Things like key signing using the AWS Key Management Service (KMS) or shipping decision logs directly to AWS Kinesis without having to go through HTTP endpoints in between.

I know you've mentioned AWS Lambda functions too.

Yeah. A lot of our APIs are lambda-based, and we'll obviously want to use this platform we're building for that too. We're currently trying to figure out how to make OPA work as well for on-demand serverless functions as it does for traditional compute environments.

Definitely a hot topic! I've seen some questions on that on the OPA Slack as well. What are the challenges there?

So, the way OPA currently works is that many of the internal "plugins", like the client downloading bundles, or the one uploading decision logs-they all work on a time-based loop. So you configure them to upload or download or to do whatever they are meant to do at a certain interval, like every thirty seconds or something.

This doesn't really work for serverless functions though as they don't have continuous compute available; functions freeze after serving a response, so nothing can run in the background. What we'd need in this context is rather something based on other types of triggers. We're waiting for a feature called AWS Lambda Extensions, which should reach general availability in May. They should make it possible to use OPA really efficiently even in a lambda context.

Anything you have found missing from OPA or would like to see on the roadmap?

I mentioned it before, but yeah, the big thing would be figuring out how to do the tracing in a more efficient way. I have come to understand that the way "full" tracing is done today is apparently pretty expensive. We just need something like rule level tracing, which has been discussed a bit in GitHub issues in the past. With that in OPA proper we wouldn't have a reason to run our own modifications at all, except for plugins.

Awesome! Finally, what are your future plans for OPA at GoDaddy?

What we're starting out with is authorization for internal systems. Once we nail that and it's working well we're looking to expand it to authorization for customer systems, and at some point possibly our infrastructure too.

With what OPA can do, I feel like the sky's the limit in terms of what you want to throw into it. We just have to make sure we fully support the platform we're building as we continue growing. I have high hopes for it!

IBM Research & Styra

What's happened?​

In a previous Medium blog, a feature released in OPA v0.27.0 was introduced that lets OPA's static type checker take JSON schemas for input documents into account, improving how errors from misused data are caught during policy authoring.

The article explains that opa eval can take a schema for the input document via the --schema(-s) flag, applied globally across the module, allowing OPA's checker to catch issues like undefined objects.

What's new?​

This section covers extending type checking to support multiple JSON schema files for both input and data documents.

Example Rego code (based on a Kubernetes admission review input) is shown with a typo bug:

package kubernetes.admission

deny[msg] {
  input.request.kind.kinds == "Pod"                       # This line has a typo, should be input.request.kind.kind
  image := input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

input.request.kind.kinds

which should be:

input.request.kind.kind

Running:

% opa eval --format pretty -i admission-input.json -d policy.rego -s schemas/admission-input.json

returns:

1 error occurred: policy.rego:4: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^
                     have: "kinds"
                     want (one of): ["kind" "version"]

The article notes that a similar typo in input.request.object.spec.containers[_].image would go undetected, since the admission review schema leaves input.request.object generically typed.

To solve this, opa eval now supports a directory of schema files via the same --schema (-s) flag, while still supporting single schema files. New Rego Metadata blocks allow specifying schema annotations and scope, improving bug detection for undefined fields.

An override feature is also introduced, described as letting users merge existing schemas and subschemas "for more precise type checking."

Additionally, schema loading is enabled for opa eval — bundle, supporting type checking of data documents across a bundle — useful for "batch type analysis of Rego policies as part of any CI/CD pipelines."

These features are available in OPA v0.28.0.

What's the big deal you say?​

Example policies and schemas are available in the opa-schema-examples repository.

The Kubernetes Admission Review example is revisited to demonstrate annotations and schema overriding. The object field in an Admission Review can contain any Kubernetes resource, and its schema leaves that field generically typed.

Annotations associate a Rego expression with an input or data schema loaded via opa eval -s, within a given scope.

Annotations use METADATA comment blocks in YAML syntax, where "every line in the block must start at Column 1."

Example schema directory structure:

mySchemasDir/
├── input.json
└── kubernetes
└──────pod.json

Loading the schema directory can be done via:

% opa eval data.kubernetes.admission --format pretty -i opa-schema-examples/kubernetes/input.json -d opa-schema-examples/kubernetes/policy.rego -s opa-schema-examples/kubernetes/mySchemasDir

% opa eval data.kubernetes.admission -format pretty -i opa-schema-examples/kubernetes/input.json -b opa-schema-examples/bundle.tar.gz -s opa-schema-examples/kubernetes/mySchemasDir

In this example, input is associated with the Admission Review schema (input.json), and input.request.object is set to the Kubernetes Pod schema (pod.json), with the second annotation overriding the first. The order of annotations is stated to matter "for overriding to work correctly."

Notes on schema reference syntax:

• Relative paths inside mySchemasDir are used, omitting the .json suffix
• The global variable schema represents the top level of the directory
• schema.input is valid; schema.pod-schema is invalid due to the hyphen — the correct syntax is schema["pod-schema"]

Combining annotations and overriding allows catching type errors in input.request.object.spec.containers[_].image:

package kubernetes.admission

# METADATA
# scope: rule
# schemas:
#   - input: schema["input"]
#   - input.request.object: schema.kubernetes["pod"]
deny[msg] {
  input.request.kind.kinds == "Pod"                       # This line has a typo, should be input.request.kind.kind
  image := input.request.object.spec.containers[_].images # This line has a typo, should be input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

2 errors occurred:
policy.rego:9: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^ have: "kinds"
                     want (one of): ["kind" "version"]
policy.rego:10: rego_type_error: undefined ref: input.request.object.spec.containers[_].images
  input.request.object.spec.containers[_].images
                                          ^ have: "images"
                                          want (one of): ["args" "command" "env" "envFrom" "image" "imagePullPolicy" "lifecycle" "livenessProbe" "name" "ports" "readinessProbe" "resources" "securityContext" "stdin" "stdinOnce" "terminationMessagePath" "terminationMessagePolicy" "tty" "volumeDevices" "volumeMounts" "workingDir"]

A second example checks whether an operation is allowed for a user, given an ACL data document. In the first allow rule, input uses input.json and data.acl uses acl-schema.json; an invalid expression like data.acl.typo would trigger a type error.

package policy

import data.acl

default allow = false

# METADATA
# scope: rule
# schemas:
#   - input: schema["input"]
#   - data.acl: schema["acl-schema"]
allow {
        access = data.acl["alice"]
        access[_] == input.operation
}

allow {
        access = data.acl["bob"]
        access[_] == input.operation
}

The article clarifies that this annotation "does not constrain other paths under data" — only the type of data.acl is statically known.

The second allow rule in the same example has no schema annotations, so it isn't type-checked against any loaded schema. Different rules in the same module can use different input schemas.

Annotations can also apply at different scopes via the scope field in Metadata, defaulting to the following statement if omitted. Supported scope values:

• rule - applies to the individual rule statement
• document - applies to all of the rules with the same name in the same package
• package - applies to all of the rules in the package
• subpackages - applies to all of the rules in the package and all subpackages (recursively)

More details: Annotation scopes documentation

What's Next?​

The article closes by noting future plans to extend schema support to additional JSON Schema features such as additionalProperties, with more updates promised in upcoming OPA releases.

Further reading: OPA schemas documentation

Links​

• Documentation: OPA schemas documentation
• Examples: opa-schema-examples repository
• JSON to JSON schema online tool: jsonschema.net
• JSON schema reference: Understanding JSON Schema reference
• Related blog: /blog/type-checking-your-rego-policies-with-json-schema-in-opa-5f7ac4c8a958

IBM Research & Styra

The Open Policy Agent (OPA) is an open-source engine that unifies policy enforcement across the cloud native stack. It provides a query language called Rego that lets the user specify policy as code and an engine that evaluates the queries given input data.

Rego is a powerful declarative language that does not require the user to specify a query strategy. This is achieved by the OPA runtime, leaving users free to reason about policies at a higher level.

Rego has a gradual type system meaning that types can be partially known statically. For example, an object could have certain fields whose types are known and others that are unknown statically. OPA type checks what it knows statically and leaves the unknown parts to be type checked at runtime. However, the input handed to OPA could by design be any JSON value and hence gradual type-checking has no way of catching policy authoring mistakes, even when the policy author knows the intended schema.

In this article, we introduce a new feature that enhances OPA's ability to statically type check Rego code by taking into account schemas for input documents. This improves programmer productivity and helps Rego programmers catch errors earlier.

To achieve this, we enable opa eval to take the schema for the input document, specified in JSON Schema format. The input schema is passed with the flag --schema (-s). Armed with this information, OPA's enhanced type checker can detect bugs stemming from incorrect usage of the input.

This feature is now available in OPA v0.27.0. Let's check it out!

Rego type checking demo

Consider the following Rego code, which assumes as input a Kubernetes admission review. For resources that are Pods, it checks that the image name starts with a specific prefix.

package kubernetes.admission


deny[msg] {
  input.request.kind.kinds == "Pod"
  image := input.request.object.spec.containers[_].image
  not startswith(image, "hooli.com/")
  msg := sprintf("image '%v' comes from untrusted registry", [image])
}

Notice that this code has a typo in it: input.request.kind.kinds is undefined and should have been input.request.kind.kind.

Consider the following input document:

{
  "kind": "AdmissionReview",
  "request": {
    "kind": {
      "kind": "Pod",
      "version": "v1"
    },
    "object": {
      "metadata": {
        "name": "myapp"
      },
      "spec": {
        "containers": [
          { "image": "nginx", "name": "nginx-frontend" },
          { "image": "mysql", "name": "mysql-backend" }
        ]
      }
    }
  }
}

% opa eval -format pretty -i admission-review.json -d pod.rego
[]

The empty value returned is indistinguishable from a situation where the input did not violate the policy. This error is therefore causing the policy not to catch violating inputs appropriately.

If we fix the Rego code and change input.request.kind.kinds to input.request.kind.kind, then we obtain the expected result:

[
  "image 'nginx' comes from untrusted registry"
  "image 'mysql' comes from untrusted registry"
]

With this feature, it is possible to pass a schema to opa eval, written in JSON Schema.

Consider the Kubernetes admission review input schema. We can pass this schema to the evaluator as follows:

% opa eval -format pretty -i admission-review.json -d pod.rego -s admission-schema.json

With the erroneous Rego code, we now obtain the following type error:

1 error occurred: pod.rego:5: rego_type_error: undefined ref: input.request.kind.kinds
  input.request.kind.kinds
                     ^
                     have: "kinds"
                     want (one of): ["kind" "version"]

This indicates the error to the Rego developer right away, without having the need to observe the results of runs on actual data, thereby improving productivity.

With this new feature, Rego developers will be able to provide JSON Schemas for their input documents and get the most out of static type checking. If you don't have a JSON Schema handy, you can easily obtain one from a sample JSON file using online tools (see links below). In the future, we will add to this feature the support to allow users to specify a directory of schemas and assign schemas to data documents, as well. This will be done via annotations (global, rule-level, rule output) that will indicate what schema is associated with what Rego path expression.

Happy Rego type checking!

For more information and limitations, see documentation and examples.