Data Governance: Context-Based Access Control at Scale

In our introductory post on the Tempus data pipeline, we described how Tempus treats data as a high-fidelity product to keep one of the world’s largest multimodal datasets secure and compliant as it grows past 500 petabytes. This post looks deeper at how context-based access control lets us enforce the right to use every piece of data, automatically, at scale.

The instinctive way to control access is by role and location: this team has permission to this folder; that project can read that bucket. It works until data starts to move and combine, at which point it quickly becomes a bureaucratic headache.

Let’s think about what it takes for a patient’s data to be handled correctly. The hospital has engaged with Tempus for TIME clinical trial matching services, with additional rights to de-identify the data. The data passes through Tempus pipelines to abstract, structure, and run against models. Those results are returned to the care team. Included in the provider’s contract is a clause that states that Tempus has rights to de-identify and commercialize the data. Ultimately, the data is joined with other data, de-identified, and used. At the end of the chain—dozens of transformations and many sources later—the resulting dataset still has to honor the most restrictive promise made anywhere along the route. The problem with an access policy around role and location is that any system attempting to use the data has to already understand how it can be used.

Now multiply that by a dataset growing past 500 petabytes, thousands of automated jobs per day, and new agreements landing every week. The problem is not “who can see which folder,” but “how does every compliance requirement attached to every piece of data survive every transformation” at scale.

The data needs to flow, and every requirement attached needs to flow with it, without bottlenecking the science. We need our access decisions to answer the only question that really matters: what services can this data be used for? And we need that answer to travel with the data automatically.

Start With the Unit: The Data Product

At Tempus, a data product is a standardized, self-describing package of data—the unit that every system uses to store and exchange information. Data products are defined by a versioned schema describing what it looks like and are governed by a policy describing who may access it.

Tempus’s Data Products Platform spans both the transaction and analytical sides of Tempus. The Data Products Service is a RESTful API that serves the transactional side. It provides a consistent interface into Tempus’s twelve billion data product files while also enforcing schema conformity and access control according to its policies. The Data Products Library automatically loads this collection of files into a data warehouse for analytical use while providing the same schema conformity and access control.

Our context-based access control is fundamentally built on top of the lineage provided by Tempus’s data products by transitioning from this role-and-location access model to one more focused on the purpose of the action being taken.

Context-Based Access Control: Authorizing by Purpose

Context-based access control reframes authorization around purpose rather than place.

Every software system, called a workload at Tempus, operates under a work context. A work context describes the permitted uses for the work being done. It defines what data the workload is allowed to read and how the data it produces is allowed to be used downstream. We call these acceptable uses. Data, in turn, carries the set of acceptable uses it inherited from the contexts under which it was collected and processed.

The access decision then becomes a mathematical comparison of these acceptable uses. A workload may read a piece of data only if everything the workload requires is among the permitted uses. If a pipeline needs data for a purpose the data was never cleared for, then the request is denied: not by a person reviewing it later, but by the platform at the moment of access.

Crucially, processing can only ever narrow acceptable uses, never broaden them. A dataset derived from more restrictive sources inherits the stricter constraints. This is the property that traditional access control lacks: usage rights are preserved through every join, transformation, and derivation, instead of being lost the moment data moves.

Attaching Context at the Edge

Context has to start somewhere, and for most data that somewhere is “Edge”—the connectivity layer where data first enters Tempus from a hospital, lab, or partner. Our conversation here around Edge will be focused on its integration with Contexts. See our previous blog post on our Edge platform for more general technical detail on this data ingestion platform.

The canonical source of patient information at Tempus is the Master Patient Index. This service houses information for all things patients, but most importantly here is the association between Contexts and Patients. When Edge ingests patient data into Tempus it will also store a Context representing the Covered Entity (CE) or the Business Associate Agreement (BAA) status under which Tempus has received this data. The CE or BAA also describe for what services Tempus is permitted to use this patient’s data.

By representing these Acceptable Uses as Contexts and attaching those Contexts directly to our canonical patient definition, we ensure that these restrictions are cleanly mapped to all of the data that we have for a given patient and data agreement, and applied to any and all transformations as that data flows through Tempus.

Carrying Context Forward: Stamping and Lineage

The Edge platform associates Contexts, and their acceptable uses, with patients at the point of ingestion. Those Contexts also need to travel with the data as it flows through Tempus’s pipeline, transformations, and joins with other data along the way. We want to ensure we always have a complete picture of how the new, derived data may be used.

When a workload begins its work, it reaches out to our Work Context Service to retrieve a Context Token. This token is cryptographically signed and passed to other Tempus platform systems to represent that workload’s authority to operate within a given context. When that workload reads a data product, it receives another cryptographically signed token that we call a “data token”. The data token proves the data was accessed legitimately and records the contexts attached to it. When that workload then writes a new, derived data product, it presents the tokens for every source it uses. The platform reads that lineage, derives the permitted uses from all inputs, and stamps the new data product with the Contexts of every source—plus the workload’s own operating Context.

When a data product is read, the Data Products Service will authorize the read by reviewing the reference collection of Contexts, representing the lineage of the data product. Contexts are represented as a two-dimensional array where each inner element within the Array represents the collection of Contexts that was applied to an upstream data product. The Data Products Service will inspect the Acceptable Uses present on each context within the inner elements, taking the mathematical union to ensure that each permissible use of that data is represented. From there, it takes the intersection of those unions to compute the final list of acceptable uses to find the set of uses permitted by every upstream data product. This guarantees that the derivative is never more permissive than anything it was built from.

The Data Products Library provides the same context-aware access control through different mechanisms due to the more limited features available in BigQuery, our data warehouse of choice. In this analytical context, we need to authorize access to not just a handful of files at a time. Instead, we may need to litigate access across potentially billions of rows at a time. We do this by computing the “intersection of unions” described above, ahead of time. We compute this list of applicable acceptable uses at data-ingestion-time then hash the result and write it to the data tables. The hash allows us to leverage BigQuery’s clustering to prune out data for which the query-er does not have access without performance penalties. We then place an authorized view in front of that table, which will query a lookup table of hash definitions and ensure that the query-er has access.

The effect is a chain of custody for usage rights. Every derived data product knows exactly what it was built from, and it inherits its access rules from that lineage rather than having them reassigned at creation time. There is an additional safeguard layered on top: data classified as identifiable cannot be read by work authorized only for de-identified use, regardless of anything else. The most sensitive data has a hard floor beneath the purpose-based rules. These automated guardrails are helping us to confidently govern and expand our de-id datasets by being able to transform larger volumes of sensitive data.

Governance That Scales

Proactively applying rules to broad data classifications, such as tables or buckets, creates bottlenecks at scale. Our approach holds up at scale because the platform proactively computes rules dynamically based on the specific lineage of a particular data product.

Because every access decision is a programmatic comparison between what a workload needs and what the data permits, governance does not depend on a person being in the loop for each request. Our data governance teams no longer need to manually review individual requests for access to data. They instead focus on defining the Contexts attached to our data sharing agreements and workload systems up front, at which point they can trust the Data Products platform to correctly make dynamic decisions. This frees them up from reviewing individual requests to data indefinitely.

Similarly, our engineering teams no longer need to concern themselves with requesting access to data and jumping through hoops to justify their access. Once their workloads are represented by Contexts, and integrated with the Work Context Service and the Context-aware features of the Data Products Platform, then they can simply make requests to read data and let the Platform sort out the access.

Contexts allow Tempus’s data to carry its permitted uses, decided by purpose, preserved through every transformation. For the health systems, life-science partners, and patients who trust us with some of the most sensitive data there is, that principle is not a feature. It is the foundation everything else at Tempus is built on.