Ephemeral values in Terraform

Ephemerality in computing refers to the ability to create something that is short-lived or temporary — a piece of data that exists only for a brief period and is discarded once its purpose is fulfilled. In Terraform, this concept was introduced to manage sensitive data or open a connection in a way that ensures it doesn’t persist beyond its immediate use. We’ve implemented new language constructs that track values only at runtime, making them transient and, therefore, ephemeral by design. These are part of what we call ephemeral values, a term that covers both ephemeral resources, ephemeral input variables, and write-only arguments.

Terraform doesn’t store write-only arguments in the plan or state file, meaning it can’t track changes to their values. As a result, write-only arguments are sent to the provider during every operation. However, providers typically include write-only version arguments (e.g. password_wo_version) alongside write-only value arguments (e.g. password_wo), where the version arguments are stored in the state file and can be tracked for changes.

These write-only version arguments enable users to update a write-only value by incrementing the version number. When the write-only version argument is updated, Terraform detects this change in the plan and notifies the provider. The provider then uses the updated write-only value to update the resource accordingly.

If a resource depends on multiple write-only arguments in the dependency graph, it’s a good idea to keep their version arguments in sync. This way, all the related write-only values are updated together, avoiding inconsistencies and ensuring the provider uses the correct values during each operation. In the next section, we’ll cover deferring ephemeral resources, and the example code will demonstrate how to synchronize versioning for write-only arguments.

If an input argument of an ephemeral resource references a value that is not yet known but will be during or after the plan, Terraform defers the resource’s execution to the apply stage instead of running it during the plan. This behavior allows Terraform to evaluate the ephemeral resource at the correct time and ensures that the resource is not executed prematurely.

Similarly, ephemeral resources form nodes in Terraform’s dependency graph. When a managed resource or data source depends on an attribute of an ephemeral resource, Terraform automatically provisions the ephemeral resource first.

As we concluded earlier, ephemeral resources are executed during every plan and apply. It’s important to model your Terraform configuration in such a way that the dependency graph of all of the managed resources and ephemeral resources ensures correct execution order and prevents unintended behavior.

By using the ephemeral random password and AWS Secrets Manager secret we created earlier, we can take advantage of ephemerality and its deferring logic to securely provision an AWS RDS instance.

In the above example, the ephemeral resource aws_secretsmanager_secret_version depends on an argument that Terraform doesn’t initially know. Terraform defers executing this ephemeral resource until the apply stage, ensuring it runs only after the necessary information becomes available.

During the apply stage Terraform first creates the secret in AWS Secrets Manager using the ephemeral random_password, then Terraform retrieves the secret using the ephemeral aws_secretsmanager_secret_version resource. The last thing Terraform does is write the password to the write-only password_wo argument of the aws_db_instance resource.

Ephemeral resources have a lifecycle that differs from other resources and data sources. The lifecycle has three steps:

• Opening: When Terraform needs the result of an ephemeral resource, it opens it — much like reading a secret from a secrets manager.
• Renewing: If Terraform needs access to the ephemeral resource for longer than the remote system’s enforced expiration time, Terraform asks the provider to periodically renew it.
• Closing: Once Terraform no longer needs an ephemeral resource, it closes it. This happens after the providers that depend on an ephemeral resource complete all of their work for the current Terraform run phase.

All ephemeral resources implement a form of opening logic, but not all implement closing or renewing logic. Whether they do depends on the ephemeral nature of the resource — handling tunnel connections, for instance, may require periodic renewal and proper closure of the connection. Leasing a temporary set of credentials from Vault would similarly imply renewal and revocation of those credentials when they’re no longer needed.

Ephemerality in Terraform prevents introspection of the value itself, which is the expected and intended behavior. However, this could lead to a potential issue. When using the ephemeral random password, you’re generating a new secret that hasn’t been persisted to a secrets manager yet, so it’s important to store it to ensure it remains accessible. This is why, in the third code snippet above, we write the ephemeral random password to a secrets manager first and then fetch it from there during the apply stage using another ephemeral resource, effectively persisting an ephemeral secret to a secrets manager.

However, if you’re reading existing secrets from a secrets manager that’s managed outside of the Terraform module you’re working in, there’s no need to persist them again.