Overview

The Vault 1.16.x upgrade guide contains information on deprecations, important or breaking changes, and remediation recommendations for anyone upgrading from Vault 1.15. Please read carefully.

Important changes

External plugin variables take precedence over system variables

Vault gives precedence to plugin environment variables over system environment variables when loading external plugins. The behavior for builtin plugins and plugins that do not specify additional environment variables is unaffected.

For example, if you register an external plugin with SOURCE=child in the env parameter but the main Vault process already has SOURCE=parent defined, the plugin process starts with SOURCE=child.

Refer to the plugin management page for more details on plugin environment variables.

How to opt out

To opt out of the precedence change, set the VAULT_PLUGIN_USE_LEGACY_ENV_LAYERING environment variable to true for the main Vault process:

$ export VAULT_PLUGIN_USE_LEGACY_ENV_LAYERING=true

Setting VAULT_PLUGIN_USE_LEGACY_ENV_LAYERING to true tells Vault to:

1. prioritize environment variables from the Vault server environment whenever the system detects a variable conflict.
2. report on plugin variable conflicts during the unseal process by printing warnings for plugins with conflicting environment variables or logging an informational entry when there are no conflicts.

For example, assume you set VAULT_PLUGIN_USE_LEGACY_ENV_LAYERING to true and have an environment variable SOURCE=parent.

If you register an external plugin called myplugin with SOURCE=child, the plugin process starts with SOURCE=parent and Vault reports a conflict for myplugin.

Known issues and workarounds

Error configuring the JWT auth method

Affected versions

• 1.16.1

Issue

An error will occur when configuring the built-in jwt auth method. This will affect new mounts and updates to existing mounts. Existing mounts should not encounter an error if no modifications are made.

See this issue for more details.

Workaround

Do not attempt to update an existing mount's config. New mounts can run the plugin as an external plugin to avoid the error.

Error logging in with LDAP auth method when anonymous group search is enabled

Affected versions

• 1.16.0

Issue

Depending on their LDAP configuration, some customers may encounter an error when attempting to login with ldap auth method when anonymous group search is enabled. See this issue for more details.

Workaround

There is no workaround.

Error logging in with LDAP auth method

Affected versions

• 1.16.0

Issue

Depending on their LDAP configuration, some customers may encounter an error when attempting to login with ldap auth method. Active Directory users are affected by this bug. See this issue for more details.

Workaround

There is no workaround.

Existing clusters do not show the current Vault version in UI by default

Affected versions

• 1.16+

Issue

Previous versions of the Vault UI fetched version information from un-authenticated endpoints like sys/health and sys/seal-status. Since introducing the redact_version TCP listener parameter, version information may no longer be available under some configurations. As a result, the Vault UI now uses a new, authenticated endpoint (sys/internal/ui/version) to fetch version information.

By default, all new Vault servers created with v1.16+ include the following rule, in the automatically-generated policy:

# Allow a token to look up the Vault version. This path is not subject to
# redaction like the unauthenticated endpoints that provide the Vault version.
path "sys/internal/ui/version" {
    capabilities = ["read"]
}

However, the default policy for existing Vault servers does not update automatically during the upgrade. You must updated the policy manually in order for the Vault version to be displayed in the Vault UI.

No other functionality in the Vault UI is affected be this issue.

You can use the Vault CLI to update the default policy and allow the Vault IU to query the Vault server for version information:

$ vault policy read default | cat - <<< '
# Allow a token to look up the Vault version. This path is not subject to
# redaction like the unauthenticated endpoints that provide the Vault version.
path "sys/internal/ui/version" {
    capabilities = ["read"]
}
' > default-policy.hcl
$ vault policy write default ./default-policy.hcl

Default lease count quota enabled when upgrading from Vault versions before 1.9

Affected versions

• 1.16+

Issue

Vault began tracking version history as of version 1.9. As of version 1.16, new Vault installs automatically enable a lease count quota by consulting the version history. If the version history is empty on upgrade, Vault treats the upgrade as a new install and automatically enables a default lease count quota.

Before you upgrade Vault from a version prior to 1.9 to versions 1.16+, you should check the current number of unexpired leases via the vault.expire.num_leases metric.

If the number of unexpired leases is below the default lease count quota, value of 300000 no extra steps are required.

If the number of unexpired leases is greater than the default threshold of 300000, there is a two step workaround to safely upgrade without the default lease count quota:

1. Upgrade to any Vault version prior to 1.16 (between 1.9 and 1.15) to populate the version store.
2. Upgrade to Vault version 1.16+.

You can review, modify, and delete the global default quota at any point with the /sys/quotas/lease-count/default endpoint:

$ vault read sys/quotas/lease-count/default
$ vault delete sys/quotas/lease-count/default
$ vault write sys/quotas/lease-count/default max_leases=<# of max leases>

Refer to Protecting Vault with Resource Quotas for a step-by-step tutorial on quota tuning.

Refer to Lease Explosions for more information on lease management.

PKI OCSP GET requests can return HTTP redirect responses

If a base64 encoded OCSP request contains consecutive '/' characters, the GET request will return a 301 permanent redirect response. If the redirection is followed, the request will not decode as it will not be a properly base64 encoded request.

As a workaround, OCSP POST requests can be used which are unaffected.

Impacted versions

Affects all current versions of 1.12.x, 1.13.x, 1.14.x, 1.15.x, 1.16.x