Adventures in the Modern Monolith: How do you keep a secret at Tilt?

At Tilt, we follow the Modern Monolith Architecture pattern that we host on Azure, leveraging App Services to run our API and Webjobs with individual instances across these scaling to 100+ at a time. All of these instances need access to the same secrets stored in Azure Key Vault—API keys for third-party services, database connection strings, and other sensitive configuration. For years, we managed these secrets using Azure App Service’s built-in Key Vault integration, but as we scaled, this approach began to show its cracks. This is the story of how we built TiltSecret to solve our secrets management challenges.

The Old Way: App Service Environment Settings

Before TiltSecret, our secrets management looked like this:

How It Worked

1. Secrets lived in Azure Key Vault - We stored all our secrets in a single Key Vault per environment (development, staging, production, etc.)
2. App Service settings referenced Key Vault - Each App Service had environment variables configured, sometimes engineers would add a default value to help note it’s a secret but at no point was it a requirement:

{
  "name": "Socure:ApiKey",
  "value": "@Microsoft.KeyVault(VaultName=test-keyvault;SecretName=TestService--ApiKey)"
}

1. Code read from IConfiguration - Our application code accessed secrets through the standard .NET configuration system:

public class SocureConfiguration
{
    public string ApiKey { get; set; }  // Populated from IConfiguration["Socure:ApiKey"]
}

This approach seemed reasonable at first. Azure handled the Key Vault integration automatically, and we could use the familiar IConfiguration pattern throughout our codebase.

The Modern Monolith Context

To understand why this became problematic, you need to understand our architecture:

• Single codebase, multiple deployments: We deploy specific assemblies from the solution across the different API and webjob App Services, building the entire solution to package each of these assemblies
• Shared secrets: All services access the same Key Vault—there’s no secret isolation per service
• 70+ instances: Between our API servers and WebJobs, we run many dozens of instances across environments
• No infrastructure as code (yet): At the time, we were managing infrastructure manually through the Azure Portal
• Self-service culture: We wanted engineers to be able to add new secrets without waiting for DevOps

The Problems We Faced

As our system grew, we encountered three categories of problems:

Self-Imposed Constraints

These were architectural decisions that, while beneficial overall, created challenges for secrets management:

• Modern monolith architecture: Having all secrets accessible from a single Key Vault meant any deployment touched all secrets
• Moving away from App Service settings: We wanted secrets closer to our code and configuration, but not in source control
• Simultaneous deployments: Deploying the entire solution at once meant all instances restarted together
• Engineer self-service: We didn’t want secret management to require manual Azure Portal work or infrastructure changes

Technical Infrastructure Problems

These were the real pain points that forced us to find a better solution:

Azure Key Vault Throttling

This was the big one. Azure Key Vault has rate limits to prevent abuse:

• 2,000 requests per 10 seconds for secrets operations
• Exceeding this results in HTTP 429 responses and exponential backoff

When 50+ App Service instances all restart simultaneously during a deployment, they all try to load secrets from Key Vault at the same time. With 200+ secrets per instance being loaded, we were easily hitting these limits.

The impact: Some instances would fail to start because they couldn’t retrieve their secrets. This was transient—eventually the throttling would clear and instances would succeed—but it meant deployments were unreliable and slow.

The .NET IHost Configuration Problem

We initially considered using Microsoft’s built-in AzureKeyVaultConfigurationProvider. However, this approach had a fatal flaw related to how .NET builds its configuration.

When an IHost builds its configuration, it enumerates all configuration sources during startup. If you add Key Vault as a configuration source, .NET will enumerate all secrets in the vault to build the configuration object.

With our single Key Vault containing all secrets for the entire application, this meant:

• Every instance would enumerate 200+ secrets on startup
• This happened even if that instance only needed 5 of those secrets
• Combined with 50+ instances starting simultaneously, we’d hit throttling limits immediately

The impact: Using the built-in provider wasn’t an option—it solved the problem of having secrets closer to code but left the throttling issue in place

Missing Secrets Block Startup

If a secret didn’t exist in Key Vault when an instance started, the application would fail to start. This sounds obvious, but the consequences were severe:

• Deployment failures: If a developer added code that referenced a new secret but forgot to add it to Key Vault, the deployment would fail
• Timing issues: Code might be merged when a developer was online, but deployed hours later when they weren’t available to fix missing secrets
• No validation: We had no way to catch missing secrets before deployment

Process Problems

Beyond the technical issues, we had workflow problems:

• Manual secret management: Developers had to remember to add secrets to App Service settings in each environment
• Machine restarts: Adding or updating a secret in App Service settings triggered a restart of that instance
• Local development pain: Engineers couldn’t easily run code locally that required secrets without manually adding them to their local appsettings.json
• Secret sharing: No simple way to share test/development secrets with the team

The Solution: TiltSecret

In October 2024, we completed the implementation of TiltSecret. This system addresses all the problems above while maintaining the security and convenience we needed.

Core Design Principles

1. Lazy loading: Only fetch secrets from Key Vault when they’re actually needed
2. Explicit declaration: All secrets are defined as strongly-typed constants in code
3. Caching: Once retrieved, secrets are cached in memory
4. Local override support: Developers can use local values instead of Key Vault for development
5. Pipeline validation: Ensure secrets exist in Key Vault before deployment

The Implementation

Strongly-Typed Secret Definitions

All secrets are defined as static readonly fields in a central TiltSecrets class:

public static class TiltSecrets
{
    public static readonly EmpowerSecret AzureWebJobsStorage = new(new("ConnectionStrings--AzureWebJobsStorage"));
    public static readonly EmpowerSecret AzureWebJobsServiceBus = new(new("ConnectionStrings--AzureWebJobsServiceBus"));
    public static class ExampleThirdPartyService
    {
        public static class Shared
        {
            public static readonly TiltSecret ApiKey = new(new("ExampleThirdPartyService--ApiKey"));
            public static readonly TiltSecret SdkKey = new(new("ExampleThirdPartyService--SdkKey"));
            public static readonly TiltSecret WebhookSecret = new(new("ExampleThirdPartyService--WebhookSecret"));
        }
    }
    // ... many, many more
}

This gives us:

• Discoverability: IntelliSense shows all available secrets
• Type safety: Can’t typo a secret name
• Refactoring support: Renaming a secret updates all references
• Single source of truth: One place to see all secrets in the system

Lazy Retrieval with Caching

The TiltSecret class implements lazy loading:

public record TiltSecret
{
    private string _retrievedSecretValue;
    public TiltSecretKey Key { get; }

    public virtual string RetrieveSecretValueFromKeyVault()
        => _retrievedSecretValue ??= TiltAzureKeyVaultSecretsManager.GetSecret(Key);
}

The TiltAzureKeyVaultSecretsManager handles the actual Key Vault interaction:

public static class TiltAzureKeyVaultSecretsManager
{
    private static readonly ConcurrentDictionary<TiltSecretKey, string> _cachedSecrets = new();
    private static readonly NamedLockingMonitor _locker = new();

    public static string GetSecret(TiltSecretKey key)
    {
        lock (_locker[key.Value])
        {
            if (_cachedSecrets.TryGetValue(key, out var secretValue))
            {
                return secretValue;
            }

            var secretResponse = SecretClient.GetSecret(key);
            _cachedSecrets[key] = secretValue = secretResponse.Value?.Value;
            return secretValue;
        }
    }
}

Key features:

• Per-secret locking: Multiple threads requesting the same secret won’t cause duplicate Key Vault calls
• In-memory caching: Once retrieved, secrets stay in memory for the lifetime of the process
• Lazy evaluation: Secrets are only fetched when RetrieveSecretValueFromKeyVault() is called
  • Note: This method was explicitly named this to make it clear to engineers that calling this method would result in actually making a call to Key Vault.

Usage in Code

Secrets are now accessed explicitly in configuration classes:

public class ExampleThirdPartyServiceConfiguration
{
    public string IdPlusBaseUrl { get; set; }
    public string DocumentUploadBaseUrl { get; set; }

    public string ApiKey => TiltSecrets.ExampleThirdPartyService.Shared.ApiKey.RetrieveSecretValueFromKeyVault();
    public string SdkKey => TiltSecrets.ExampleThirdPartyService.Shared.SdkKey.RetrieveSecretValueFromKeyVault();
    public string WebhookSecret => TiltSecrets.ExampleThirdPartyService.Shared.WebhookSecret.RetrieveSecretValueFromKeyVault();
}

This pattern means:

• Secrets are only loaded when the configuration class property is accessed
• If a service doesn’t use ExampleThirdPartyService, it never loads these secrets
• The secret value is cached after first access

Integration with IConfiguration (When Needed)

Some Azure services (like Azure Functions triggers) require secrets to be available through IConfiguration. For these cases, we created a custom configuration provider:

public static class TiltSecretsForConfiguration
{
    public static readonly HashSet<TiltSecret> AllSecrets =
    [
        TiltSecrets.AzureWebJobsStorage,
        TiltSecrets.AzureWebJobsServiceBus,
        TiltSecrets.DataStorageAccount,
        // Only secrets that MUST be in IConfiguration
    ];
}

This provider is added to the configuration builder:

builder
    .AddJsonFile("appsettings.json")
    .AddJsonFile($"appsettings.{environment}.json")
    .AddTiltSecretsConfiguration(TiltSecretsForConfiguration.AllSecrets)
    .AddEnvironmentVariables();

The custom TiltSecretConfigurationProvider overrides TryGet to lazily fetch secrets:

public override bool TryGet(string key, out string value)
{
    if (_secrets.TryGetValue(key, out var secret))
    {
        value = secret.RetrieveSecretValueFromKeyVault();
        return true;
    }
    value = null;
    return false;
}

This means:

• Only the secrets we explicitly add to TiltSecretsForConfiguration are available through IConfiguration
• They’re still loaded lazily—only when accessed
• We avoid the enumeration problem of the built-in provider

Local Development Support

For local development, engineers can override secrets without needing Key Vault access:

// appsettings.debug.Development.json
{
  "AzureKeyVault": {
    "UseSecretOverrideValues": true
  },
  "Socure": {
    "ApiKey": "test-api-key-for-local-dev"
  }
}

The TiltAzureKeyVaultSecretsManager checks for overrides first:

public static string GetSecret(TiltSecretKey key)
{
    if (_useSecretOverrideValues)
    {
        var localOverrideValue = GetOverrideValue(key);
        if (!string.IsNullOrEmpty(localOverrideValue))
        {
            return localOverrideValue;
        }
    }

    // Fall back to Key Vault...
}

This enables:

• Frictionless local development: New engineers can run the app without Key Vault access
• Test values: Use sandbox/test API keys locally
• Faster iteration: No network calls to Key Vault during local development

Pipeline Validation

To prevent missing secrets from causing deployment failures, we built a validation tool that runs in our CI/CD pipeline:

public class ValidateSecretsExistRunner
{
    internal async Task<IList<SecretValidationResult>> ValidateSecretsForEnvironment(string environment)
    {
        var secretKeysInCode = GetTiltSecretsDefinedInCode();
        var secretClient = _secretClientFactory.CreateSecretClient(keyVaultName);

        var keyVaultSecretKeys = await secretClient
            .GetPropertiesOfSecretsAsync()
            .Select(sk => sk.Name)
            .ToListAsync();

        var missingSecrets = secretKeysInCode
            .Except(keyVaultSecretKeys, StringComparer.OrdinalIgnoreCase)
            .ToList();

        if (missingSecrets.Count > 0)
        {
            return new SecretValidationResult { WasValidationSuccessful = false };
        }

        return new SecretValidationResult { WasValidationSuccessful = true };
    }
}

This runs before deployment and:

• Uses reflection to find all TiltSecret fields in the compiled code
• Compares them against what’s actually in Key Vault
• Fails the pipeline if any secrets are missing
• Provides clear error messages showing which secrets need to be added

The Results

Since implementing TiltSecret, we’ve seen significant improvements:

Eliminated Throttling Issues

• Before: Deployments regularly hit Key Vault throttling limits, causing instance startup failures
• After: Zero throttling issues. Each instance only loads the secrets it actually uses, and only when needed

Prevented Deployment Failures

• Before: Missing secrets caused production deployment failures
• After: Pipeline validation catches missing secrets before deployment

Improved Developer Experience

• Before: Engineers needed to manually download the secrets they needed and put them in a local json file that they need to remember to not check-in.
• After: Engineers can use their own credentials to load the secrets they need at runtime for local development, and override any secrets they want locally.

Better Code Quality

• Before: Secret names were magic strings scattered throughout the codebase
• After: Strongly-typed secret references with IntelliSense support and compile-time checking

Conclusion

Building TiltSecret was a journey from a simple, built-in solution that didn’t scale to a custom implementation that fits our specific needs. The key insights were:

• Lazy loading eliminates unnecessary Key Vault calls
• Explicit declaration makes secrets discoverable and type-safe
• Pipeline validation prevents deployment failures
• Local override support improves developer experience

If you’re running a .NET application on Azure with similar scale challenges, we hope our experience helps you avoid some of the pitfalls we encountered. The modern monolith architecture has many benefits, but it requires thoughtful solutions to problems like secrets management.

---

Have questions about our implementation or want to discuss secrets management strategies? Reach out to our engineering team!

---

About Aaron Mumm

Experienced Software Engineer at Tilt with a demonstrated history working across diverse technology domains. Holds a U.S. patent for “Design and Systems Architecture for Internet of Things.” Washington State University alum, TEALS teaching assistant, and Mentors in Tech mentor. Passionate about building scalable systems and solving complex infrastructure challenges.