google-cloud-solution-multi-agent-security

Agent Gateway multi-agent security

Critical Enforcement Rules & Rationale

• Gcloud Release Tracks: Always use the exact release tracks specified in
  the commands (e.g., gcloud beta network-services agent-gateways). Omitting
  these prefixes causes commands to fail because Agent Gateway features are
  located in specialized, non-default namespaces.
• API Enablement: Include modelarmor.googleapis.com in the API
  enablement list when setting up guardrails. Excluding it prevents Model
  Armor policies and filters from successfully attaching to the Gateway.
• Egress Verification: Egress policy verification requires using the
  Python script
  (scripts/verify_egress_policies.py),
  not curl. Egress gateways rely on runtime SDK lifecycle handling and JWT
  context that a standard curl command cannot simulate correctly.
• Model Armor Keys: In model-armor-config.yaml, always include both
  piAndJailbreakFilterSettings and sdpFilterSettings (filterEnforcement: ENFORCE). Invalid or missing filters cause deployment validation failures
  or lead to silent bypasses of the guardrails.
• Subnet Private Access: Any subnet hosting a Private Service Connect
  network attachment for Egress Gateways must have private_ip_google_access = true enabled in Terraform. Disabling this blocks connectivity to
  Google-managed endpoints, causing total routing failures for agents.
• Direct Delivery: Immediately provide the requested architecture,
  configuration files, CLI commands, scripts, and diagrams in full. Do not
  stop at a planning phase, do not generate a plan artifact, and do not ask
  for user confirmation before delivering outputs.
• No Infrastructure Execution: Do not attempt to run deployment or
  verification commands (such as gcloud, kubectl, terraform, or curl)
  against real cloud resources during design. You are generating plan
  configurations, not executing them.

[!IMPORTANT] Just-In-Time (JIT) Resource Loading Protocol: Inspect
template files in assets/ and executable scripts in
scripts/ using view_file as needed for extended configurations,
deployment scripts, and test suites.

---

Quick Reference: Required Filenames

Always generate files with these exact names when requested:

1. agw-ingress-config.yaml
   (assets/agw-ingress-config.yaml)
2. agw-egress-config.yaml
   (assets/agw-egress-config.yaml)
3. agw-authz-extension.yaml
   (assets/agw-authz-extension.yaml)
4. agw-authz-policy.yaml
   (assets/agw-authz-policy.yaml)
5. model-armor-config.yaml
   (assets/model-armor-config.yaml)
6. sgp-policy.yaml (assets/sgp-policy.yaml)
7. iap-policy.json (assets/iap-policy.json)
8. model-armor-payload.json
   (assets/model-armor-payload.json)

---

1. Dual Ingress & Egress Architecture Design (dual_ingress_egress_architecture_design)

• Ingress Pattern: CLIENT_TO_AGENT fronted by Ingress Control Plane
  (Agent Gateway, Model Armor).

• Egress Pattern: AGENT_TO_ANYWHERE utilizing Egress Control Plane
  (Agent Gateway, roles/iap.egressor CEL policies, Cloud DNS) and Egress
  Data Plane (PSC Interface, Cloud Run, PSC Google APIs Global Endpoint),
  coordinated via Agent Registry & Agent Engine runtime.

• Mermaid Diagram:

  graph TD
      Client["External Clients"] -->|HTTPS / MCP| GLB["Global Load Balancer"]
      GLB --> Ingress["Ingress Agent Gateway (CLIENT_TO_AGENT)"]
      Ingress --> MA["Model Armor (CONTENT_AUTHZ)"]
      MA --> Agent["Agent Engine Agents (BillingAgent, SupportAgent, FraudAgent)"]
      Agent --> Egress["Egress Agent Gateway (AGENT_TO_ANYWHERE)"]
      Egress --> PSC["Private Service Connect Network Attachment"]
      PSC --> Tools["Private MCP Tool Backends"]
  

---

2. Ingress & Egress Guardrail Policy Config (ingress_and_egress_guardrail_policy_config)

When requested for Ingress & Egress guardrail policy configs, you MUST generate
and create all required files in the workspace:

• agw-ingress-config.yaml
  (assets/agw-ingress-config.yaml): Declares
  governedAccessPath: CLIENT_TO_AGENT with protocols HTTP and MCP.
• agw-egress-config.yaml
  (assets/agw-egress-config.yaml): Declares
  governedAccessPath: AGENT_TO_ANYWHERE with protocol MCP.
• agw-authz-extension.yaml
  (assets/agw-authz-extension.yaml):
  Configures AuthzExtension service for IAP authorization.
• agw-authz-policy.yaml
  (assets/agw-authz-policy.yaml): Configures
  AuthzPolicy action ALLOW targeting both Ingress and Egress gateways.
• iap-policy.json (assets/iap-policy.json): Binds
  roles/iap.egressor with CEL condition checking
  iap.googleapis.com/mcp.toolName == 'get_account_balance' && iap.googleapis.com/mcp.tool.isReadOnly == true.
• model-armor-config.yaml
  (assets/model-armor-config.yaml): Enables
  piAndJailbreakFilterSettings and sdpFilterSettings with
  filterEnforcement: ENFORCE.
• sgp-policy.yaml (assets/sgp-policy.yaml):
  Implements Natural Language Constraints blocking transactions > $1000 and
  sanitizing PII.

---

3. Ingress & Egress Infrastructure Deployment (ingress_and_egress_infrastructure_deployment)

Inspect and provide the step-by-step gcloud CLI commands from
scripts/deploy_infrastructure.sh:

1. Enable Required APIs: compute, networkservices, networksecurity,
   modelarmor, iap, agentregistry, serviceextensions, and aiplatform.
2. Import Agent Gateways: Ingress (agw-ingress-config.yaml) and Egress
   (agw-egress-config.yaml) via gcloud alpha network-services agent-gateways import.
3. Import Authz Extension: agw-authz-extension.yaml via gcloud beta service-extensions authz-extensions import.
4. Import Authz Policy: agw-authz-policy.yaml via gcloud beta network-security authz-policies import.

---

4. Ingress & Egress Security Validation (ingress_and_egress_security_validation)

When validating security for Ingress and Egress:

1. Ingress 403 Unauthenticated Test: Provide the copy-pasteable
   verification curl command from
   scripts/validate_ingress_unauth.sh
   sending an unauthenticated POST request to the Reasoning Engine endpoint
   expecting HTTP 403 Forbidden.
2. Python Egress Verification Script (MUST use Python script snippet, NOT
   curl): Provide the Python verification script snippet from
   scripts/verify_egress_policies.py
   sending JSON-RPC tools/call requests (get_account_balance) through the
   Egress Gateway to verify HTTP 200 for allowed tools.
3. Model Armor Test Payload: Generate model-armor-payload.json
   (assets/model-armor-payload.json)
   containing prompt injection/jailbreak instructions.

---

5. Troubleshooting Ingress & Egress Failures (troubleshooting_ingress_and_egress_failures)

• Ingress 403 (Client-to-Agent):

  • Root Cause: Unauthenticated client requests or missing/invalid OAuth
    2.0 / IAP identity tokens.
  • OAuth Configuration Steps:
    1. Configure OAuth 2.0 Client ID credentials in Google Cloud Console.
    2. Grant the client identity / service account
       roles/iap.httpsResourceAccessor permission.
    3. Exchange credentials with Google OAuth to acquire an OIDC / OAuth ID
       token.
    4. Pass the token in the Authorization: Bearer <TOKEN> header.
  • Verification Command: Provide the curl command from
    scripts/verify_ingress_auth.sh.

• Egress 403 (Agent-to-Anywhere):

  • Root Cause: Missing roles/iap.egressor IAM bindings on the Agent
    Identity, malformed principal ID, or mismatched CEL condition on tool
    metadata.
  • Fix Command: Provide the exact gcloud command from
    scripts/fix_egress_iap.sh.

---

6. Hybrid VPN Connectivity & Egress Routing (hybrid_vpn_connectivity_egress_routing)

• Terraform HCL: Refer to baseline Terraform config in
  assets/main.tf for VPC, subnets
  (private_ip_google_access = true), PSC network attachment, Cloud DNS
  private forwarding for aws.internal., and HA VPN gateway/router.
• Egress Gateway Config (agw-egress-config.yaml): Generate configuration
  declaring governedAccessPath: AGENT_TO_ANYWHERE, pointing to the PSC
  network attachment, and referencing aws.internal. in dnsPeeringConfig
  (see assets/agw-egress-config.yaml).
• Python SDK Deployment Script: Refer to
  scripts/hybrid_vpn_agent.py for the complete
  script initializing Vertex AI with agent_to_anywhere_config referencing
  the Egress Gateway, enabling telemetry, and deploying HybridAgent using
  types.IdentityType.AGENT_IDENTITY.

---

7. Private Egress GKE Internal Load Balancer (private_egress_gke_internal_load_balancer)

• Expose GKE internal MCP tool server via an Internal Load Balancer (ILB) at
  literal IP 10.0.1.50, connecting via Agent Gateway PSC Interface + Cloud
  DNS Private zone.
• Cloud DNS Record Mapping: Provide the command from
  scripts/create_gke_dns_record.sh mapping
  the private domain to GKE's private ILB IP 10.0.1.50.
• Explicit TLS Warning: Agent Gateway egress does not natively trust
  self-signed certificates or private enterprise CAs. You must use
  publicly trusted TLS certificates signed by a trusted Certificate Authority
  (e.g., Let's Encrypt).

---

8. Governance Controls Model Armor SGP (governance_controls_model_armor_sgp)

When configuring dual safety layers with Model Armor on Ingress and SGP on
Egress:

1. Model Armor Config: Generate model-armor-config.yaml
   (assets/model-armor-config.yaml) with
   piAndJailbreakFilterSettings and sdpFilterSettings (filterEnforcement: ENFORCE).
2. Semantic Governance Policy: Generate sgp-policy.yaml
   (assets/sgp-policy.yaml) with Natural Language
   Constraints blocking transactions > $1000 and sanitizing PII.
3. Curl PATCH Command: Provide the curl command from
   scripts/enforce_sgp_patch.sh to update
   authzExtensions with sgpEnforcementMode set to ENFORCE.

---

9. Multi-Agent Cloud Run Egress Routing (multi_agent_cloud_run_egress_routing)

Do NOT produce a plan artifact or stop at planning. When configuring multi-agent
Cloud Run egress routing, you MUST directly provide and generate ALL required
components:

1. Egress Gateway Config (agw-egress-config-run.yaml): Generate
   configuration declaring governedAccessPath: AGENT_TO_ANYWHERE, PSC network
   attachment, and DNS peering for *.run.app (see
   assets/agw-egress-config-run.yaml).
2. Register Cloud Run Services in Agent Registry: Provide the registration
   commands from
   scripts/register_cloud_run_services.sh
   registering all 3 Cloud Run services (marketing-tool-service,
   sales-tool-service, support-tool-service) in the us-east4 Agent
   Registry.
3. iap-policy.json (Multi-Agent): Generate iap-policy.json
   (assets/iap-policy-multi-agent.json)
   containing all 3 principal:// bindings in the members list under
   roles/iap.egressor.
4. Python SDK Deployment Script: Refer to
   scripts/multi_agent_cloud_run.py for the
   complete GenAI SDK deployment script.

---

10. Advanced Model Armor Filtering (advanced_model_armor_filtering)

For custom keyword matching, configure userDefinedFilterSettings (see
assets/model-armor-advanced.yaml).

---

11. Known Traps & Gotchas (known_traps_and_gotchas)

• network_attachment is ForceNew: Enabling Semantic Governance
  Policies (SGP) or modifying network attachments after the initial Terraform
  apply will force-recreate the gateway resource. If not managed carefully,
  this can cause dependency deadlocks during destroy operations. Plan
  infrastructure sequencing accordingly.
• Authz Policy Limit: An Agent Gateway allows at most 4 custom
  authorization policies attached concurrently. Ensure your security posture
  consolidates rules within this limit.