AI Gateway Backend Engineer, Python & Kubernetes
Overview
You will own the core of our AI Gateway platform, a high-performance intermediary layer that routes and governs traffic between enterprise clients and leading LLM providers. You will design and operate a latency-sensitive, policy-driven system handling massive concurrent streaming connections at scale. You will join a focused team of engineers building next-generation infrastructure with Python, Kubernetes, and Redis Streams. This role demands deep expertise in async networking and cloud infrastructure, offering the chance to shape the backbone of AI adoption for global enterprises.
What You'll Do10
- 1Build high-throughput streaming APIs with Python and asyncio, handling WebSocket and SSE connections with low-latency and zero-copy buffering.
- 2Design and operate the AI Gateway's data plane, managing bidirectional streaming, back-pressure, and chunked transfer for real-time LLM inference.
- 3Develop production-grade REST and streaming endpoints using FastAPI, Pydantic, and dependency injection, with comprehensive OpenAPI documentation.
- 4Deploy and scale the gateway on Kubernetes using Docker, AKS/EKS/GKE, and configure HPA/VPA scaling and rolling updates.
- 5Implement asynchronous, event-driven architectures with Redis Streams, RabbitMQ, or Kafka for decoupled task queues and retry logic.
- 6Tune and manage PostgreSQL with query optimization, partitioning, and PgBouncer pooling, while securing secrets with HashiCorp Vault.
- 7Configure cloud networking with L4/L7 load balancing, NGINX/Envoy reverse proxies, mTLS between services, and automated certificate lifecycle via ACME.
- 8Automate infrastructure provisioning with Terraform and build CI/CD pipelines with GitHub Actions or GitLab CI, including container scanning and canary deployments.
- 9Integrate enterprise security gateways such as CASB, DLP, or WSS, and ensure centralized logging and audit trails.
- 10Monitor gateway performance with Datadog, Prometheus/Grafana, or Azure Monitor, and enforce SLA/SLOs with proactive incident response.
Requirements10
- 13+ years building async applications with Python and asyncio, including low-level TCP/UDP sockets and HTTP/2 or HTTP/3.
- 23+ years developing backend services with FastAPI, including Pydantic models and OpenAPI specs.
- 3Production experience with streaming protocols (WebSocket, SSE) and handling large payloads with back-pressure.
- 4Hands-on Kubernetes deployment on AKS/EKS/GKE, with knowledge of HPA/VPA and service mesh like Istio or Envoy.
- 5Familiarity with async messaging systems such as Redis Streams, RabbitMQ, or Kafka for decoupled workloads.
- 6Advanced PostgreSQL skills including query tuning, partitioning, and connection pooling with PgBouncer.
- 7Experience with HashiCorp Vault for dynamic secrets and encryption.
- 8Deep knowledge of cloud networking: L4/L7 load balancing, TLS termination, reverse proxies, and mTLS.
- 9Proficiency with Terraform for infrastructure-as-code and CI/CD with GitHub Actions or GitLab CI.
- 10Experience with IAM and SSO: configuring Azure AD/Entra ID, Okta, SAML 2.0 or OIDC flows.
Salary Insight
$100 - $130k per year
Similar open positions
Explore active roles that match your skills and interests.
System One
VerifiedFull Stack Developer (Python, FastAPI) in Pittsburgh
You will own the full lifecycle of backend services for enterprise AI and analytics platforms in the telecommunications sector, built with Python, FastAPI, and Azure. You will design and optimize data access layers using Snowflake, Databricks, and Apache Iceberg, and deploy on Microsoft Azure with modern DevOps practices. This role stands apart by integrating GenAI frameworks like Azure OpenAI into production APIs, working with a close-knit team with global reach.
Atash Enterprises, LLC
VerifiedAI Systems Engineer, Agentic AI & RAG Pipelines
You will design, build, and operate production-grade agentic AI systems at Atash Enterprises, owning multi-agent orchestration workflows, RAG pipelines, and natural-language interfaces to enterprise data. You will deploy and monitor these systems, ensuring reliability and performance at scale. Collaborating with data and platform teams, you will integrate AI with AWS and Kubernetes infrastructure. This role stands out for its focus on autonomous agents and direct ownership of the full lifecycle.
Sage Recruiting
VerifiedSenior Backend Engineer, Agent-First Data Infrastructure
You will own the core systems that let AI agents run data workloads on object storage, rebuilding storage, compute, and governance from first principles. You'll join a small NYC-based team of systems builders, database researchers, and product designers, with authority over technical decisions and code you write daily. This role shapes the next generation of data platforms where the primary user is a swarm of AI agents, moving beyond human-centric design. 7+ years of distributed systems experience and hands-on Python or Rust is the bar, with tech-lead ownership and mentorship in scope.
360 Privacy
VerifiedStaff Software Engineer, Backend (Python AWS)
You own the backend systems and APIs that power our digital security platform, protecting high-risk individuals across the web. Python and AWS are your primary tools, and you make architectural decisions that shape our core infrastructure. You collaborate with cross-functional teams, mentor engineers, and drive engineering standards. This role offers a Principal trajectory, real ownership, and the chance to build systems that handle genuinely sensitive data.
Copart, Inc
VerifiedSenior DevOps Engineer, Kubernetes & AI/ML
You will own and operate Copart's DevOps platform for AI, machine learning, and agentic applications, serving a global auction network of over 750,000 buyers. Kubernetes, Docker, and Python are core to your daily work, with production support responsibilities across on-premises and cloud environments. You will partner with Data Science, Engineering, and Security teams to design and deploy scalable, secure infrastructure. This role stands out by letting you drive the internal AI stack, including custom agents and MLOps pipelines, from model selection to monitoring.
AgreeYa Solutions
VerifiedSoftware Engineer II Project Location: Los Angeles CA (Hybrid)
Own scalable backend services and internal platforms to support enterprise solutions. Lead development of AI-driven applications using generative AI models and agentic systems. Drive performance improvements across distributed systems and cloud infrastructure.