Principal Infrastructure Engineer

We are seeking an exceptional Principal Infrastructure Engineer to design, build, operate, and scale the platform that powers Sezzle. Your focus will be the hardest infrastructure problems: increasing throughput, reducing latency, removing capacity bottlenecks, strengthening resilience, and making production operations more automated and predictable as the business growsYou will own complex technical initiatives from architecture and prototyping through implementation, production rollout, and ongoing operation. Your impact will come from the systems you build, the problems you solve, and measurable improvements in reliability, performance, and cost efficiencyOur stack runs on AWS, with workloads orchestrated on Kubernetes and data anchored in Aurora RDS (MySQL and Postgres). You should know these technologies deeply and be comfortable moving between cloud architecture, networking, cluster internals, database performance, and application behavior to understand how the entire system scales. You will write code and infrastructure-as-code, debug production systems, and deliver changes that hold up under real traffic and failure conditionsOperational ownership is part of the job. You will participate in the on-call rotation and take a hands‑on role in recovering from major incidents, including full outages. We need someone who can form and test hypotheses using logs, metrics, and traces, make sound mitigation decisions with incomplete information, and turn incident findings into lasting engineering fixesYou will also build and apply AI‑assisted infrastructure and SRE tooling for incident investigation, capacity analysis, runbook automation, and toil reduction. You will evaluate these tools through practical results and apply appropriate access controls, validation, and auditability to their use in productionThis role reports to engineering leadership and works closely with application engineers, Security, and Compliance. You will develop a deep understanding of how Sezzle’s business operates and how customer journeys, transaction patterns, and product decisions shape infrastructure demand and behaviorYou will use that understanding to identify and deliver improvements across teams and technical domains, connecting infrastructure decisions to better customer outcomes and business performanceOwn the technical architecture and evolution of core infrastructure: identify system limits, prioritize technical improvements, and implement changes that support increasing traffic, data volume, and workload complexityConnect business understanding to improvements across the system: learn how key business workflows behave in production, trace their impact across applications, data, and infrastructure, and partner across teams to improve performance, reliability, and cost efficiency beyond any single service or team’s scopeEngineer for scale and performance: build capacity models, run load and stress tests, diagnose bottlenecks across compute, networking, Kubernetes, and databases, and validate improvements against throughput, latency, saturation, and cost per workloadDesign and build AWS infrastructure: implement resilient account, IAM, network, and service architectures; address service quotas, fault isolation, and multi‑AZ or multi‑region requirements as workloads growBuild and operate the Kubernetes platform: improve cluster architecture, lifecycle automation, workload isolation, resource allocation, autoscaling, safe upgrades, and deployment reliabilityScale and optimize Aurora RDS for MySQL and Postgres: tune queries and indexes, address connection and replication bottlenecks, plan capacity, improve failover behavior, and implement safe schema changes and database migrations with application engineersImprove reliability through engineering: define and instrument service‑level objectives and error budgets with service owners; implement failure isolation, backpressure, load shedding, and safe retry behavior where needed to prevent cascading failuresParticipate in on‑call and drive technical recovery during serious incidents: use evidence‑based triage, execute mitigations, communicate findings, and implement corrective actions from postmortemsImplement and test disaster recovery: design backup, restore, and failover mechanisms against agreed recovery time and recovery point objectives; run recovery exercises and document measured resultsBuild infrastructure‑as‑code and operational automation: make provisioning, configuration, deployments, upgrades, and recovery reproducible, reviewed, testable, and recoverable. Eliminate recurring manual work through codeBuild observability that makes production diagnosable: improve metrics, logs, traces, dashboards, and actionable alerts, with visibility into service health, scaling limits, and customer impactDeliver safe infrastructure migrations: design phased rollouts, compatibility checks, validation, and rollback paths for changes to shared production systemsImprove cloud cost efficiency through technical changes: right‑size resources, improve utilization, tune autoscaling and storage, and quantify savings while maintaining reliability and performance targetsBuild and evaluate AI‑assisted operational tooling: apply AI to investigation, runbooks, anomaly analysis, and repetitive operations, with bounded permissions, reviewable actions, and measurable improvements in accuracy or toilMake technical decisions clear and executable: write architecture proposals, evaluate technology tradeoffs through prototypes and benchmarks, review changes affecting shared infrastructure, and document how systems operate and failSezzle’s Technology Stack:Languages: Golang, PythonDatabase: MySQL, PostgresDevOps and Cloud: AWS, KubernetesVersion Control: GitCI/CD: GitlabOpen Source: Sezzle is focused on using open source, and we build what we can before buying!BenefitsComprehensive Benefit PlansGenerous Parental & Family LeaveCompetitive 401k MatchPaid Time Off & Volunteer Time OffOwnership Through Equity100% of Donations to Charity MatchedRemote Friendly CompanyHighly Discounted Fitness MembershipYou earn trust – you listen attentively, speak candidly, and treat others respectfullyYou measure results: you can explain the impact of your work in availability, latency, capacity, recovery time, cost, or hours of toil removedActive use of AI tooling in engineering or operations, with practical judgment about its limitations and how to verify generated code, recommendations, and operational actionsExperience implementing and testing disaster recovery against defined recovery objectives, including restoring data and validating service recoveryYou have backbone; disagree, then commit – you can respectfully challenge decisions when you disagree, even when doing so is uncomfortable or exhausting. You have conviction and are tenacious. You do not compromise for the sake of social cohesion. Once a decision is determined, you commit whollyDeep expertise with AWS: production experience across compute, IAM, multi-account architectures, and networking, including VPC design and private connectivityYou go deep: you investigate how systems behave under load and failure, follow the evidence, and fix underlying causesStrong systems fundamentals: Linux, networking, DNS, TLS, storage, concurrency, and distributed system failure modes, with the ability to debug problems across infrastructure and application boundariesPractical experience with observability, load testing, capacity planning, and safe CI/CD practices for shared production infrastructureStrong coding and automation skills, using Golang, Python, or similar languages to build production tooling and eliminate operational toil, alongside infrastructure-as-code experience with Terraform or equivalentExperience operating a 24/7, high‑availability platform where downtime has direct customer or revenue impact, including hands‑on incident response and postmortem remediationBachelor’s degree in Computer Science or a similar technical field (required)You earn trust: you listen carefully, communicate clearly, challenge technical decisions respectfully, and follow through on commitmentsDeep expertise with relational databases at scale, specifically RDS/Aurora (MySQL and/or Postgres): query performance, indexing, connection management, replication, high availability, failover, and verified backup and recoveryA track record of personally delivering infrastructure scaling improvements: identifying constraints, measuring baseline behavior, implementing changes, and demonstrating gains in capacity, latency, reliability, or cost efficiencyWillingness to participate in an on-call rotation and demonstrated ability to recover production systems under pressure using evidence-based triage, safe mitigation, and clear technical communicationDeep expertise with Kubernetes in production: cluster lifecycle, scheduling, resource management, autoscaling, networking, and troubleshooting business-critical workloads. EKS experience is strongly preferredAbility to carry ambiguous technical problems from investigation through production delivery and collaborate across engineering disciplines to resolve system-wide constraintsYou’re not bound by convention – your success—and much of the fun—lies in developing new ways to do thingsYou deliver results – you focus on the key inputs and deliver them with the right quality and in a timely fashion. Despite setbacks, you rise to the occasion and never settleYou build and ship: you turn architecture into working code, tested infrastructure, and safe production changes12+ years of experience across infrastructure, platform, site reliability, software development, or related engineering disciplines, with substantial hands‑on depth designing and operating production infrastructure at scaleYou need action – speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk‑takingYou have relentlessly high standards – many people may think your standards are unreasonably high. You are continually raising the bar and driving those around you to deliver great results. You make sure that defects do not get sent down the line and that problems are fixed so they stay fixedYou value simplicity: you choose systems that are understandable, operable, and appropriate to the problem, and reduce unnecessary complexityYou stay accountable for the outcome: you verify that changes work in production and that fixes remain effective as the platform scalesProficiency with Prometheus, Grafana, Loki, Tempo, or comparable observability systemsExperience in fintech, payments, or banking, operating infrastructure with demanding reliability, security, and audit requirementsExperience with multi-region architectures, chaos engineering, and failure testing, including the consistency and recovery tradeoffs of distributed data systemsExperience building AI-assisted incident investigation or operational automation with restricted access, auditable execution, and clear human review pointsExperience building internal platform capabilities and self‑service tooling, including deployment automation, progressive delivery, and reusable infrastructure components
#J-18808-Ljbffr