Senior Infrastructure Engineer - Middleware Operations - Kafka

Senior Infrastructure Engineer (Middleware Operations & Kafka)

About this role:

·         Wells Fargo is seeking a Senior Infrastructure Engineer to join the Middleware Operations team, with a primary focus on supporting and engineering enterprise Apache Kafka and Confluent Platform environments.

·         The role is responsible for designing, deploying, operating, securing, and optimizing large-scale messaging infrastructure. The successful candidate will combine strong hands-on Kafka expertise with production support, automation, observability, risk management, and stakeholder collaboration skills.

In this Role:

·         Design, deploy, upgrade, and operate enterprise-scale Apache Kafka and Confluent Platform environments.

·         Configure and manage KRaft controllers, brokers, Schema Registry, REST Proxy, Kafka Connect, ksqlDB, and Cluster Linking.

·         Deploy and operate Confluent Platform on OpenShift Container Platform using the Confluent Operator.

·         Provision and administer Kafka topics, ACLs, producers, consumers, partitions, retention policies, quotas, and replication settings.

·         Configure Kafka security using TLS, certificates, authentication, authorization, secrets management, and key rotation practices.

·         Monitor and optimize platform performance, including JVM garbage collection, heap utilization, partition distribution, ISR health, disk capacity, throughput, and consumer lag.

·         Troubleshoot production issues such as message loss or duplication, offset errors, broker failures, disk saturation, latency, throughput degradation, and consumer lag.

·         Design and maintain platform health dashboards and alerts using Prometheus, Grafana, Datadog, Splunk, or comparable enterprise monitoring tools.

·         Automate provisioning, monitoring, recovery, regression testing, and operational activities using Python, Bash, Ansible, or similar technologies.

·         Conduct capacity planning, performance benchmarking, and throughput optimization using tools such as KafkaPerf and kcat.

·         Support incident, problem, change, and recovery management for business-critical middleware services, including root-cause analysis and preventive remediation.

·         Develop infrastructure specifications, test solutions, support complex system upgrades, and contribute to application and infrastructure requirement validation.

·         Establish long-term solutions for recurring production issues and identify opportunities to improve availability, resilience, and recovery time.

·         Support risk and control initiatives by following technology policies, operating procedures, compliance requirements, and established work standards.

·         Collaborate with application teams, engineering teams, cybersecurity partners, product managers, and infrastructure stakeholders to resolve issues and deliver platform objectives.

·         Create and maintain technical documentation, standard operating procedures, recovery procedures, architecture diagrams, and operational runbooks.

·         Participate in an on-call support rotation and provide implementation support for production releases and critical risk initiatives.

Required Qualifications:

·         4+ years of Technology Infrastructure Engineering, Middleware Engineering, Site Reliability Engineering, or equivalent experience demonstrated through work, education, training, or military experience.

·         Minimum 4 years of hands-on experience supporting large-scale middleware environments, including Apache Kafka and Confluent Platform.

·         Strong production experience with Kafka brokers, KRaft, Schema Registry, REST Proxy, Kafka Connect, ksqlDB, and Cluster Linking.

·         Hands-on experience deploying and operating Kafka on OpenShift using the Confluent Operator.

·         Experience managing topics, ACLs, partitions, replication, retention, quotas, idempotent producers, and consumer groups.

·         Strong troubleshooting experience covering consumer lag, broker crashes, offset issues, message loss or duplication, disk saturation, latency, and throughput problems.

·         Experience implementing Kafka security using TLS, certificates, authentication, authorization, and enterprise key-management practices.

·         Experience configuring Prometheus and Grafana dashboards for platform monitoring and health reporting.

·         Proficiency in automation and scripting using Python, Bash, Ansible, or comparable technologies.

·         Experience supporting highly available, enterprise production environments.

·         Understanding of technology risk, control, compliance, and change-management practices within the Banking, Financial Services, and Insurance sector.

Desired Qualifications:

·         Experience with KRaft migrations and upgrades from ZooKeeper-based Kafka environments.

·         Knowledge of performance testing and benchmarking using KafkaPerf, kafkacat, or kcat.

·         Experience with Confluent Cloud, Amazon MSK, or Azure Event Hubs for Apache Kafka.

·         Knowledge of schema evolution, compatibility modes, and Avro, JSON, or Protobuf serialization.

·         Experience with additional messaging technologies such as Solace, IBM MQ, or RabbitMQ.

·         Understanding of enterprise networking concepts, including routing, BGP, multicast, load balancing, DNS, and certificate management.

·         Experience with capacity management, workload forecasting, and throughput optimization.

·         Experience with Prometheus, Grafana, Datadog, Splunk, or similar monitoring and observability platforms.

·         Confluent Certified Developer for Apache Kafka or Confluent Certified Administrator certification.

·         ITIL 4 Foundation or higher certification.

·         Job Expectations

·         Ability to provide production implementation and operational support outside standard business hours when required.

·         Ability to participate in an on-call rotation for critical incidents and service restoration.

·         Ability to work in a controlled enterprise environment with strong risk, security, audit, and documentation requirements.

·         Ability to communicate complex technical issues clearly to engineering teams, business partners, management, and control functions.

Posting End Date: 

*Job posting may come down early due to volume of applicants.

We Value Equal Opportunity

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.

Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.

Candidates applying to job openings posted in Canada: Applications for employment are encouraged from all qualified candidates, including women, persons with disabilities, aboriginal peoples and visible minorities. Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.

Applicants with Disabilities

To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo.

Drug and Alcohol Policy

Wells Fargo maintains a drug free workplace.  Please see our Drug and Alcohol Policy to learn more.

Wells Fargo Recruitment and Hiring Requirements:

a. Third-Party recordings are prohibited unless authorized by Wells Fargo.

b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.

c. Please be prepared to provide accurate information and supporting documentation in connection with any lawful employment and education verification conducted by Wells Fargo.