What we need
Fairsearch is looking for a Site Reliability Engineer with deep knowledge of Linux and distributed systems to help take care of its distributed crawler and ensure all systems are up and running 24/7. Working experience with bare-metal servers and ability to participate in daily on-call rotation are required.
Our system is big part custom OCaml code and also employs third-party technologies - Debian, ELK, Puppet, and anything else that will solve the task at hand. In this role, be prepared to deal with 25 petabytes storage cluster, 2,000 bare-metal servers, experimental large-scale deployments and all kinds of software bugs and hardware deviations on a daily basis.
If you possess a healthy desire to automate everything while being able to quickly resolve urgent issues manually, then we want you! We strive to keep humans away from doing repetitive jobs that can be done by computers and focus instead on foreseeing problems and defining programmatic means to handle them.
Deep understanding of operating systems and networks fundamentals
Practical knowledge of Linux userspace and kernel internals
Working experience with bare-metal servers
Participation in on-call rotation (6 hours every weekday + one weekend per month)
The ideal candidate is expected to:
Understand the whole technology stack at all levels: from network and user-space code to OS internals and hardware
Independently deal with and investigate infrastructure issues on live production systems including dealing with hardware problems and interact with datacenters
Develop internal automation - monitoring, setup, statistics
Have the ability to foresee potential problems and prevent them from happening. Apply first-aid reaction to infrastructure failures when necessary
Help developers with deployment and integration
Make well-reasoned technical choices and take responsibility for it
Approach problems with a practical mindset and suppress perfectionism when time is a priority
Setup automatic systems to control infrastructure
Possess a healthy detestation for complex shell scripts
Be ready to work in a small team and take responsible independent decisions