Anton Karamanov

Software Engineer

Contacts

Experience

2012 – 2013

Intern Software Developer

I joined Yandex as an intern at Statistics department, working on ad hoc analytics MapReduce scripts written in Perl and Python, which tought me:

  • MapReduce model and essentials of data processing and analytics;
  • basic design of MapReduce systems and APIs;
  • pragmatic programming approach with a tight feedback loop of writing and testing code, due to high costs of running scripts on a cluster;
  • effecient application of Linux command line tools (grep, sed, awk and other usual suspects) for basic local data processing;
  • love for terminals, vim and tmux.

After that I had briefly worked a chart editor web app for internal statistical reports publication platform. Editor allowed users to load, combine, process and visualize data from different stat reports. I was responsible for developing a JavaScript library for loading and processing report data and then converting it to representation accepted by HighCharts rendering engine, which gave me experince with developing user-facing APIs and essentials of user-friendly DSLs design. The library itself was written in CoffeeScript but exposed JavaScript API to the user and consisted of two primary parts:

  • DSLs for describing input data requirements and chart settings based on builder pattern;
  • data processing core written in Functional Programming paradigm which provided a set of useful combinators for transforming loaded data.

2013 – 2015

Junior Software Developer

Since 2014 I joined LogBroker project – data queue system based on Apache Kafka written in Scala. LB replaced an older implementation of data queue with the same api and provided features that Kafka lacked at the moment including exactly-once delivery guarantees, ACLs, exact data mirroring (preserving the same amount of partitions and order message within partition unlike Kafka's own mirror maker solution) between clusters in different geographical locations. LogBroker rapidly became an improtant pieces of company infrastructure, reaching hundreds of terabytes of throughput on a daily basis. It aggregated data from servers across the company, preprocesses it to unify format and delivered to the processing endpoints such as MapReduce clusters for batch processing or real-time log analytics systems.

During my involvement in the project I:

  • migrated core components to Akka actors to achieve scalability;
  • cleaned up service API written in Spray for coherence, consistency and ease of use;
  • developed a dynamic configuration which allowed to change settings at runtime by updating HOCON config file or special nodes in ZooKeeper;
  • developed an infrustructure for functional testing using Jenkins and Docker which automaticaly runs a full suite of tests for each PR;
  • wrote a tonn of unit and functional tests using ScalaTest;
  • developed a cluster management microservice from scratch, which was responsible for aggregating metrics and monitorings and deliverying them to downstream systems such as Graphite, reliably running distributed automated maintenance tasks such as Kafka cluster balancing and rolling restarts.

2015 – 2017

Middle Software Developer

LogBroker had become the de-facto standard for internal data transfer and spanned multiple data centers in different geographical regions with hundreds of nodes.

As a middle developer I had continued developement of important LogBroker features:

  • integrated distributed quotas library and developed ACLs support to deal with increased number of users migrating to the system;
  • developed a programmatic configuration declaration DSL and auto-generated validations using shapeless, which dramatically simplified introduction and support of new settings and effectively prevented all dynamic configuration mistakes which could lead to system failure due to human error;
  • developed a dedicated configuration microservice with HTTP API which enforced correctness of all applied settings;
  • migrated client offset storage from ZooKeeper to new Kafka API;
  • developed internal tools to simplify debugging and maintenance of the system;
  • developed a tool for automated Kafka cluster restart, which performs rolling cluster update, checking that cluster is healthy and balanced between node restarts;
  • helped develop a tool to check and restore Kafka cluster on failures leading to potential data loss due to log trancation on some replicas;
  • refactored a large portion of code, building a layer of abstraction over Kafka to simplify migration between versions and provide support for an alternative storage backend;
  • contributed to Akka, Spray and Kafka;
  • supported an in-house storage backend (developed by another team), which provided stronger persistence quarantees than Kafka.

2017 – Now

Confident Software Developer

Sure, "confident" is not a usual term, but let's call it that to introduce some distinction with "middle" :)

At the time of writing LB spans almost 1200 nodes across 5 data centers and accepts about 2.5 pebibytes and 112 billion messages a day (peaking at 40 gibibytes and 1.7 million messages per second) including mirroring between clusters, all of which passes through a write pipeline designed and implemented by me on top of Akka TCP.

  • completely migrated LB from Kafka to in-house backend;
  • migrated non-critical microservices APIs from Spray to Akka HTTP;
  • supported backpressure to handle increased system load including across all project subsystems, including legacy HTTP API, which required rewriting HTTP server from scratch directly on top of TCP to support asynchronous multiplexing of messages from thousand of connections on each cluster node in constant memory;
  • rewrote processes responsible for mirroring data between clusters and delivery to MapReduce clusters using "classic" implementation using threads and locks to Akka actors, leading to simpler code and improved throughput of both systems;
  • developed a centralized configuration management microservice for all project-related services which used event sourcing model to guarantee consistency, correctens, atomicity and detailed history of all configuration updates.

Education

2010 – 2014

Bachelor of Information Technology