Saturday, October 10, 2009

Looking for a passionate Systems Engineer !

Not a typical post, but I might as well use this platform to promote it. Come join the best startup ever !

kaChing is an investing talent marketplace where individual investors gain access to the best investing talent on the Web. An SEC registered investment adviser, kaChing enables customers to mirror automatically the trades of the "kaChing Geniuses" who have an Investing IQ of 140 or greater. The company's investors include Marc Andreessen, Jeff Jordan, CEO of Open Table (OPEN) and former president of PayPal, and retired partners from Benchmark Capital and Kleiner Perkins Caufield & Byers.

Join a stellar engineering team whose team members worked on massive backend infrastructure, Gmail's high performance frontend, the Native Scala Compiler with Prof. Odersky and more. We are test-driven and have a continuous integration process that lets us achieve a 5 minute build and release cycle.

Job description:

  • In charge of the backend of a revolutionary web site doing large scale high availability deployments and configuration management using open source software.
  • Keeping the system having a fully automated release cycle, continues testing and monitoring.
  • Build tools for monitoring, security, systems management and deployment if the available ones are not enough.
  • Help engineers architect the next levels of the system and making sure it stays agile and easily absorb high capacity.
  • Extremely smart and self motivated
  • Passion for test-driven development and automated functional QA
  • Live and breathe performance
  • Thinks out of the conventional box of IT management
  • Experience with databases, preferable MySQL
  • Good knowledge about monitoring and deployment tools
  • Good knowledge about network and distributed computing
  • Expert in at least one scripting language and know at least one more
  • Ability to work independently
  • A BS in computer science or related fields
I'll soon add a link to the official page.

Wednesday, October 07, 2009

Speeding up with Voldemort and Protobuf

In order to supply some of the analytics behind kaChing we have a some nice number crunching processes working over piles of financial data. The end results are displayed on the site (e.g. the find investors page).

The post is about one of such services which for the sake of this post I'll refer to as the SuperCruncher. In the first iteration we grabbed the data straight from the DB into the SuperCruncher. Needless to say, relational databases are not handling stress nicely and a pattern of fast iteration over all of the DB kills it very fast. Results of first iteration: 25 hours of computing.

Since we couldn't manage to squeeze more hours into the day we run a second iteration.
Voldemort & Protobuf for the rescue!
Though we have piles of data to compute, some of the data does not change though we do need to read it each time. We shoved the data into a Protobuf data structure which made the binary size considerably smaller, and pushed the protobuf into Voldemort. We used the Voldamort ProtoBufSerializer which provides an extremely simple way to store and use protobufs.
So essentially Voldemort in this case is used as a persistent cache to store normalized data. When reading the date the SuperCruncher first check it it exist in the naturally sharded Voldemort cluster and gets the delta from the DB.
Using Voldemort and Protobuf the I/O problem vanished, the SuperCruncher became four times faster (!) and bounced the performance ball back to the CPU. Cutting down CPU time is usually easier then cutting on I/O and indeed we managed to make the running time considerably faster in later iterations (hint: Joda Time & GC).

Shameless promo: kaChing is hiring!
We are looking for an world class Systems Engineer.

Our release cycle is 5 minute long, we release from the branch and have 100% tests passing on all time therefore we do not need a typical sysops person. We are looking for an excellent engineer to run, architect and automate the system.

Saturday, October 03, 2009

Scala @ Silicon Valley Code Camp

Had a great day at the Silicon Valley Code Camp. Gave couple of talks about Scala and met lots of interesting people. Here are the slides for the talks I gave.

Absorbing Scala Into Java Ecosystem

A First Look at Scala on Google App Engine

Friday, October 02, 2009

kaChing on Yahoo! Homepage

println("Hello kaChing!")

Creative Commons License This work by Eishay Smith is licensed under a Creative Commons Attribution 3.0 Unported License.