Rotem Hermon

Is It My Project?

2026-03-17T00:00:00+00:00

In my last post I introduced my side project, BEAR - a tool that tries to show how agentic development may be done in a safer way, by creating boundaries for agents as they develop, and allowing humans to be alerted when agents try to expand these boundaries.

But I want to focus on something else in my project. I want to focus on the “my”.

Because while “I” was “developing”, I kept asking myself - is it “my” project?

This new world of agent-assisted development creates some discomfort for those of us who still remember the time when we wrote code by hand. But it’s more than writing code. The whole process - brainstorming, designing, defining tasks, managing the roadmap, all of it was done using my team of agents.

Now, it’s not that this process is new to me. I have been working for years as a software architect, so my day-to-day has for a long time been about talking to people and not writing code. This is what I do. I talk to people to brainstorm, design, manage the roadmap, plan, and review implementation.

So you can say - people, agents, what’s the difference?

But somehow, there is a difference.

With a team of humans, there is shared ownership of the product. It’s not “mine”, it’s “ours”. But what is this ownership when your coworkers are a team of agents? When there isn’t really a “they” in the sense that we are used to thinking about what “they” is?

This is beyond the simple question of ownership, of how much you actually understand the code, how much you know what’s really going on under the hood. That’s an important and practical question that teams will have to tackle as agents start to write more and more code.

I’m talking about a more existential question. About that boundary of what is human, what is creativity, and how they correlate when creation is done with so much automation by a non-human.

Those boundaries are getting blurred, and it feels like we need to calibrate ourselves again, redefine how we perceive what we do, what we own, and what it means when, in that human-nonhuman collaboration, the non-human part is getting bigger.

I started with a project that tried to understand and create boundaries for agent development, but it ended up questioning my own boundaries, and the boundary between me and the machine.

BEAR: Structural Boundaries for Agentic Development

2026-03-08T00:00:00+00:00

In the last few posts I’ve been thinking about how development might change when agents can write large amounts of code very quickly. If generating implementation becomes easy, the constraint in development shifts. The interesting question is no longer how we produce code, but where control should live when a system can evolve at that speed.

One thing that kept coming to mind is that many of the most dangerous changes in a system are not ordinary code edits. They are structural changes. Accessing something new, dependency changes, side effects. These are important architectural changes, but they often get lost in a pull request, especially when an agent can introduce many changes in the same PR.

That made me wonder what would happen if the important architectural structure of a system was described explicitly at a level above the generated code.

I started thinking about describing a system as a set of blocks. Each block represents an architectural unit with clearly defined boundaries. It declares what it can access, what it exposes, and what dependencies it is allowed to have. Inside those boundaries, development can move quickly. Agents can generate code, refactor it, and evolve it freely, but the boundaries themselves remain visible and enforceable.

It’s a bit like putting a bear in a cage. Inside the cage the bear can move however it wants, but the cage still defines the limits of where it can go.

I started experimenting with this idea in a small project called BEAR - Block Enforceable Architectural Representation. The idea is that an agent starts from a small representation of the application as a set of blocks, and then works within those boundaries. BEAR CLI generates deterministic checks from it, and CI can use BEAR to surface structural changes. If a pull request expands the authority of a component, that expansion becomes visible rather than hidden inside ordinary code changes, and a human-in-the-loop review can be triggered.

This is still very much an experiment, but the project and how it works are available here: BEAR on GitHub

As development velocity continues to increase, it seems likely that some form of explicit structural boundary layer will become necessary.

BEAR is a small attempt to explore what that layer might look like.

Not All Friction Is the Same

2026-03-05T00:00:00+00:00

Lately there are many posts celebrating the death of friction, praising how AI removes the friction of writing code and increases development velocity.

But is friction always bad?

From my current personal perspective, for example, the friction a shelter wall provides when a rocket hits it is good friction.

But let’s take a less sensitive example.

When Git came along, it brought a real change to how we manage source code. Anyone could create branches locally. Branching became trivial and cheap, merging became easy. Multiple developers could work on the same code in parallel. A lot of friction that central version control systems used to impose was removed. But that also increased the velocity at which changes could reach the main codebase.

To regain visibility and control over those changes, we introduced a new intentional point of friction: Pull Requests.

With PRs, a change becomes visible before it is merged. Someone else reviews it. We can discuss it. CI gates can run. PRs deliberately slow the final step so that we regain visibility and control.

So not all friction is the same.

Some friction slows us down because of effort: writing boilerplate code, setting up environments, repetitive work. Removing this friction improves velocity.

But there is also friction that we introduce intentionally to maintain control, safety, or correctness. Things like PRs, deployment gates, controlled releases, feature flags, and even type checks.

The interesting thing is that when “bad” friction disappears and velocity increases, we often need to introduce new “good” friction so things don’t get out of control.

And that’s what we’re witnessing now with AI. The friction of writing code is being reduced dramatically, and development velocity increases accordingly. But the risks don’t disappear. So we now need to find ways to introduce new and refined friction that allows us to move at this new speed with regained visibility and confidence.

Abstraction Has Never Removed Structure

2026-03-01T00:00:00+00:00

When we look at the history of programming language revolutions, there’s a consistent pattern. Every major shift in programming increased expressiveness, but it also introduced new structure and constraints. Each shift created a new level of abstraction. We could express what we wanted more clearly, with a larger vocabulary, but always under formal rules.

But this shift feels different. For the first time, our abstraction layer is natural language. In previous transitions, we introduced new programming languages to support the new abstraction, but they remained formal and deterministic. Natural language is the most expressive language we have, yet it is not formal or deterministic. It is open-ended and interpretive. That changes the nature of the abstraction itself.

Natural language is a powerful tool. The fact that you can describe what you want, almost as you would to another person, and see it built, is a real leap. It works especially well in contexts where ambiguity is acceptable and iteration is fast. But in systems that require precision and high confidence, ambiguity can become a liability.

I think we may see a divergence in how language is used for software development. For many projects, natural language may be sufficient. But history suggests that when expressiveness rises, constraint follows. In high-precision systems, structure will need to reappear.

I see two levels where this might happen:

First, the specification layer. We may need a more formal representation of intent. Not just free-form requirements, but structured ways to describe what a system must do, or to review how natural language was translated.

Second, the structural boundary layer. As code generation becomes more fluid and high velocity, boundaries matter more. We will need clearer definitions of what parts of a system are allowed to access or modify other parts, so critical behavior does not change without being visible.

Abstraction has never removed structure. Structure needs to reappear. The question is where.

Not Every Coding Is a Vibe

2026-02-22T00:00:00+00:00

Look, vibe coding is great. But not every coding is a vibe.

Coming from years of building products and backend services that provide critical business functionality for high-profile and demanding customers, you can’t “vibe” this. Working on these kinds of systems requires real confidence. And while LLMs are great, they’re still non-deterministic. For high-scale, high-performance, business-critical services, you need determinism somewhere in the loop.

From where I’m standing, vibe coding is perfect for personal projects, internal tools, even a lot of B2C-type applications. But in environments that demand reliability, accountability, and strong non-functional guarantees, vibe coding alone isn’t enough.

You need agentic development. And that means serious tooling. Tooling that makes sure agents write code within clear boundaries, that drift is visible, that non-functional requirements are enforced, and that architectural decisions aren’t silently overridden.

There’s a whole layer of infrastructure being built right now to support this shift. I think that’s what will really drive agentic development into the more demanding parts of the industry.

Ethical Questions In Software Engineering

2018-12-06T00:00:00+00:00

Software, as a technology, is now underlying almost every aspect of our world. Computers and software are part of nearly every area we can think of. And this means that software engineering is practically becoming a practice that is relevant in everything we do. Or, if we want to be more dramatic, we can say that software engineering is the most influencing trade existing today.

It is therefore quite alarming that no part of the professional education of software developers deals with the ethical, social, political and cultural consequences of what they - we - do. We are trained in technology. We study how to build software, how to write algorithms. But technology is not the end goal - it is the means to an end. And no one teaches us that. No one tells us to stop and think about what we’re producing, about how it affects the world around us, how it influences people’s lives.

I believe we’re at a point where this becomes critical. We must raise awareness of the importance of ethics in technology. We need to educate developers not only to be good developers but also to be conscious and critical of the implications of what they release into the world. If we want to sustain a healthy society we don’t really have a choice.

My small contribution to this subject was this talk, given at the O’Reilly Software Architecture Conference in London (October 2018). I hope that those of us who care about it will keep writing and speaking and stressing the importance of changing the way we think about the part of ethics in our work and in the way we train developers. With enough voices, we will be able to shift the industry to a better and more conscious place.

Is Emotionally Intelligent AI possible?

2017-10-11T00:00:00+00:00

I came across this recent post by Mikko Alasaarela - The Rise of Emotionally Intelligent AI. I’m re-posting here my comment to it.

In his post, Mikko Alasaarela contemplates about AI and emotional intelligence. He claims that AI is moving fast towards being emotionally intelligent, in a way that might be better and less biased than what we humans employ.

It’s an interesting view, but there is a base hypothesis in Mikko’s post that is not too solid:

Our emotions and feelings are organic algorithms that respond to our environment. Algorithms, that are shaped by our cultural history, upbringing and life experiences. And they can be reverse engineered.

Mikko mentions some supporters of this concept of “organic algorithms”, like Yuval Noah Harari and Max Tegmark.

The view of human cognition as an “algorithm”, or in other words, the computational model of cognition, has gained a lot of traction in the last decade. It’s not very surprising. We are, after all, in the age of algorithms.

But that is also the problem. We’re using a metaphor, something we know, to describe something we don’t know. That’s our modern lens, we see through the software/algorithm prism. And it’s quite a good metaphor. It’s useful for a lot of things. It explains things. But then we tend to forget it’s a metaphor.

There is no real evidence that human cognition is a set of “organic algorithms”. More over, there’s no evidence to support the assumption that it’s based on what we call algorithms; algorithms that can be reversed engineered by us, that is, can be expressed in some algorithmic notation that we can produce and reproduce. There is so much unknown regarding consciousness, that to make this statement is really kind of absurd.

And there is another thing overlooked. Our emotional intelligence is largely based on, or connected to, our ability for empathy. We feel what others feel. But we have no clue how to reproduce that, because we have no clue what it means. We know what it’s like, to feel. We don’t know what it means. We don’t know what causes that. We don’t know what is cognition. We don’t know what makes a subjective experience. We have the experience, we have not been very successful in understanding it.

So we have no reason to believe an algorithm can reproduce that. At least not until we have some kind of understanding what this that really is.

How we built our own .NET microservices framework (and made it open source along the way)

2017-08-22T00:00:00+00:00

There are some things in life that everyone tells you not to do.
Sometimes you still do them.
One of these things is building your own framework. (Don’t you just hate it when your parents tell you again and again - don’t build your own framework?).

Well, we have some excuses.

When we (at Gigya) embarked on our microservices journey two years ago, the .NET landscape was somewhat barren. (Why .NET? We had a lot of application code in .NET we wanted to use and reuse. Our developers where .NET people. We had a lot of experience already. Moving away from that was not really a reasonable option).
Java and other common languages had multiple libraries, frameworks and technologies for building microservices. But the .NET open source universe was not really expanding (there were some interesting discussions about that). So in a way, we didn’t have too much of a choice.

But the nice thing about being “forced” to make such a choice, is that it makes the journey very interesting. It gives you a chance to learn what options are out there, how people approach the problem of microservices and what various design decisions they make. It allows you to select what fits your own use case and constraints. It also introduces multiple never ending discussions… That eventually you need to end.

One of the essential questions was a meta-discussion about the direction of our path. Do we need a microservices framework at all? Alternatively we can define a set of basic rules (HTTP transport, some common practices) and let everyone take their own approach (and maybe technology) in writing their microservice.

This polyglot approach is valid, and is considered sometimes one of the benefits of a microservices environment. But there’s overhead to consider when working in multiple languages, technologies or design choices. Both in terms of operations (deployment, monitoring, infrastructure) and of people and knowledge. As a not-too-big development organization, this was an overhead we didn’t want to introduce at the time.
We decided to have a common framework for our microservices that will dictate some of the basic patterns of how we see our microservices working. Solve the common problems in a single place, and let everyone reuse it. That was our goal.

Another reason was a non-trivial technology choice we took as the common ground of our new microservices. This was a new technology that was just open sourced by Microsoft Labs - Orleans, a Virtual Actors framework. I won’t go too much into Orleans here (I talked about it on other occasions). Since we are handling high loads of state-sensitive traffic, we believed Orleans and the Virtual Actors model gives a good way of handling problems of distributed concurrency that we knew we need to handle. Orleans gave us a productive and developer friendly way of building such high scale services. We wanted our microservices framework to play nicely with Orleans and solve the common hosting and deployment scenarios, so that putting up a new Orleans base microservice will be a common and simple routine. Hopefully we managed to do that.

Another organizational question was how to manage the development of the framework. Will it be a common library that all teams work on and improve as they go along? Or will we have a dedicated team that will be responsible for this common framework? There are pros and cons for each approach. We preferred clear ownership and focusing the effort. So we created a dedicated infrastructure team that is responsible for the development of the common microservices framework.
It was not a smooth process, of course. It creates a situation of dependencies between teams (application teams might need to wait for a feature to be developed in the infrastructure). But we tried to put in place processes to identify needs and plan accordingly. We also encouraged an internal “open source” approach, where each team can make changes to the common framework on its own and create a pull request that is reviewed by the infrastructure team. And as the framework matured and stabilized, those inter-dependency issues became less problematic.

The end (“end” might not be the right terminology, it’s still a work in process) result is a pretty robust framework with some interesting features. We selected an RPC-like approach for our inter-service communication, modeling it after Orleans own inter-grain communication pattern. We added transparent client side response caching to accelerate responses when possible. We support distributed tracing, client side load balancing, health checks, a hierarchical file based configuration system, and more. Our framework implements a lot of the patterns that are needed for building up microservices.

Since there are less than a handful open source frameworks for microservices in .NET, we thought that our own home grown framework might be a nice addition. We believe its feature set is rather unique, and it is production proven.
But getting it to be ready to open source took some time. First, you need to get buy-in from the developers working on it. It took time and the maturity of the framework until people agreed that this is something that is worth putting out.
Then there’s some time investment needed. We had to refactor and make the code more modular so we can take out internal things and make the core framework general purpose and generic. Make our internal framework use the open source version. Write some documentation. We allocated time and our developers did a great job making it ready for open source.

It was a nice moment switching the privacy setting on the repository in github, making Microdot open to the world - https://github.com/gigya/microdot.

Cross-type joins in Elasticsearch

2014-12-31T00:00:00+00:00

When modeling data in Elasticsearch, a common question is how to design the data to capture relationships between entities, to allow at least some level of “joins”.

Elasticsearch has a good guide about data modeling. One of the options provided for expressing relationships is the parent-child model.

A parent-child relationship in Elasticsearch is a way to express a one-to-many relationship (a parent with many children). The parent and child are separate Elasticsearch types, bounded only by specifying the parent type on the child mapping, and by giving the parent ID for every child index operation (this is used for routing the child to the shard of the parent).

It’s a useful model when a parent has many children and when the child update pattern is different from that of the parent. (Since every child is a separate document, updating the child does not require re-indexing the parent).

But this model also provides an interesting (if limited) way to capture relationships between sibling types.

Lets consider the following data:

Bill has two children - Adam and Eve, and a Dog (Apple).
Bob has no children or pets (ah, freedom!).
Mary has a little newborn child called Lamb.
Jane has a boy named Xander, a cat (Buffy) and a dog (Willow).

Lets create this data in Elasticsearch.
We will have a parent type - “person”, and two child types - “children” and “pets”.
First we’ll create the mapping for the child types.

    #!/bin/bash
    
    export ELASTICSEARCH_ENDPOINT="http://localhost:9200"
    
    # Create indexes
    
    curl -XPUT "$ELASTICSEARCH_ENDPOINT/es-joins" -d '{
        "mappings": {
            "children": {
                "_parent": {
                    "type": "person"
                }
            },
            "pets": {
                "_parent": {
                    "type": "person"
                }
            }
        }
    }' 

Next, index all the documents - parents, children and pets.

    # Index documents
    curl -XPOST "$ELASTICSEARCH_ENDPOINT/_bulk?refresh=true" -d '
    {"index":{"_index":"es-joins","_type":"person","_id":1}}
    {"name":"Bill","gender":"male"}
    {"index":{"_index":"es-joins","_type":"person","_id":2}}
    {"name":"Bob","gender":"male"}
    {"index":{"_index":"es-joins","_type":"person","_id":3}}
    {"name":"Mary","gender":"female"}
    {"index":{"_index":"es-joins","_type":"person","_id":4}}
    {"name":"Jane","gender":"female"}
    {"index":{"_index":"es-joins","_type":"children","_parent":1,"_id":1}}
    {"name":"Adam","gender":"male"}
    {"index":{"_index":"es-joins","_type":"children","_parent":1,"_id":2}}
    {"name":"Eve","gender":"female"}
    {"index":{"_index":"es-joins","_type":"children","_parent":3,"_id":3}}
    {"name":"Lamb","gender":"male"}
    {"index":{"_index":"es-joins","_type":"children","_parent":4,"_id":4}}
    {"name":"Xander","gender":"male"}
    {"index":{"_index":"es-joins","_type":"pets","_parent":1,"_id":1}}
    {"name":"Apple","type":"dog"}
    {"index":{"_index":"es-joins","_type":"pets","_parent":4,"_id":2}}
    {"name":"Buffy","type":"cat"}
    {"index":{"_index":"es-joins","_type":"pets","_parent":4,"_id":3}}
    {"name":"Willow","type":"dog"}
    '

Now we can do some searches on it.
The usual example will be searching a parent by its children. Lets find all the parents that has a girl. We expect to get back only Bill.

    curl -XPOST "$ELASTICSEARCH_ENDPOINT/es-joins/person/_search?pretty" -d '
    {
        "query": {
            "filtered": {
                "filter": {
                    "and": [
                        {
                            "has_child": {
                                "type": "children",
                                "query": {
                                    "term": {
                                        "gender": "female"
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
    '

We can also combine conditions on multiple child types.
Lets find parents that have a boy and a dog. This time we expect to get back both Bill and Jane.

    curl -XPOST "$ELASTICSEARCH_ENDPOINT/es-joins/person/_search?pretty" -d '
    {
        "query": {
            "filtered": {
                "filter": {
                    "and": [
                        {
                            "has_child": {
                                "type": "children",
                                "query": {
                                    "term": {
                                        "gender": "male"
                                    }
                                }
                            }
                        },
                        {
                            "has_child": {
                                "type": "pets",
                                "query": {
                                    "term": {
                                        "type": "dog"
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
    '

Another commonly used option is finding children by their parents.
But a more interesting possibility is finding children by their siblings.
Lets lookup all boys that have a dog. To do that we’re searching on the “children” type, and doing a has_parent filter that contains a has_child filter on the “pets” type.
This time we expect to get back the children - Adam and Xander.

    curl -XPOST "$ELASTICSEARCH_ENDPOINT/es-joins/children/_search?pretty" -d '
    {
        "query": {
            "filtered": {
                "filter": {
                    "and": [
                        {
                            "has_parent": {
                                "parent_type": "person",
                                "filter": {
                                    "has_child": {
                                        "type": "pets",
                                        "query": {
                                            "term": {
                                                "type": "dog"
                                            }
                                        }
                                    }
                                }
                            }
                        },
                        {
                            "term": {
                                "gender": "male"
                            }
                        }
                    ]
                }
            }
        }
    }
    '

Of course, our data model here is a bit simplified as it allows only a single parent. If we were to extend it, we would create a “family” parent type, with child types - “parents”, “children” and “pets”.

Currently, in order to get the details of the “joined” entity, another query is needed. For example, when searching “all boys that have a dog”, if we want the details of the dogs we need a second search for “all dogs with parents that have children with _id=…” (and the _ids of the children from the first search).
This will change with the new upcoming inner hits feature that will allow getting the data of the inner entities in a single query.

One should note that this method is not exactly recommended by Elasticsearch. Because of the memory requirements and performance hit, the official recommendation is: “Avoid using multiple parent-child joins in a single query”. So as always, test, measure and choose your modeling wisely.

Pub-Sub messaging with Zookeeper

2014-04-10T00:00:00+00:00

When building a distributed service, there are cases where you need to broadcast messages to all nodes running your service.
For example, if your service holds a cache of items in memory, and an operation on one of the nodes can invalidate an item, you need to notify all of the nodes to remove that item from the cache.

These types of notifications are a good use case for a Pub/Sub messaging service, which usually runs on top of some kind of a message queue.

But if your service is coordinated via zookeeper, it can make sense to utilize zookeeper also for managing the messages between the nodes, instead of incorporating and managing a full blown message queue. In this post I’ll present such an option.

For those impatient - zkms is a ready to use scala library that implements the concept presented in this post. It’s available on github and as a Maven dependency (instructions on the project page).

This comes with a big warranty note attached:

Zookeeper is not built with this kind of use case in mind, and as stated in this Netflix remark - “ZooKeeper makes a very bad Queue source”.
So this solution can be valid on a medium sized cluster (a few dozens of nodes) and a rather low message rate. It should probably not be used when high throughput or large number of nodes are expected.

General Concept

Since zkms comes to answer a specific need (“cache revocation” type of notifications), it follows a few design guidelines:

Topic based Pub/Sub.
A message is broadcasted to a single topic.
A consumer can subscribe to multiple topics.
Messages are sent to online nodes only.
Messages are not persisted and cannot be played-back.
You cannot send messages to a topic if there are no subscribers for that topic.

For subscribing to messages, a consumer registers itself to a topic by creating an ephemeral node under the topic name.
For sending messages, the producer gets all the consumers that subscribed to the topic, and places the message under each of the subscribers queue.

The details

The library uses 3 top zNodes:

/clients - holds the list of connected consumers.
/subscribers - holds the lists of subscribed consumers per topic.
/messages - hold the lists of messages for each consumer.

Another zNode is used as a leader selection path for the cleaner. More on that later.

Subscribing

Subscribing to messages under a topic consists of the following steps:

The consumer creates an ephemeral node with its consumer ID under /clients. This is used by the cleaner to remove message queues for disconnected consumers.
The consumer creates an ephemeral node with its consumer ID under /subscribers/[topic].
The consumer creates a watcher on /messages/[consumerID]/[topic]. The producer will place the topic messages under this node.

Unsubscribing from a topic means deleting the topic messages zNode and the subscription zNode for the consumer.

Broadcasting

To broadcast a message to a topic the producer:

Gets all the child nodes of the topic subscribers path (/subscribers/[topic]). If there are no subscribers, an error is returned.
For each subscriber, create a sequential persistent node under its message queue node (/messages/[consumerID]/[topic]). The node value is the message.

The Cleaner

Each consumer generates its random consumer ID when it’s created. Since consumers can go up and down frequently, and since the consumer messages node cannot be ephemeral (ephemeral nodes are not allowed to have children), we can end up with a lot of messages zNodes for dead consumers that are no longer valid.

To handle that, one of the zkms instances (either producer or consumer) functions as a cleaner (it’s elected via a leader selection recipe).

The cleaner wakes up every now and again, and looks for messages zNodes that do not have a corresponding zNode under the /clients path (since the consumer registers an ephemeral node there, it will be gone when it gets disconnected). It deletes (recursively) message nodes with no connected consumers.

Performance

Since broadcasting messages means creating a zNode for every consumer which is subscribed to the topic, the performance of the publisher degrades linearly with the number of topic subscribers.

In a not very accurate test conducted on my local development machine, a single publisher managed to push around 1000 messages per second to a topic with a single subscriber. When increasing the number of subscribers to 10 the message rate went down to ~250/sec. With 20 subscribers it was ~125/sec.

So the warning is in order again - this solution will work when there are not too many subscribers and a reasonable message rate. It will not scale! Use with care.

Building a social music service - the technology behind serendip.me

2014-03-10T00:00:00+00:00

During 2011-2013 I had the pleasure of working as Chief Architect for serendip.me, helping to build it from the ground up together with Sagee and Asaf, Serendip’s founders.
I wanted to share some of the experience of building serendip’s backend - the technologies we used, architecture and scaling considerations - since personally I always find it very insightful to hear about real-life start-up experiences, especially when trying to create a new venture.

serendip.me is a social music service that helps people discover great music shared by their friends, and also introduces them to their “music soulmates” - people outside their immediate social circle that shares a similar taste in music.

Serendip is running on AWS and is built on the following stack: scala (and some Java), akka (for handling concurrency), Play framework (for the web and API front-ends), MongoDB and Elasticsearch.

Choosing the stack

One of the challenges of building serendip was the need to handle a large amount of data from day one, since a main feature of serendip is that it collects every piece of music being shared on Twitter from public music services. So when we approached the question of choosing the language and technologies to use, an important consideration was the ability to scale.

The JVM seemed the right basis for our system as for its proven performance and tooling. It’s also the language of choice for a lot of open source system (like Elasticsearch) which enables using their native clients - a big plus.
When we looked at the JVM ecosystem, scala stood out as an interesting language option that allowed a modern approach to writing code, while keeping full interoperability with Java. Another argument in favour of scala was the akka actor framework which seemed to be a good fit for a stream processing infrastructure (and indeed it was!). The Play web framework was just starting to get some adoption and looked promising. Back when we started, at the very beginning of 2011, these were still kind of bleeding edge technologies. So of course we were very pleased that by the end of 2011 scala and akka consolidated to become Typesafe, with Play joining in shortly after.

MongoDB was chosen for its combination of developer friendliness, ease of use, feature set and possible scalability (using auto-sharding). We learned very soon that the way we wanted to use and query our data will require creating a lot of big indexes on MongoDB, which will cause us to be hitting performance and memory issues pretty fast. So we kept using MongoDB mainly as a key-value document store, also relying on its atomic increments for several features that required counters.
With this type of usage MongoDB turned out to be pretty solid. It is also rather easy to operate, but mainly because we managed to avoid using sharding and went with a single replica-set (the sharding architecture of MongoDB is pretty complex).

For querying our data we needed a system with full blown search capabilities. Out of the possible open source search solutions, Elasticsearch came as the most scalable and cloud oriented system. Its dynamic indexing schema and the many search and faceting possibilities it provides allowed us to build many features on top of it, making it a central component in our architecture.

We chose to manage both MongoDB and Elasticsearch ourselves and not use a hosted solution for two main reasons. First, we wanted full control over both systems. We did not want to depend on another element for software upgrades/downgrades. And second, the amount of data we process meant that a hosted solution was more expensive than managing it directly on EC2 ourselves.

Some numbers

Serendip’s “pump” (the part that processes the Twitter public stream and Facebook user feeds) digests around 5,000,000 items per day. These items are passed through a series of “filters” that detect and resolve music links from supported services (YouTube, Soundcloud, Bandcamp etc.), and adds metadata on top of them. The pump and filters are running as akka actors, and the whole process is managed by a single m1.large EC2 instance. If needed it can be scaled easily by using akka’s remote actors to distribute the system to a cluster of processors.

Out of these items we get around 850,000 valid items per day (that is items that really contains relevant music links). These items are indexes in Elasticsearch (as well as in MongoDB for backup and for keeping counters). Since every valid item means updating several objects, we get an index rate of ~40/sec in Elasticsearch.
We keep a monthly index of items (tweets and posts) in Elasticsearch. Each monthly index contains ~25M items and has 3 shards. The cluster is running with 4 nodes, each on a m2.2xlarge instance. This setup has enough memory to run the searches we need on the data.

Our MongoDB cluster gets ~100 writes/sec and ~300 reads/sec as it handles some more data types, counters and statistics updates. The replica set has a primary node running on a m2.2xlarge instance, and a secondary on a m1.xlarge instance.

Building a feed

When we started designing the architecture for serendip’s main music feed, we knew we wanted the feed to be dynamic and reactive to user actions and input. If a user gives a “rock-on” to a song or “airs” a specific artist, we want that action to reflect immediately in the feed. If a user “dislikes” an artist, we should not play that music again.
We also wanted the feed to be a combination of music from several sources, like the music shared by friends, music by favorite artists and music shared by “suggested” users that have the same musical taste.
These requirements meant that a “fan-out-on-write” approach to feed creation will not be the way to go. We needed an option to build the feed in real-time, using all the signals we have concerning the user. The set of features Elasticsearch provides allowed us to build this kind of real-time feed generation.

The feed algorithm consists of several “strategies” for selecting items which are combined dynamically with different ratios on every feed fetch. Each strategy can take into account the most recent user actions and signals. The combination of strategies is translated to several searches on the live data that is constantly indexed by Elasticsearch. Since the data is time-based and the indexes are created per month, we always need to query only a small subset of the complete data.
Fortunately enough, Elasticsearch handles these searches pretty well. It also provides a known path to scaling this architecture - writes can be scaled by increasing the number of shards. Searches can be scaled by adding more replicas and physical nodes.

The process of finding “music soulmates” (matching users by musical taste) is making good use of the faceting (aggregation) capabilities of Elasticsearch. As part of the constant social stream processing, the system is preparing data by calculating the top shared artists for social network users it encounters (using a faceted search on their shared music).
When a serendip user gives out a signal (either by airing music or interacting with the feed), it can trigger a re-calculating of the music soulmates for that user. The algorithm finds other users that are top matched according to the list of favorite artists (which is constantly updated), weighing in additional parameters like popularity, number of shares etc. It then applies another set of algorithms to filter out spammers (yes, there are music spammers…) and outliers.
We found out that this process gives us good enough results while saving us from needing additional systems that can run more complex clustering or recommendation algorithms.

Monitoring and deployment

Serendip is using ServerDensity for monitoring and alerting. It’s an easy to use hosted solution with a decent feature set and reasonable pricing for start-ups. ServerDensity natively provides server and MongoDB monitoring. We’re also making heavy use of the ability to report custom metrics into it for reporting internal system statistics.
An internal statistic collection mechanism collects events for every action that happens in the system, and keeps them in a MongoDB collection. A timed job reads those statistics from MongoDB once a minute and reports them to ServerDensity. This allows us to use ServerDensity for monitoring and alerting Elasticsearch as well as our operational data.

Managing servers and deployments is done using Amazon Elastic Beanstalk. Elastic Beanstalk is AWS’s limited PaaS solution. It’s very easy to get started with, and while it’s not really a full featured PaaS, its basic functionality is enough for most common use cases. It provides easy auto-scaling configuration and also gives complete access via EC2.
Building the application is done with a Jenkins instance that resides on EC2. The Play web application is packaged as a WAR. A post-build script pushes the WAR to Elastic Beanstalk as a new application version. The new version is not deployed automatically to the servers - it’s done manually. It is usually deployed first to the staging environment for testing, and once approved is deployed to the production environment.

Takeaways

For conclusion, here are some of the top lessons learned from building serendip, not by any special order.

Know how to scale. You probably don’t need to scale from the first day, but you need to know how every part of your system can scale and to what extent. Give yourself enough time in advance if scaling takes time.
Prepare for peaks. Especially in the life of a start-up, a single lifehacker or reddit post can bring your system down if you’re always running at near top capacity. Keep enough margin so you can handle a sudden load or be ready to scale really fast.
Choose a language that won’t hold you back. Make sure the technologies you want to use have native clients in your language, or at least actively maintained ones. Don’t get stuck waiting for library updates.
Believe the hype. You want a technology that will grow with your product and will not die prematurely. A vibrant and active community and some noise about the technology can be a good indication for its survival.
Don’t believe the hype. Look for flame posts about the technology you’re evaluating. They can teach you about its weak points. But also don’t take them too seriously, people tend to get emotional when things don’t work as expected.
Have fun. Choose a technology that excites you. One that makes you think “oh this is so cool what can I do with it”. After all, that’s (also) what we’re here for.

Securing Elasticsearch with Finagle

2014-02-17T00:00:00+00:00

Elasticsearch is the hot new kid on the NoSQL-Big-Data block. It is a powerful tool and there are a lot of things you can do with it. But as with any system, there are just some things that are not built in. For instance - security.

What if you need HTTP access to your Elasticsearch, but want to protect it from unauthorized access? How do you add a security layer to your cluster?

There are a couple of common ways to handle this. One option is to use a security plugin (like this one, for example) that replaces the internal Netty server with another server that allows securing HTTP requests. Using a plugin does have its caveats, though, like being dependent on a plugin update if you want to update Elasticsearch to a newer version.

Another option is to use nginx as a proxy in front of Elasticsearch (check out this blog post for detailed instructions). While this is usually the simplest way to go, it might not always give you the full flexibility you need.

In this post I’ll explore a third possibility - building a small proxy server in scala using Twitter’s Finagle library to create an isolation layer in front of Elasticsearch. We will also see how to use the Elastcisearch native client within the proxy for increased performance.

Finagle is:

an extensible RPC system for the JVM, used to construct high-concurrency servers. Finagle implements uniform client and server APIs for several protocols, and is designed for high performance and concurrency.

For creating our server we’ll use the cool Twitter Server template, which gives us out of the box some administrative APIs, statistics and other goodies.

Constructing an HTTP proxy using Finagle is as simple as getting the request via the Finagle server and passing it on to a Finagle HTTP client (we’ll get back into the client thing in a bit):

// our simple service - just proxy the request to ES
val service = new Service[Request, Response] {
  def apply(request: Request) = {
    _client.send(request)
  }
}

Next we want to add some security to our requests. This is easy to do with Finagle’s composable nature. We can create a filter that allows our requests to go through only when using a magic word:

val authorization = new SimpleFilter[Request, Response] {
	def apply(request: Request, continue: Service[Request, Response]) = {
	  if ("abracadabra" == request.getParam("token")) {
	    continue(request)
	  }
	  else {
	    Future.exception(new IllegalArgumentException("Authorization enabled, wrong token"))
	  }
	}
} 

And now we’ll compose our service to use the filter, tell Finagle to route all requests to our service, and start the server:

def main() {
  // add the filter to our service
  val protectedService = authorization andThen service
  // add routing to the service
  HttpMuxer.addRichHandler("", protectedService)
  HttpMuxer.addRichHandler("/", protectedService)
  val server = Http.serve(":8080", HttpMuxer)
  onExit {
    server.close()
  }
  Await.ready(server)
}

Until now we assumed that our proxy is connecting to Elasticsearch via HTTP, so it just takes the request and re-sends it using Finagle as an HTTP client. But since our server is written in scala (or Java for that matter), we have the native Elasticsearch client at our disposal.

But first - why should we prefer to use the native client over Elasticsearch HTTP interface?

The reason is that we can use the Java node client of Elasticsearch. This is an operational node in the cluster that functions as a client (and not as a data node). It is cluster aware, it knows the structure of the cluster and can route requests to the appropriate data nodes (the ones that hold the data we need), thus saving a potential redundant hop that is likely to happen when using HTTP. The node client also takes some load off the data nodes by doing some of the work locally (like merging results from the shards that where queried).

Using the node client in our proxy is not as straightforward and requires a bit more hacking into the core of Elasticsearch. We’re going to pass the HTTP request we get directly into our Elasticsearch node client, and catch the response it generates in order to return it back.

First we extract the interfaces we need from the client - the Rest controller that handles Rest requests, and the internal Netty transport:

// build the node
val node = nodeBuilder().settings(settingsBuilder).data(false).client(true).build;
// start the node
node.start();
val internalNode = node.asInstanceOf[InternalNode];
// get the REST controller
_restController = internalNode.injector().getInstance(classOf[RestController]);
// get the transport layer
val networkService = internalNode.injector().getInstance(classOf[NetworkService])
_transport = new NettyHttpServerTransport(internalNode.settings(), networkService);

Now instead of re-sending the request via HTTP we’ll copy it and dispatch it to the Rest controller:

def send(request: Request) = {
	// copy the request to the ES namespace classes
	val esReq = ESClient.copyESRequest(request.httpRequest);
	// create a mock REST channel to capture the response from ES and return it
	val mockChannel = new MockRestChannel();
	// dispatch the request to the REST controller, using our own channel to receive the response
	val channel = new NettyHttpChannel(_transport, mockChannel, esReq)
	controller.dispatchRequest(new NettyHttpRequest(esReq, mockChannel), channel)
	mockChannel.future.asInstanceOf[Future[Response]];
}

As you can see, we provide our own mock channel to the controller. This is a Netty channel implementation that only implements the “write” method. Inside it we hold the future that is returned to Finagle. When the Elasticsearch client writes the response to the channel we update the future, so it will be returned back as a response to the original request:

class MockRestChannel extends MockChannel {
    // finagle future
	val future = new Promise[Response]
	override def write(any: Object): ChannelFuture = {
		if (any.isInstanceOf[HttpResponse]) {
			val rep = any.asInstanceOf[HttpResponse];
			// copy the response back to Netty namespace
			val response = ESClient.copyESResponse(rep);
			// update the future
			future.setValue(Response(response))
		}
		return new MockChannelFuture;
	}
}

So now we have a proxy server that can enforce security on HTTP requests to Elasticsearch AND use the native node client for better performance. Let the fun begin!
(The full code for this post can be found here on github).