Refresh your elasticsearch index with zero downtime

Posted 20/07/2014 by
Tagged As: | Categories: Elasticsearch, Ruby, Ruby on Rails | No Comments

I’m using a basic elasticsearch-rails setup on one of the applications I’m working on. The thing is, we sometimes need to make changes to the ES index, add new fields, or redo the existing ones. And when you have a database that needs at least 10 minutes(it was closer to 1 hour before we reworked the implementation), you can’t really afford that downtime. As ES-rails sets things up for you in a certain way, it was needed to use some of the underlying ES features to get the job done correctly.

So, it was needed to completely remove the downtime while doing the full reindex(deleting and creating the elasticsearch index). If you are already using elasticsearch-rails then you are familiar with the nomenclature, and probably with the methods described. For doing a complete reindex we require a rake task to do the job for us like this:

namespace :elasticsearch do
  task :reindex => :environment do
    index_name = Person.index_name
    Person.__elasticsearch__.create_index! force: true
    Person.all.find_in_batches(batch_size: 1000) do |group|
      group_for_bulk = group.map do |a|
        { index: { _id: a.id, data: a.as_indexed_json } }
      end
      Person.__elasticsearch__.client.bulk(
        index: index_name,
        type: "person",
        body: group_for_bulk
      )
    end
  end
end

This will effectively remove the index if it exists and reload the new data in the index, making it practically unusable until the task is done. Because this wasn’t an option, we looked into ES aliases and found them to be helpful. Basically, what we needed to do was to create an index with a unique name, and assign an alias to it, so that we could create and fill the index while the current one was still operational. So no downtime needed.

#...
index_name = "#{Person.index_name}_#{SecureRandom.hex}"
client = Person.__elasticsearch__.client
Person.__elasticsearch__.create_index! index: index_name, force: true
Person.all.find_in_batches(batch_size: 1000) do |group|
  #...
end
# to be sure there is no index named Person.index_name
client.indices.delete(index: Person.index_name) rescue nil
  # collecting old indices
  old_indices = client.indices.get_alias(name: Person.index_name).map do |key, val|
    { index: key, name: val['aliases'].keys.first }
  end
  # creating new alias
  client.indices.put_alias(index: index_name, name: Person.index_name)
  # removing old indices
  old_indices.each do |index|
    client.indices.delete_alias(index)
    client.indices.delete(index: index[:index])
end

So that is about it, you just call bin/rake elasticsearch:reindex and you have refreshed your elasticsearch index with zero downtime. Of course, you will have to implement some system to track the changed records while you were reindexing(remember we are working on a live system, so data is changing all the time). We used redis for that, to mark when reindexing has started $redis.set('elasticsearch:reindex_running', true) collected all changed record ids in a redis array, processed them via the regular indexing worker, and deleted the key after the alias was linked with $redis.del('elasticsearch:reindex_running').

What I have learned from a hackathon

Posted 13/04/2014 by
Tagged As: | Categories: Business, Learning, Ruby on Rails | Comments Off

I’ve attended a hackathon yesterday, that was the first “real” one in my life, and i’m still a bit under the influence. It was organised by the great people from Osijek Software City, and i thank them so much for the experience. I’ve learned so much, and almost nothing technical, because hackathons aren’t about it, as you will find out.

The topic: Crowd funding application

The topic was to make a simple crowd funding application. Nothing really complex, just a basic feature list with points for each complete feature.

The implementation: Rails

We chose Ruby on Rails as our implementation platform. That was a great choice, as we know it so well, and it would show later, that rails was the best framework for the rapid prototyping needed to win the hackathon.
The only thing lacking, where we lost a lot of points, was the front-end design, but you can’t expect a team of 3 backend developers and one software tester to create something really pretty. Although our UX was great, as we were using bootstrap, and not really in a great way.

The mistakes…

Overthinking it

We had a feature list to complete, which is fairly easy to do with rails, but we managed not to complete every functionality, just because we were loosing time hacking on the front-end to make it look acceptable.

Making it too complex

In a hackathon, time is working against you, you can say the same in product work, as unlimited time time is a thing no one has.
You have to make your features concise, and make them do only the stuff that needs to be done.

The learning…

KISS

Keep it simple, stupid. The best thing anyone can tell you, for any kind of product. Make it do only a couple of required features, and make that features well. Everything else doesn’t need to be done, at least at the moment. Don’t overthink the implementation, there are many currents in the rails development world which pull to this side or the other, but in this case, i must lean to @dhh and the pure rails way. You can optimise and improve later, after you have a product that is selling, and only the things that have to be optimised at the moment to keep the product running smooth.

Time is scarce, so make use of it

You don’t have time, as you are reading this, you have less and less of it. If you are still wasting it on nonsense, you won’t accomplish anything tangible. We had about 9 hours to make the application, and looking back, it was a couple of hours more than we needed, but we wasted time on initial configuration, setting up stuff, design. Lesson learned from this, Do the thing you are best at, and delegate everything else, if you can’t design to save your life, hire a great designer. If you can’t code, hire a great programmer. Just go outside, mingle with the community and you will surely find someone to team up with, if that isn’t the case, ask some of your acquaintances or friends, they will probably know someone who can help you build your dream.

Have a great team

The crucial thing any team needs is great communication, you don’t need ninjas, rockstars and whatnot. Anyone can build a web app, including high school kids who did a great job yesterday, i’m really happy that kids are going this path, as that will make their lives better.

Have fun

The main thing you are building a software product shouldn’t be money, or investors, but to have fun while doing it, if you aren’t having fun in your life, what is it worth to you then. I really like this job, and the people in the community are nice, if you are nice to them.

Final notes:

We had fun, we worked hard, i was swamped after spitting ruby code for 10 hours straight, and the experience was great. I met new people, had some interesting conversations, and saw other technologies people used to solve the same problem. We finished second, with a point difference of 15 points(the total possible was 1000), but it doesn’t really matter. The real reward is participating, and staying until the end, giving your best to solve the problem in hand, and supporting your team, as much as you possibly can.

Running a freelance business

Posted 24/01/2014 by
Tagged As: | Categories: Business, Freelance | Comments Off

So, you have decided to start freelancing? I am really happy for you! It is a
great choice, one of the best work related decisions I’ve made in my life. You
maybe have some clients, maybe you just got laid off, and want to try this
cool and fun freelancing life everyone seems to talk about. There surely isn’t
much about it, you have a client, you work for some amount of time for that
client, and get paid for the work. And you will be doing only the things you
love, none of that crap on that ancient architecture your last boss was making
you do. And you will work for whoever you want, because you are the boss
now
.

Yes, you have read that well, you are the boss now, and the CEO, and the
CFO, and every other department that existed in the regular job you used to
work. You also get to be the cleaner, and the system administrator.

How will you ever learn all of that, or even know everything there is to be
done? The easiest way is to find someone in the same business you want to get
into and ask them. Most people will gladly help, because they don’t mind
helping out someone in need. Some of them, well, won’t help you, maybe even
undermine you or your work, but, c’est la vie, some people are just assholes
and you have to get over it. The sooner you grow a thicker skin, the better
for you. So when you find out everything there is, and that depends from
country to country so I won’t get into details about what needs to be done
here, you have to figure out how to do it. Let’s get to that now.

You can, and should outsource a lot of that work, let the people better than
you do their jobs and pay them for your own piece of mind. Accounting, legal,
even assistant work can be outsourced to other people, freelancers or
companies. They are professionals, like you are in your field, and should be
on the edge of their domain, so you don’t have to. For example, I know very
little about the process of accounting, I could if I wanted to, as could
almost every one of us, but we don’t, because we have other stuff to think
about, learn about, and to run our now business. Or law, which I know nothing
about, but I’m sure a lawyer knows everything, or can refer me to the one that
does.

So, when you have outsourced everything that someone else can do instead of
you, there is still some work for you to do. Finding clients, marketing,
scheduling, IT work, being the CEO and the secretary at once, and taking care
of yourself, as you are now your company’s best asset. I’ll get to those in my
next couple of posts, because I don’t want you to fall asleep reading.

Working Remote

Posted 23/01/2014 by
Tagged As: | Categories: Business, Freelance, Remote | Comments Off

You are surely tired of your commute, every day, waking up way before it’s
convenient for you, cramming in public traffic, with people around you,
wasting your precious time day after day. I had a 10 minute walk to the last
place I worked at before going freelance, and it was nice, the commute before
that was 45 minutes each way. But even that, being tied to one place, for 8
hours each day, at a fixed time, can be nauseating. There is a solution:
Remote Working.

As you are aware, many companies nowadays offer the option for you to work
remotely full time. As a freelance consultant, this is probably the most
common way to work. Now, by saying remote, I don’t mean only and always
working from your home, but working from a place that is not your main office,
that can be a lot of places. Some people prefer coffee shops, some like me,
really love to work from home, some go to a co-working space, whatever floats
your boat.

So here are some benefits that you will see immediately or after some time,
and pitfalls that you can encounter in your remote working life if you don’t
take care of yourself.

Benefits

  • Less time commuting == More time spent with your family
  • No commuting expenses
  • Ability to choose the best time of day when you are most productive
  • Less stress and interruptions
  • Less meetings
  • Your written communication levels rise to the roof
  • You learn how to work better

Pitfalls

  • Not being able to separate work from private life
  • Isolation from other people can affect your mental health in a bad way
  • Letting yourself go physically

I am still a lousy writer, Jason and David are far better than me, so if you
want to really learn about remote work, and everything it carries with, go and
read great book on this subject Remote: Office Not Required
by guys from 37signals.

Why would you want to become a freelancer?

Posted 12/01/2014 by
Tagged As: | Categories: Business, Freelance | Comments Off

So what are your top reasons you want to become a freelancer? Feel free to
leave them in the comments section below. I will briefly give you my reasons
for doing it over the next couple of posts.

Be your own boss

The main reason everyone quotes is being fed up with the management at their
current place of work. That may be due to a lot of reasons. For example: Stuck
on boring projects; Can’t work from home sometimes; Don’t have a flexible schedule;
You can’t implement the new and hip technology you read about last week, because
you are stuck doing a VB6 project; And this list goes on and on…

The good parts

Being your own boss can improve that situation a lot, you get to choose the
projects to work on, the clients to work with and the technologies to
implement those projects to some extent. You get to choose when you work, and
where you work from.

The bad parts(not really)

You are your own boss, you are the one to set the working schedule, organize yourself, and
be proactive with work. There is no Boss telling you what you are going to
work on next. Also you will find that you have to work on some other
issues, as you have to run a company right now, a one man company of yourself,
you are your own boss right now, and manager, HR, and marketing. Each client is a kind of
a boss, as you are responsible to them for the work you are contracted to do.

The cure

Hire an accountant, and outsource as much stuff as you can. Time you spend on
learning your skill, they spend on learning their skills. Don’t do everything
by yourself, and remember, always be polite.

Flexible working hours/location

This is also a classic. All the freelancers out there seem to work whenever
they want, wherever they want.

The good parts

You work whenever/wherever you want, you can travel around the country, or
even abroad, and your work won’t suffer, your clients may not even notice that
you are 5000 km away from the place they think you are. But it really doesn’t
matter. Work happens where it happens for you, and it’s in your and your
clients’ best interest to be as productive as you can for them, and for
yourself, of course.

The bad parts

You can work the whole day, there is always something to be done. First you
forget lunch, then you forget your kids big game or play at school, as you
have so much work to do. If you don’t plan your time you will loose your
health/family and everything.

The cure

Make a schedule and stick to it. Being constrained to work 5 or 6 hours each day
makes you the most productive machine there is, because you haven’t got more
time, just those 6 hours each day, so about 30 hours a week.

This is it for this post, I will continue writing on this subject and the
tools to help you while freelancing or being a remote worker.