How I solve those tricky technical issues

Wait, Think and Wait some more. Huh? you say. That isn’t an answer. Let me explain. A tricky issue usually doesn’t have a clear solution. Because of this fact you are likely to botch up the solution with something that is complex and hard to maintain if you try to tackle it head on right away. What do I do in this case?

Simply hack together something that works and can easily be thrown away. The last part is key you DONT want this to hang around for years. You will need to be disciplined about this. Now every couple of days I think about the problem again. I don’t work on it, I just think about it. Potential solutions etc. I also talk to people about it and see what they say. After a few days, weeks or months of this and then BAM BOOoM. The solution is clear as day and it ROX. The key to this process is that you need to gain more and more insight into the problem and the potential solution. The key phase of waiting lets the information sink in and and most of all PREVENTS wasted effort. Avoiding extra work is a HUGE productivity booster.

Let me know what you think of this method or how you handle similar issues.

 

Using mysql as a bootleg message queue

This is not one of those typical high scalability articles about how to serve 10 billion pages from your iphone. This article is for the people just starting out, people who don’t want to spend a fortune on hosting, or even people who are just learning to program. As you may very well know it’s a good idea to push long running tasks off the execution of a web request. This is often done using a message q like rabbitmq or writting some erlang…however this brings me back to my point. We are either a newb or don’t have a server that can run additional process like rabbit mq on top of our apps that do the real work. Plus who wants to evaluate message q’s, learn the api, install it..etc. Lets keep it simple. We know mysql and we know…(insert language here) Just use them both.

The idea is simple. We will run a separate process that polls the database and picks off tasks. This process can be run from ANY server that can connect to the database including machines located in your house. Here is a sample table structure

CREATE  TABLE `mysql_message_q` (

`message_id` INT NOT NULL AUTO_INCREMENT ,

`message` MEDIUMTEXT NULL ,

PRIMARY KEY (`message_id`) );

I would recommend encoding your message as JSON. Now I can just run a few sql commands and my process will pick messages off the Q. OMG that is so simple. Not so fast. If we have multiple servers polling the database for tasks we will need to obtain an exclusive lock before picking off records. This can be done using mysql as well. Below are some sql queries and steps you should take to implement this.

 

SELECT GET_LOCK(‘someuniquename.lock.polling’,10)

select * from messageq limit 5

delete from messageq where message_id in (1,2,3,4,5) The numbers 1,2,3,4,5 will be replaced by the real id’s from the db that the previous select statement pulled in

SELECT RELEASE_LOCK(‘someuniquename.lock.polling’)  – Make sure to do this in a finally block

Do the long running work here

Update the database when the work is done

Rinse and Repeat

 

Boom you are done. Double love it. Enjoy it while your site is young and growing…once you hit twitter level traffic you can migrate to something that is more “enterprise level”.

Thanks for listening. Does anyone else have a simpler message Q idea?

Update:

@Nicolas Sebban suggested

I used linux native FIFO file type (man mkfifo) as a simple first-in-first-out queue. Each line of my file contains a json-encoded job, that I pop from the file when I’m ready to process a task.

It is simple and fast, and it has been working great for about 2 years now. It’s very low-tech, but it just works.

Great comments on hackernews

http://news.ycombinator.com/item?id=2528528

 

 

Sugar is Toxic? You be the judge

http://www.nytimes.com/2011/04/17/magazine/mag-17Sugar-t.html?_r=1&src=me&ref=general

This is terrible. As you know I am a lover of all things sugar. Icing, Coke, Pop, Coke, Soda Pop and Coke. This is an outrage. Why why someone create such wonderful products and then tell me it’s not healthy to consume them. It’s just not fair. I think it’s time to invent a sugar substitute that will actually make us healthier. I put this challenge to you…America. Thank you.

Announcing pushJS: Send javascript to any connected browser

http://github.com/mastaskillz33/pushJS Fork me on github

live demo:

http://li50-5.members.linode.com:8081/demo.html

http://li50-5.members.linode.com:8081/whoison

 

I created pushJS to send javascript updates to users of any website.

This project should make it very easy to send Javascript to connected user’s browsers.The communication is one way. In the future I would like to get a live feed of who is logged on and what they are doing.

This project has so many uses it’s hard to list them all. I could see a news website pushing the latest breaking news or a shopping website pushing a brand new deal to everyone who is logged in.

It works by using redis as a message q. When a message is pulled off the Q it is sent to the browser via socket.io of the specified user. Very simple

 

Let me know what you think.

 

Welcome to my new blog